QML.jl release pending

barche · July 27, 2020, 12:39pm

Hi everyone,

It has been in the works for a long time, but QML.jl is finally ready for release, with the registration PR pending. In the mean time, you can test it using:

add https://github.com/barche/QML.jl.git

See the README and the documentation (contributed by @bramtayl) for more info.

There is also a JuliaCon lightning talk about QML.jl Thursday. ( See trailer below )

qml

jheinen · July 27, 2020, 2:10pm

Works fine with GR, too (qml_ex.jl).

hexbin+qml

Thanks a lot, Bart!

ufechner7 · July 27, 2020, 5:18pm

Great job!

arhik · November 28, 2020, 12:30am

@barche Thanks for this repo. Does this work on arm64 as is or I will have to fiddle a bit. It worked on x86 directly but on arm64 currently I have this “UndefVarError: libjlqml not defined” errror. I tried jlqml building; it fails too.

arhik · November 28, 2020, 1:05am

Qt version could be reason. I was using 5.12. Default qt on my system failed first. I tried fetching newer version 5.12. Any suggestions here ? Thanks for your time. OS: Ubuntu 18.04 and aarch64 platform.

giordano · November 28, 2020, 1:36am

What version of Qt_jll do you have?

]status -m Qt_jll

arhik · November 28, 2020, 1:51am

Thanks for looking into this 5.15.1+0
[ede63266] Qt_jll v5.15.1+0

[edit] Thanks this gives me a hint to use qt 5.15 instead for building jlqml.

giordano · November 28, 2020, 1:59am

This version of Qt_jll should support your platform

arhik · November 28, 2020, 2:01am

Current error trace is something like this:

ERROR: LoadError: UndefVarError: libjlqml not defined
Stacktrace:
[1] top-level scope at /home/xxx/.julia/packages/QML/AvdBY/src/QML.jl:30
[2] include(::Function, ::Module, ::String) at ./Base.jl:380
[3] include(::Module, ::String) at ./Base.jl:368
[4] top-level scope at none:2
[5] eval at ./boot.jl:331 [inlined]
[6] eval(::Expr) at ./client.jl:467
[7] top-level scope at ./none:3
in expression starting at /home/xxx/.julia/packages/QML/AvdBY/src/QML.jl:30
ERROR: Failed to precompile QML [2db162a6-7e43-52c3-8d84-290c1c42d82a] to /home/xxx/.julia/compiled/v1.5/QML/JLkMo_W3O9X.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
[3] _require(::Base.PkgId) at ./loading.jl:1030
[4] require(::Base.PkgId) at ./loading.jl:928
[5] require(::Module, ::Symbol) at ./loading.jl:923

arhik · November 28, 2020, 2:06am

I will launch a new thread. I feel I am polluting this.

giordano · November 28, 2020, 2:45am

Oh, right, sorry, the missing library is libjlqml, not any of Qt

barche · November 29, 2020, 1:29pm

For now, you would still have to build the jlqml library yourself. Very soon now ™ we should have official bianries, though. You can track progress here, we just had a bit of setback with libcxxwrap-julia (a dependency for jlqml).

github.com/JuliaPackaging/Yggdrasil

Updating JLLs with `Dependency("libcxxwrap_julia_jll")`

opened 11:47PM - 21 Nov 20 UTC

closed 10:56PM - 05 Jul 21 UTC

fingolfin

Once we have versions of `libcxxwrap_julia_jll` for all Julia versions, we'll ne…ed to update the various JLLs using it to also provide multiple versions. Here I try to describe what's going on and why a change is needed; explain how to make that change; and track the progress of each affected package. ## What's going on Julia does not currently provide a stable ABI, meaning that things that were compiled against, say, Julia 1.3, may or may not work when linked at runtime against Julia 1.4/1.5/... -- and this holds of course for any combination of Julia versions. For libcxxwrap and anything using its headers (including all (?) JLLs using `Dependency("libcxxwrap_julia_jll")`), it is indeed a fact that a binary compiled against Julia 1.3 can lead to segfaults during GC when used with any later Julia version, and vice versa. Right now it is not definitely known whether other Julia combinations can lead to issues, but it is not unlikely that this will eventually happen. Thus the only safe way forward is to provide binaries for `libcxxwrap_julia_jll` which are compiled against each of Julia 1.3, 1.4, 1.5, .... In an ideal world, the julia version would simply be part of the `Platform`, and we'd be done. Indeed, @staticfloat added support for that -- alas, it will only work in Julia >= 1.6; older versions will balk when they see such platform identifiers. So as long as one wants to support any older Julia versions (which I'd expect to be everybody right now, as 1.6 has not yet been released, not even branched), extra work is needed. On the upside to all this, cxxwrap now supports many more platforms, which you can potentially benefit from for your own JLLs, too! ## How to deal with it **The following is an initial suggestion, and will be revised based on feedback; I suggest that people wait a bit with trying to implement this, until we've ironed out everything** In a nutshell, here's the plan: for each affected JLL, we build it several times, once for each supported Julia minor version (so e.g. one each for Julia 1.3, 1.4 and 1.5+), but built from the same source. For technical reasons, each such build will have to carry a separate version. The convention I propose is to set the patch level of your JLL to the minor version of Julia (so `libcxxwrap_julia_jll` 0.8.4 was built against Julia 1.4). Yes, that means you can't have a one-to-one mapping with the actual versions of the code your JLL wraps; but the longterm plan for JLLs is to completely decouple JLL versions from the versions of the wrapped software anyway. There are many ways to deal with this; I'll describe one of them here, which I am using in PR #1798. The core idea is to rename your current `build_tarballs.jl` to `common.jl` and then add one subdir for each supported Julia version, each with a `build_tarballs.jl` including `common.jl`. For example, PR #1798 renames `L/libsingular_julia/build_tarballs.jl` to `L/libsingular_julia/common.jl` and adds `L/libsingular_julia/libsingular_julia@1.3/build_tarballs.jl` which looks like this: ```julia const julia_version = v"1.3.1" include("../common.jl") ``` For technical reasons, the `build_tarballs.jl` for Julia 1.4 and 1.5 will be added in later PRs (AFAIK Yggdrasil tooling does not allow adding multiple versions of a JLL in a single PR -- so far that would have been a weird thing to do anyway). In `common.jl`, some changes are made: - the `version` is adjusted to set the patchlevel to match the Julia minor version: `version = VersionNumber(0, 2, julia_version.minor)` - the list of platforms is expanded to match the platforms supported by `libcxxwrap_julia_jll` (this could be neater, see also https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/957) - the dependencies are adjusted: - instead of `Julia_jll`, we now need `libjulia_jll` in the right version: `BuildDependency(PackageSpec(name="libjulia_jll", version=julia_version)),` - we inform `build_tarballs` about the runtime Julia requirements, by adding this keyword argument: `julia_compat = "$(julia_version.major).$(julia_version.minor)"` Note that this is not using `~` as one might expect, simply so that the build for Julia 1.5 can be used with Julia 1.6 until a `libjulia_jll` for 1.6 exists (yes, that might not work, due to ABI incompatibilities; but at least it has a chance of working, which is better than what we'd have if we used `~` in the version specification. Technically this means one could also use the JLL version made for Julia 1.3 in Julia 1.4; but since the JLL version for Julia 1.4 is strictly newer than that for Julia 1.3, this shouldn't happen unless a user forces that, and well, then they deserve it... UPDATE: we also recommend that all packages using `libcxxwrap_julia_jl` should use `preferred_gcc_version=v"8"` or higher; see https://github.com/JuliaPackaging/Yggdrasil/pull/2236 and https://github.com/JuliaInterop/libcxxwrap-julia/issues/75 ## Affected JLLs and their status Note: I plan to ping the respective authors once we have a stable plan, but for now, it seems premature to bother them. In particular until PR #2158 is ready... - [ ] `ConnectFourSolver` -- @findmyway - [ ] Julia 1.3 - [ ] Julia 1.4 - [x] Julia 1.5 -- PR #2323 - [x] `FastJet_Julia_Wrapper` -- @jstrube - [x] <s>Julia 1.3</s> - [x] <s>Julia 1.4</s> - [x] Julia 1.5 -- PR #2180 - [ ] `HelFEM` -- @mortenpi - [ ] Julia 1.3 - [ ] Julia 1.4 - [x] Julia 1.5 -- PR #2320 - [x] `LCIO_Julia_Wrapper` -- @jstrube - [x] <s>Julia 1.3</s> - [x] <s>Julia 1.4</s> - [x] Julia 1.5 -- PR #2181 - [ ] `OpenSpiel` -- @findmyway - [ ] Julia 1.3 - [ ] Julia 1.4 - [ ] Julia 1.5 -- PR #2325 - [ ] `SDPA` -- @ViralBShah - [ ] Julia 1.3 - [ ] Julia 1.4 - [x] Julia 1.5 -- PR #2327 - [ ] `jlqml` -- @barche - [ ] Julia 1.3 - [ ] Julia 1.4 - [x] Julia 1.5 -- PR #2324 - [ ] `libcgal_julia` -- @rgcv - [x] Julia 1.3 -- PR #2333 - [ ] Julia 1.4 - [ ] Julia 1.5 -- <s>PR #2326</s> - [x] `libcxxwrap_julia` - [x] Julia 1.3 -- PR #1948, #2150 - [x] Julia 1.4 -- PR #2106 - [x] Julia 1.5 -- PR #2158 - [x] `libpolymake_julia` -- @benlorenz - [x] Julia 1.3 -- PR #2306 - [x] Julia 1.4 -- PR #2313 - [x] Julia 1.5 -- PR #2316 - [x] `libsingular_julia` -- @fingolfin - [x] Julia 1.3 -- PR #1798, #2300 - [x] Julia 1.4 -- PR #2263 - [x] Julia 1.5 -- PR #2303 - [ ] `z3` -- @ahumenberger - [ ] Julia 1.3 - [ ] Julia 1.4 - [ ] Julia 1.5 -- PR #2321

arhik · November 30, 2020, 9:21am

Thanks. I managed to built it and I followed instruments of jlqml repo (overrides.toml). The library doesn’t pick the libjlqml.so still. I feel jlqml_jll needs aarch64 specific script in src folder.

arhik · November 30, 2020, 2:41pm

Had to add aarch64 wrapper script (just a copy of arm-linux). I made an aarch64 fork here. But I have a general question. How do you get git-tree-sha1 value. In my case I just replaced with expected value in error.

barche · November 30, 2020, 3:33pm

You should be able to add an entry to your ~/.julia/artifacts/Overrides.toml like this:

[6b5019fb-a83d-5b4e-a9f7-678a36c28df7]
jlqml = "/home/user/src/build/jlqml"

[ede63266-ebff-546c-83e0-1c6fb6d0efc8]
Qt = "/usr"

The directories passed here need to contain a directory lib that holds the actual libraries. Note that it’s best to also override Qt to the Qt you used to compile jlqml.

giordano · November 30, 2020, 4:30pm

Note that this is also documented in the JLL packages section of BinaryBuilder docs

arhik · December 1, 2020, 7:07am

@barche Yes. I agree it should work because the changes I made look trivial but as I mentioned I followed instructions in jlqml. It didn’t work. I am not sure why. May be its my system in some stale state is the problem. Thanks for this work. It works well so far.

barche · December 1, 2020, 10:52pm

Ah, I think I understand now, probably the override doesn’t work because the aarch64 architecture is not present at all in the JLL. The only way I can think of to do that is to check out the Yggdrasil tree, add aarch64 to the build_tarballs.jl for jlqml and then build using --deploy to your own github fork of the jlqml JLL package.