[Yggdrasil] How to call Julia?

Tooling Package Management
1

Hi,

I’m starting a new Yggdrasil script (is that what you are supposed to call them?) to build a large and complex C++ library, OR-Tools. The end goal is to provide a Julia wrapper around it.

At some point during the build, I would need to generate Julia code (automatically, that is) using ProtoBuf.jl, during the build process of the library to ensure that the generated code is up-to-date with respect to the library.

However, in my script (in build_tarballs.jl), I cannot call julia: the command is not recognised! Here is the exact error message (full log):

[13:45:47]  ---> julia -e "using ProtoBuf; protojl()"
[13:45:47] /bin/bash: line 50: julia: command not found
[13:45:47]  ---> julia -e "using ProtoBuf; protojl()"
[13:45:47]  ---> julia -e "using ProtoBuf; protojl()"
[13:45:47] Previous command exited with 127
[13:45:47] Child Process exited, exit code 127

Here was my script:

cd $WORKSPACE/srcdir/or-tools*
# Apply patches.
mkdir build
cmake -S. -Bbuild …
cmake --build build
cmake --build build --target install
julia -e "using ProtoBuf; protojl()"

I know that the provided Julia script has zero chance of being correct, but the failure is that Julia cannot be built. How is that possible? Do I have to use some magic dependency? I can only find libjulia_jll that I could depend on, but that does not seem even remotely what I need (it only exposes libjulia, not the standard Julia interpreter).

Thanks!

2

I think you can just interpolate Base.julia_cmd() into the build string. e.g.

script = """
cd \$WORKSPACE/srcdir/or-tools*
# Apply patches.
mkdir build
cmake -S. -Bbuild …
cmake --build build
cmake --build build --target install
$(Base.julia_cmd()) -e "using ProtoBuf; protojl()"
"""

Note that if you want to interpolate into the string, you can’t use raw""" ... """, so you have to escape any literal $ symbols you want to stay in the script, e.g. for $WORKSPACE.

1 Like
3

Thanks, I’m giving this a try!

4

Aren’t you better off making ORTools_jll.jl which just builds the library, and then ORTools.jl which provides the Julia wrapper?

1 Like
5

That’s the plan, ORTools.jl will contain the actual MOI wrappers.

6

So why does the jll need Julia code?

1 Like
7

For the generated code, unrelated to MOI, using ProtoBuf.jl. Storing this code near the JLL ensures that it always corresponds to the installed binary, with no risk that the generated code is out of date and no manual operation to update it (unlike CPLEX.jl or Gurobi.jl) — although doing everything in the .jl package would be simpler.

8

I’ve tried this solution, but the corresponding Julia binary is not accessible:

/bin/bash: line 78: /cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.7/julia-1.7-latest-linux-x86_64/bin/julia: No such file or directory

The full value of $(Base.julia_cmd()) is:

/cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.7/julia-1.7-latest-linux-x86_64/bin/julia -Cnative -J/cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.7/julia-1.7-latest-linux-x86_64/lib/julia/sys.so -g1

The /cache/ mounting is not available at all (I’ve tried ls on it), it’s more than just this plugin.

I also tried to download the latest version of Julia, depending on the installed libc:

if [[ "$MACHTYPE" == *musl ]]
then
  curl -o julia-1.9.3.tar.gz https://julialang-s3.julialang.org/bin/musl/x64/1.9/julia-1.9.3-musl-x86_64.tar.gz
else
  curl -o julia-1.9.3.tar.gz https://julialang-s3.julialang.org/bin/linux/x64/1.9/julia-1.9.3-linux-x86_64.tar.gz
fi
tar -xvf julia-1.9.3.tar.gz
julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'

The glibc version fails because of the libc, it seems:

[15:13:36]  ---> julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'
[15:13:36] WARNING: failed to select UTF-8 encoding, using ASCII
[15:13:36] 
[15:13:36] [20864] signal (11.1): Segmentation fault
[15:13:36] in expression starting at none:0
[15:13:36] get_random_secret at /workspace/srcdir/musl-1.2.2/src/malloc/mallocng/glue.h:45 [inlined]
[15:13:36] __malloc_alloc_meta at /workspace/srcdir/musl-1.2.2/src/malloc/mallocng/malloc.c:50
[15:13:36] unknown function (ip: 0x1c4d20d)
[15:13:36] Allocations: 2997 (Pool: 2985; Big: 12); GC: 0
[15:13:36] /bin/bash: line 78: 20864 Segmentation fault      julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'

Using the musl libc yields an even stranger error:

[16:39:47]  ---> julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'
[16:39:47] ERROR: Unable to dladdr(&jl_get_libdir)!
[16:39:47]  ---> julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'
[16:39:47]  ---> julia-1.9.3/bin/julia -e 'using InteractiveUtils; versioninfo()'
[16:39:47] Previous command exited with 1
[16:39:47] Child Process exited, exit code 1
9

I’d just do this in the .jl package. It’s a lot easier.

2 Likes
10

To add to @odow’s suggestion to do this in the wrapper - if you want to make sure that it will always use the correct jll, you can use tight constraints in the compat section of your wrapper (e.g. 6. Compatibility · Pkg.jl).

When you update the version of the jll and the bindings in your wrapper, bump the version in the compat, and it all should stay closely coupled.

1 Like
11

To answer the original question: there is no way to call Julia from within Yggdrasil. That’s a bug no one is interested in solving (apparently, I’m the first one to have the problem). Therefore, do everything in the Julia package(s) you are building you JLL for, possibly throwing a new package in for good measure.

1 Like