I have found that generating an app using PackageCompiler.jl only works for kind of academic examples since any dependency potentially breaks relocatabiliy. Our solution to this is to actually use docker images and bundle everything. Fine for deployed software but not really for “normal” users.
Will juliac solve this problem?
To the best of my knowledge this is an un-related problem that can be solved separately.
Unrelated to what? Isn’t the goal of juliac to build standalone executables?
The usefulness is very limited if they are not relocatable.
Unrelated to reducing the code size of the compiled shared library or executable which is the main goal of juliac.
At least when reading about it, it didn’t sound that way.
The --trim option is mandatory for generating smaller binaries. If you don’t use it the compiler will generate a binary file that is not trimmed.
Also the JuliaCon 2024 talk sounds different.
--trim is a part of the juliac app building, but not essential.
But maybe we are both right. v1.12 will add the first ability to produce small stand alone executables. But it will not work for every case as the relocatabiliy issue is not solved.
Was compiling a fully linked shared library or executable possible before?
it “solves” the problem across micro architecture (JULIA_CPU_TARGET), which PackageCompiler.jl can do already. But you won’t be able to cross-compile (e.g. from x86_64 to aarch64) (because Julia itself doesn’t)
How would that happen, and is there a way for dependencies to test for this kind of breakage?
FWIW I am currently building a relocatable julia library, wrapped into a python module (but can be used as an executable or C library aswell). See here and also the discussion in Creating fully self-contained and pre-compiled library. However, it is true that this has similar amounts of relocatability as PackageCompiler.jl but comes with the extra guarantee that no further code will be compiled at runtime. My goal is currently just to support x86 Linux usecases with glibc>=2.30. This is going well (after quite some effort). Do you need relocatability also to other architectures? In that case it may be possible to make separate builds for x86 and arm and different OSes and then handle somehow that the target system downloads the correct version (common usecase for package managers). In theory I believe this will be no problem.
For an example of the “relocatability” you can check this github workflow file which first builds the library on one machine and then tries to run it on another machine.
Actually, can you elaborate what you mean by this? Generally I don’t think this is true, dependencies should work fine with juliac and PackageCompiler alike.
well, you need worry more about downloading Julia right? because everything links to Julia runtime (or am I dreaming)
I think they’re referring to the fact that any dependency can potentially rely on paths created through @__FILE__, or simply assuming a specific working directory and blindly opening files relative to that. juliac won’t help with detecting that.
If I’m understanding this right, this works for the typically locally installed packages because those relative paths are freshly evaluated for the local installation during precompilation, but it doesn’t work for a relocated AOT compiled product because the paths are stuck at the machine that compiled it? Something else to check, this should be fine for some things like precompile-time includes, right, it’s only a problem if those paths are used at runtime?
OP might be talking about something else, so if that’s the case I don’t want to derail anything.
I mean relocatabiliy as it is stated in the PackageCompiler.jl docs.
I’ll give a few examples:
If I currently build an app with Oxygen.jl and run it on another machine, I cannot open the docs endpoint, since there is a js bundle missing where it points to the build machine.
I had the same problem with MakieGL.jl where some files were missing. And if I started the app it accessed the Pkg server and downloaded some artifact, totally unacceptable.
The problem is, that this is some kind of a code quality issue for each package. I think, however it has to be handled at language level.
It is not about cross compilation. Although that would be nice too. However I can achieve that using emulation, e.g. in docker, it is just really slow.
New juliac JuliaCon talk
Starts at 6:50:20
Please open an issue if you run into relocatability problems with GLMakie. We specifically test that basic relocation works in CI, but maybe the tests could be improved
Link to Relocatability section for discovery: Apps · PackageCompiler. Applies to apps and libraries, not sysimages.
That can only go so far. Nothing stops people from making a program that tries to open a file at a path that may not exist; end-users may even be instructed to manually make the path and files for the program to work (I really don’t like how often this has happened to me). Even most of the things we shouldn’t do at precompilation, which also automatically runs during installation like the build system discussed in the Relocatability section, can’t be prevented automatically; maybe I really did want to run rand at precompile-time. I expect there to be case-by-case issues for implementing relocatability just as there were many issues for implementing 1.9+ precompilation. There should be improvements and documented practices for relocatability over time, though, this isn’t something people should keep stumbling into and guessing.
True, but if think this should be alerted to the user. It might be a mistake that at precompile time a path is stored in the package scope somewhere. I would propose for this to even be forbidden by the compiler. Are there any good reasons for it?
If you mean a String being assigned to a global variable or a field/element of an object assigned to a global variable, then obviously nothing should stop that. A string that looks a lot like a file path may not be one. A path may not be dangerous if it’s only read at precompile-time (most includes) or it’s being written to. Besides, I think that’s breaking; every unrelocatable dependency suddenly becomes unusable.
Ok, it seems there is no easy fix, which is probably why it hasn’t been done. However, I feel that the goal of juliac should be to deliver “small relocatable stand alone executables”. 
For now what could be done to at least check if a package is surely relocatable or not? Could I somehow regex for paths? Could this new tracing stuff for --trim be used to check if a path to the host machine is used ever again and warn? If I build some software and distribute it to a customer or simply another user, I would like it to work.
And just for my understanding, the juliac executable will eventually be the thing that make_app from PackageCompiler.jl produces?