For example, where do I have to put the Preferences.jl file such that the choice of system MPI library with MPI.jl is respected while compiling the library?
Furthermore, is it possible to also change preferences after the library has been compiled - especially when it affects compile-time preferences such as (in the example above) the path to the system library and ABI.
Our use case is libtrixi, and more specifically LibTrixi.jl, where we want to compile a standalone library from a Julia package and then use it in an HPC context that includes external libraries such as MPI, HDF5, p4est, oder t8code.
Theoretically, if preferences are saved within the package compiler depot, they should apply, but compile-time preferences will not trigger recompilation and will have no effect, whereas runtime preference lookups will have an effect. This may not be what happens in practice, but that’s what should happen (theoretically).
Thanks again for the feedback! Based on this, I tried a couple of things, but was - mostly - unsuccessful
The following do not seem to work:
Setting compile-time preferences by storing them in the LocalPreferences.toml next to the Project.toml that was loaded when invoking PackageCompiler.jl’s create_library.
Setting compile-time preferences by storing them directly in the loaded Project.toml.
Setting load-time preferences by creating an appropriate LocalPreferences.toml in the library’s LOAD_PATH/DEPOT_PATH (i.e., in PREFIX/share/julia).
Setting load-time preferences by creating an appropriate Project.toml in the library’s LOAD_PATH/DEPOT_PATH (i.e., in PREFIX/share/julia).
This does seem to work:
Setting compile-time preferences by storing them in the Project.toml of the corresponding package that is being compiled into the library (the package has been dev’d into the PC.jl environment).
It seems to me that I’m either missing something about how preferences should be set to be considered by the created library, or there is something not working properly in PackageCompiler.jl. Any further suggestions?
Well, originally I didn’t think it would be worth to create another example, especially after @staticfloat having already provided a great test repo. However, I then decided to create a proper MWE myself, and lo!, I was unable to reproduce my earlier findings. Since this basically means that most of my original complaints are gone, I am not sure if I should or .
Anyhow, here’s the MWE repo I created for testing using Preferences.jl with PackageCompiler.jl-generated libraries:
With this, I was able to make the following work:
Setting compile-time preferences by storing them in the LocalPreferences.toml located in the package_dir argument to PackageCompiler.create_library
Setting compile-time preferences by storing them in the Project.toml located in the package_dir argument to PackageCompiler.create_library
Setting run-time preferences by creating an appropriate LocalPreferences.toml in the library’s LOAD_PATH/DEPOT_PATH (i.e., in PREFIX/share/julia).
Setting run-time preferences by creating an appropriate Project.toml in the library’s LOAD_PATH/DEPOT_PATH (i.e., in PREFIX/share/julia).
What still does not work:
Setting compile-time preferences in the library by setting them in the environment that invokesPackageCompiler.create_library (such that an existing package does not have to be modified in-source but can be loaded, e.g., from the default depot)
Setting run-time preferences in the library without having to add files to the install prefix of the library (such that it can really be done at runtime)
Re 1: I feel like this should be doable, to respect the preferences set in the PackageCompiler.jl environment. Alternatively, maybe it would be possible to add a local_preferences argument that accepts a user-prepared LocalPreferences.jl file with settings to be respected in the compiled library? I am thinking about use cases such as MPI.jl, HDF5.jl etc. where you need to switch out the JLL-provided binaries for system-local binaries.
Re 2: Right now, because of these two lines, there is no way to modify the DEPOT_PATH or LOAD_PATH from outside. Maybe it would be an option to not force setting these environment variables if they have already been set by the user? Or, alternatively, to have an additional set of environment variables that, if set, are used for the depot and load path?
However, with respect to the ability to set compile-time preferences, most of my original troubles have been solved by providing a LocalPreferences.toml in the package dir. It is now less a matter of necessity but more of convenience.
or 
.