Creating fully self-contained and pre-compiled library

General Usage
17

Yes, I have managed to make a self-contained library that can be relocated.
You can find the reference here: GitHub - RomeoV/RunwayLib.jl

In essence, my build and bundling steps are as follows:

  • Write your normal julia code. Make Base.@entrypoints (example). Write tests that test your code with JET.@test_opt and fix all tests, possibly by forking upstream repositories and fixing things. Depending on what you’re doing, this can be easy or challenging. It was challenging for me.
  • Make a new dir next to src, I call it juliac. Put a Makefile with something like
    COMPILED_DIR = MyLibCompiled
LIB_DIR = $(COMPILED_DIR)/lib

$(LIB_DIR)/libmylib.so: loadmylib.jl Project.toml Manifest.toml $(SRC)
    @mkdir -p $(LIB_DIR)
    JULIAC=$$(julia -e 'print(normpath(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "juliac", "juliac.jl")))' 2>/dev/null || echo "") ; \
    $(JULIA) --project=. --depwarn=error "$$JULIAC" --experimental --trim=unsafe-warn --compile-ccallable --output-lib $(LIB_DIR)/libmylib loadmylib.jl --relative-rpath

Manifest.toml: Project.toml ../Project.toml
    -rm -f Manifest.toml
    $(JULIA) --project=. -e 'using Pkg; Pkg.instantiate()'
    @touch $@ # Pkg.instantiate doesn't update the mtime if there are no changes
    Notice the --relative-rpath, which will allow us to put all the required libraries next to each other.
    Remember to run juliaup override set 1.12 in the directory.
  • Manually write a header file, and add
    # Create include directory with header file
$(INCLUDE_DIR)/libmylib.h: libmylib.h
	@mkdir -p $(INCLUDE_DIR)
	cp libmylib.h $(INCLUDE_DIR)/
    @tim.holy has started automating this: GitHub - JuliaInterop/JuliaLibWrapping.jl: Generate language-bindings for Julia-generated shared libraries
  • Make a Project.toml in juliac that contains something like
    [sources]
MyLib = {path = ".."}
    Also copy over any other custom sources from your main Project.toml file (it seems sources are not applied transitively…)
  • Make a simple loadmylib.jl file that just runs using MyLib
  • Run the compilation
    make all

Now it’s time to create the “bundle” that we can ship. This includes three aspects.

  1. ship all the required libraries (such as libjulia.so, libjulia-internal.so, libopenblas, etc) but try to strip out the libraries that we don’t need
  2. figure out which artifacts (share/julia) we have to ship
  3. make sure to compile on a machine where the GLIBC version is older than on your target deployment machine.
  • to bundle up all the libraries, you can do something like see EDIT below
    $(LIB_DIR)/.installed:
	@echo "Creating directory structure..."
	@mkdir -p $(LIB_DIR)
	@echo "Copying Julia libraries..."
	JULIA_LIB_DIR=$$($(JULIA) -e "print(joinpath(Sys.BINDIR, \"..\", \"lib\"))"); \
	cp -r "$$JULIA_LIB_DIR"/* $(LIB_DIR)
	@touch $(LIB_DIR)/.installed
	@echo "Julia libraries installed to $(LIB_DIR)"
    in your Makefile.
    To then filter for only the required libraries, I have not found a reliable way. What I have done is make a small mainc file that you can execute and see if it runs, and then by hand start deleting libraries and see if mainc crashes. Through this process, I made a required_libraries.txt file and then filter the bundled libraries with
    $(LIB_DIR)/.cleaned: $(LIB_DIR)/.installed required_libraries.txt
	@echo "Cleaning $(LIB_DIR)/julia directory based on required_libraries.txt"
	@echo "Before cleanup: $$(du -sh $(LIB_DIR)/ | cut -f1)"
	cd $(LIB_DIR)/julia && \
	for file in *.so*; do \
		[ -f "$$file" ] && \
		stem=$$(echo "$$file" | sed 's/\.so.*/.so/') && \
		! grep -q "$$stem" ../../../required_libraries.txt && \
		rm -f "$$file" || true; \
	done
	@echo "After cleanup: $$(du -sh $(LIB_DIR)/julia/ | cut -f1)"
	@touch $(LIB_DIR)/.cleaned
    I would warmly welcome a better way to do this.
  • to figure out which artifacts to bundle, I do something equally terrifying. see EDIT below I use PackageCompiler.jl to create a bundled version of my code, and then just copy over the share directory that PackageCompiler comes up with. Probably it’s possible to directly use PackageCompiler.bundle_artifacts, but I haven’t dug into it enough yet.
  • finally, you can compile everything first locally, and then on a CI machine running e.g. a ubuntu:22.04 container. I have had trouble getting everything to run under ubuntu:20.04, unfortunately. For examples, check my juliac workflow with relocation test or my python packaging (glibc) workflow.

Then, when this is all done you can even write a python interface to the c_api of your code and package it up as a python package, see e.g. here.
I am currently in the process of streamlining this.

Happy to have more discussions on this, and am looking forward to each step becoming more automated.

EDIT: It seems I have also found a way to bundle the libraries and artifacts more easily with PackageCompiler.jl. I think something like this should do the trick:

using PackageCompiler
ctx = PackageCompiler.create_pkg_context(".")
PackageCompiler.bundle_artifacts(ctx, ENV["COMPILED_DIR"]; include_lazy_artifacts=false)
stdlibs = PackageCompiler.gather_stdlibs_project(ctx)
PackageCompiler.bundle_julia_libraries(ENV["COMPILED_DIR"], stdlibs)
for (root, _, files) in walkdir(ENV["COMPILED_DIR"]), file in files
  (contains(file, "libjulia-codegen") || contains(file, "libLLVM")) && rm(joinpath(root, file); force=true)
end
16 Likes