Creating a lIbrary with PackageCompiler takes ages

New to Julia
1

I’m trying to creating a library for my module using PackageCompiler but it takes forever. At this point the script is already running for 9 hours. Any ideas how I can speed this up?

create_library(
    package_or_project,
    dest_dir;
    lib_name = "lib.so",  # Custom name for the compiled library
    incremental = false,  # Create a fresh sysimage from scratch for maximum reliability
    filter_stdlibs = false,  # Only include standard libraries that are in the project file
    force = false,  # Force re-creation of the library if it already exists
    include_lazy_artifacts = false,  # Avoid lazy artifacts for simplicity and better performance
    include_transitive_dependencies = true,  # Include all dependencies to ensure full functionality
    include_preferences = true,  # Include project preferences
    sysimage_build_args = `-O3 --check-bounds=no`,  # Optimize for speed with bounds checking disabled
    cpu_target = "native",  # Optimize for the CPU of the current machine
    version = v"1.0.0",  # Version number of the library
    compat_level = "major",  # Set compatibility level to major
    script = nothing,  # No additional scripts are executed
    base_sysimage = nothing  # Do not base this sysimage on an existing one
)
2

-O3 looks a bit like overkill to me, I would use -O2

How much RAM do you have and how many cores?
On Linux you might need 32 GB RAM if you have 8 cores for the parallel build process, if you have less RAM you must limit the number of cores that are used.

3

Thanks a lot for the response @ufechner7. I’m creating the library in my local machine (4 cores/16GM RAM) on top of a docker container. I assume this is the problem. I could spin up an AWS instance with 16VCPUs and 32GBs of RAM. In that case what would be the ideal setting for JULIA_NUM_THREADS? Is there any other variable I need to set apart from this one?

4

https://docs.julialang.org/en/v1/manual/environment-variables/#env-image-threads

If you are running very low on memory, set this variable to a lower value, e.g. to 1, 2 or 3 on a machine with 16GB RAM:
JULIA_IMAGE_THREAD

But please check if you are really short of memory or not using free -h or htop.