PackageCompiler bundled libraries

[Julian]

I have successfully compiled a DLL using create_library:

But its file size and the amount of bundled libraries seem a bit excessive to me:

Basic DLL: 186MB
Bundled libraries: 331MB

I will not need most of the bundled libraries, is there a way to configure which Base and StdLibs libraries should be included?

Or, at least, to reduce the footprint of the generated DLL (186MB + additional 331MB of external DLLs!?!)?

Thank you!

[Palli]

Yes, a lot can be stripped, and I assume from libraries too, I at least know of:

create_sysimage(packages::Vector{String}; kwargs...)

…

• incremental::Bool: If true, build the new sysimage on top of the sysimage of the current process otherwise build a new sysimage from scratch. Defaults to true.
• filter_stdlibs::Bool: If true, only include stdlibs that are in the project file. Defaults to false, only set to true if you know the potential pitfalls.
• include_transitive_dependencies::Bool: If true, explicitly put all transitive dependencies into the sysimage. This only makes a difference if some packages do not load all their dependencies when themselves are loaded. Defaults to true.

Advanced keyword arguments

• base_sysimage::Union{Nothing, String}: If a String, names an existing sysimage upon which to build the new sysimage incrementally, instead of the sysimage of the current process. Defaults to nothing. Keyword argument incremental must be true if base_sysimage is not nothing.

E.g. that filter_stdlibs is important. You build incrementally on the already rather large sysimage, and it can be trimmed to be much smaller.

Fastest Julia startup challenge/smallest sysimage

@vchuravy or anyone who would like to help in the challenge. sys.so shrinks from 173M to 96M With julia --startup-file=no startup time improves to 77ms from 127ms on my machine That’s 39% faster startup, with a 44% smaller sysimage, so to a first approximation, the startup time of Julia scales linearally with the sysimage size. With that speedup I think it would really help Julia make top spot at the Debian benchmark game. My only requirement is that the sysimage/julia be usable for that…

I think you can already just delete libLLVM (and maybe OpenBLAS, at least could theoretically, to name some big offenders), if you do not need the compiler at runtime. I might be wrong, and it would bomb right away, or only later if you need it at runtime. Hypothetically you do not, e.g. LLVM is not used by (official, or will be) juliac, but it has not been released yet. The yet to be merged PR for it can be used, and then there are other unofficial projects, such as this Juliet language (not to be confused with https://www.kevinalbrecht.com/code/juliet/index.html the names chosen are kind of difficult to google for, see at least below), a fork of Julia (meant for hard-real-time robotics), statically typed that doesn’t use Julia’s runtime or LLVM, I just heard of, but I think it’s proprietary, and same with the Chinese Julia compiler that compiles to C++ (note a different Chinese company then Estun below).

Juliet may be a bit off-topic, since it’s a new static (no longer dynamic) language, with e.g. extra var keyword, but it seems it it can be “saved as Julia”, i.e. exported to run in the Julia environment, and maybe it can be used for libraries (it’s in many ways similar to the official upcoming juliac, except there no new keywords, only some of the dynamisim of Julia forbidden):

The Julia-based Juliet&Romeo motion-programming technology will be...

Short 8 min video on Juliet & Romeo very intriguing, and the parts before and after, I had seen this video separately before, and assume is the same one there at JuliaCon, the companies behind it are JuliaCon sponsors):

Juliet&Romeo; Making Julia a Hard Real-Time Robot Language | Cognibotics & Estun Automation | JCon24

[Julian]

Thank you for your very detailed reply, actually I can strip a few MB with the right combination of parameters (+deleting unnecessary DLLs), but I assume that the compiled DLL will still be in the range of 100+MB, despite the fact that it still depends on many other libraries.

[Julian]

Given the amount of external dependencies, I would expect a very slim DLL, limited to my code, but it exports literally “everything”:

These are the explicit dependencies of the generated DLL:
bin\libjulia-internal.dll
bin\libjulia.dll
bin\libgcc_s_seh-1.dll

Maybe I am doing something really wrong, or maybe that’s actually how the PackageCompiler module works…

[Palli]

I’m not an expert on PackageCompiler, but yes, it’s know to make large binaries, 200MB+ is too be expected, and while you can slim down, I don’t believe you can expect tiny binaries or libraries with it, yet at least. Upcoming juliac is for that, and those other options I mentioned.

There are even more like StaticCompiler.jl if it works for you, but it’s limited, when it works binaries are tiny.

Can I ask, why are you making libraries? Is it to call from C, or C++ (then consider jluna), or other? If you want to call from Python then I would simply use PythonCall.jl/juliacall, and not make libraries.

[jbytecode]

The legendary pull request of producing smaller binary executables has been merged to master today.

The classical hello world program is about 1.5 mbs and takes nearly 30 msecs to run, whereas a numerical integration of normal distribution (extensive loops) is 1.7mbs and takes nearly 40 msecs.

I prepared a thread about this on fosstodon, this thread is about my first tries.

Mehmet Hakan Satman (@mhsatman@fosstodon.org)

The generated output is currently depended on two Julia dynamic libraries. I think the next step will be producing statically linked executables.

edit: for those who don’t have fosstodon, the same thread in Twitter:

Mehmet Hakan Satman

[gdalle]

For reference, how big was hello world before?

[jbytecode]

As far as I remember, wait, hmm, 180mbs?