Minimal Julia: What do you want in Julia, or not?

Palli · April 27, 2024, 5:01pm

We could have radically smaller Julia; at least 177 MB smaller (even more so with UPX compression), e.g. compiled Julia binary executables, if we want to, by a) dropping stuff and b) compressing the rest more. A lot of what I describe can be done in 1.x without breaking any compatibility, the rest with trivial breaking changes for 2.0.

The question is how far are we willing to go, what are we willing to drop from Julia for a non-default variant of it (later to become the default possibly).

Everything you (or someone else) want dropped from Julia could be added back easily for those that want some missing function.

The standard library has a cost, and it’s greater in Julia, e.g. because of (matrix) math other languages do not provide. Python has also had a big standard library and is drastically reducing it in 3.13 alpha:

PEP 594 (Removing dead batteries from the standard library) scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3.

Many other removals of deprecated classes, functions and methods in various standard library modules.

New deprecations, most of which are scheduled for removal from Python 3.15 or 3.16.

All of those can go away:
$ ls -lrSh .julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/

[…]
? -rwxr-xr-x 1 pharaldsson pharaldsson 117K mar 1 11:11 libz.so.1.2.13
[…]
-rwxr-xr-x 1 pharaldsson pharaldsson 175K mar 1 11:11 libmbedx509.so.2.28.2
? -rwxr-xr-x 1 pharaldsson pharaldsson 216K mar 1 11:11 libklu.so.2.2.1
-rwxr-xr-x 1 pharaldsson pharaldsson 222K mar 1 11:11 libopenlibm.so.4.0
-rwxr-xr-x 1 pharaldsson pharaldsson 288K mar 1 11:11 libmbedtls.so.2.28.2
-rwxr-xr-x 1 pharaldsson pharaldsson 312K mar 1 11:11 libssh2.so.1.0.1
-rwxr-xr-x 1 pharaldsson pharaldsson 463K mar 1 11:11 libspqr.so.4.2.1
probably not: -rwxr-xr-x 1 pharaldsson pharaldsson 505K mar 1 11:11 libunwind.so.8.0.1
? -rwxr-xr-x 1 pharaldsson pharaldsson 601K mar 1 11:11 libuv.so.2.0.0
-rwxr-xr-x 1 pharaldsson pharaldsson 640K mar 1 11:11 libmbedcrypto.so.2.28.2
-rwxr-xr-x 1 pharaldsson pharaldsson 653K mar 1 11:11 libpcre2-8.so.0.11.2
-rwxr-xr-x 1 pharaldsson pharaldsson 698K mar 1 11:11 libgmp.so.10.4.1
? -rwxr-xr-x 1 pharaldsson pharaldsson 715K mar 1 11:11 libgcc_s.so.1
-rwxr-xr-x 1 pharaldsson pharaldsson 728K mar 1 11:11 libnghttp2.so.14.24.1
-rwxr-xr-x 1 pharaldsson pharaldsson 739K mar 1 11:11 libcurl.so.4.8.0
-rwxr-xr-x 1 pharaldsson pharaldsson 820K mar 1 11:11 libumfpack.so.6.2.1
-rwxr-xr-x 1 pharaldsson pharaldsson 981K mar 1 11:11 libquadmath.so.0.0.0
-rwxr-xr-x 1 pharaldsson pharaldsson 1,4M mar 1 11:11 libcholmod.so.4.2.1
? -rwxr-xr-x 1 pharaldsson pharaldsson 1,5M mar 1 11:11 libgomp.so.1.0.0
-rwxr-xr-x 1 pharaldsson pharaldsson 1,7M mar 1 11:11 libgit2.so.1.6.4
-rwxr-xr-x 1 pharaldsson pharaldsson 2,5M mar 1 11:11 libmpfr.so.6.2.0
-rwxr-xr-x 1 pharaldsson pharaldsson 2,7M mar 1 11:11 libblastrampoline.so.5
-rwxr-xr-x 1 pharaldsson pharaldsson 9,1M mar 1 11:11 libgfortran.so.5.0.0
? -rwxr-xr-x 1 pharaldsson pharaldsson 13M mar 1 11:21 libjulia-internal.so.1.10.2
? -rwxr-xr-x 1 pharaldsson pharaldsson 21M mar 1 11:11 libstdc++.so.6.0.32
-rwxr-xr-x 1 pharaldsson pharaldsson 32M mar 1 11:11 libopenblas64_.0.3.23.so
-rwxr-xr-x 1 pharaldsson pharaldsson 65M mar 1 11:21 libjulia-codegen.so.1.10.2
-rwxr-xr-x 1 pharaldsson pharaldsson 91M mar 1 11:21 libLLVM-15jl.so
-rwxr-xr-x 1 pharaldsson pharaldsson 231M mar 1 11:19 sys.so

The last one, the sysimage, sys.so, will not go away but would be much smaller, by at least half (when I did my experiment to drop LinearAlgebra and more), probably way more.

B.
In addition for what is left (or even if nothing dropped) you can additionally compress:

UPX will typically reduce the file size of programs and DLLs by around 50%-70%

excellent compression ratio: typically compresses better than Zip, use UPX to decrease the size of your distribution!

very fast decompression: more than 500 MB/sec on any reasonably modern machine

no memory overhead for your compressed executables because of in-place decompression

It supports all the (tier 1) platforms Julia does (and more, e.g. NetBSD), with few exceptions (for non-tier-1) seemingly; I see in the code:

throwCantPack("This test UPX cannot pack .so for MIPS or PowerPC; coming soon.");

It has a good license seemingly:

… or (at your option) under the GPLv+2 with special exceptions and restrictions granting the free usage for all binaries including commercial programs …

Historically the sysimage, sys.so, didn’t have any native machine code in it(?), or at least packages didn’t, but now both do, and it’s a large fraction. UPX is a compressor/“packer” for executable machine code specifically (but handles all I believe, meaning e.g. doc sections in the sysimage, that also might actually be just dropped…).

Currently Julia bundles zlib, and it’s a generic compressor, not for executables, x86, or ARM code like UPX supports. Why it or other similar can’t be as good. Historically when packages distributed only source code or half compiled LLVM bitcode then it may have been better.

C.
I think we should go all in: do as minimal Julia as we can since we’re breaking compatibility with this new non-default anyway, though I could see compromising for now (or not) on e.g. regex support, i.e. on e.g. smaller dependencies like 653K libpcre2-8.so. [Julia’s standard lib may also need it, for now.]

Otherwise here are the low-hanging fruit, in order from most payoff:

Historically C had libc, and separate libm for math (I believe merged in some platforms, Android? Windows?), i.e. for floating-point math, the basic operators and square root (and more). It made sense when done in software, when memory was very tight. By now, those compile to individual assembly instructions, though maybe not square root. I don’t believe libm has any (2D) array operations, e.g. not square root of a matrix (though such element-wise operations are possible).

The cost of e.g. *, / (and \) or square roots is not high in Julia, for scalars or element-wise for matrices/arrays, but it’s huge (in code size; and at runtime) when applied to matrices in full, and all operators/methods/functions are generic in Julia (doesn’t mean we need to support all, i.e. for non-scalars with the sysimage/standard lib, i.e. going way above what most languages/libm do e.g. C and Python that has NumPy separate). The 32M libopenblas64 can simply be dropped without breaking compatibility, in 1.x. It has better alternatives like BLIS.jl and MKL.jl enabled by the 2,7M libblastrampoline.so that could be only kept, but actually it could also be dropped for a minor inconvenience, then you need to add it through a package. The cost of those .so is born by e.g. any GUI app regardless of if the app (matix) math-based or not. OpenBLAS doesn’t work in WebAssembly, so if simply dropping it (or such capability) then Julia would be more cross-platform, work better on the web (for that subset).

Julia has Downloads.download available, that I would want to deprecate, and it, and Pkg that uses it indirectly, need libcurl.so (and currently libmbedtls.so. and libmbedx that are being dropped). For many (compiled) Julia programs, e.g. CLI/scripts, also GUI, it’s just NOT needed (still very convenient in the REPL and download could be kept there; would it be possible to have functions ONLY defined there, but NOT in a general program, running from that REPL?), but almost all uses of the download functionality also need some TLS/SLL library, and OpenSSL is being added to replace MbedTLS. Julia needs to be secure when you ask for downloads (and uploads), but I’m not convinced any download/upload functions (needing moving-target security libs, when done right) should be in Julia itself, only in a separate stdlib, that Pkg depends on (it’s already separate and not loaded by default in the REPL), and that library could be used with using Security (from the General registry, and it ideally would auto-update), and it would bring in Downloads (I don’t see much value in non-https download capability in standard Julia) and libssh2.so. [I suppose also the 728K libnghttp2.so, for now, though it seems fully not needed, we want HTTP/3 and HTTP/1.1, HTTP2 is redundant.]

What is 21M libstdc++.so actually needed for? I think it’s only needed for the largest dependency of Julia 91M libLLVM-15jl.so, that CAN be dropped already, well for compiled apps, sometimes (in cases when you don’t need the compiler at runtime). Julia like most projects at languages depends on libc, but a lot of code in and out of Julia ecosystem is not C++ and doesn’t needs its standard library, so it seems libstdc++.so it should only be a dependency of CxxWrap.jl and similar, for when you actually need a C++ JLL package. I suppose it’s a breaking change to drop it, but if done correctly it shouldn’t in effect be, i.e. all would just need to updated to latest CxxWrap that would be made to include it.

What got me thinking this time around was e.g. seeing the title of this new bug: Error in Sqrt for julia > 1.10.2 · Issue #54062 · JuliaLang/julia · GitHub

sqrt(A) is failing when A is non-symmetric with repeated eigenvalues.
I’m for now posting this in General rather than Community or Internal, it can be moved, though I like to hear from the public.

And I was thinking, what, square root has a bug all of a sudden? But it’s not the scalar kind (no worry it’s ok, nor am I, for now, proposing dropping it from Julia), only for matrices.

So when did you last take a square root of a matrix? By dropping it the rest of Julia would have lower bug-density. Some functionality can reside elsewhere requiring using LinearAlgebra to work (I’m thinking an ENV var would do it impicity for you, or not, to not require breaking 2.0 release).

Actually besides the bug, it has a performance regression “bug” now takes 23 sec. 5x longer than the 5 sec in 1.6:

$ time julia +1.6 -e "sqrt([1.0 2.0; 3.0 4.0])"

I’m not sure why, likely it’s no longer compiled into the sysimage, a good thing, but neither into LinearAlgebra, or a huge regression in the compiler? But both are fast after first use, so I’m not really worried.

I like the “platform” feature of the Roc language, i.e. them selectable. It has e.g. CLI platform (Unix-like, i.e. with filesystem support) “platform” and then alternatives like a web-programming platform, then no filesystem support available there (client side) nor wanted (in the standard library). I suppose Julia running on such, i.e. WebAssembly, could also do without, and I guess then (and only then?) the 601K libuv.so can be dropped. [Web work also benefits from different CG-strategy, eliminate it or defer it, or at least non-multi-threaded GC on but client and server, and I understand Roc to do differently for the Web platform.]

I’m for now posting this in General rather than Community or Internal, it can be moved, though I like to hear from the public.

mkitti · April 27, 2024, 6:37pm

A standard Julia Runtime distribution with no codegen or LLVM would be great as a deployment target.

Benny · April 27, 2024, 11:14pm

I presume you are thinking of loading a cached subset of Julia via imports or PackageCompiler.jl features? I also wonder if interpretation can loosen that “no codegen” limitation while adding much less to the runtime than the compiler.

mkitti · April 27, 2024, 11:41pm

In Java, you can zip up your classes into a MyApp.jar file. If the user has the Java Runtime Environment installed, they can double click on the MyApp.jar file and the application runs. You can also run it from the command line via java -jar MyApp.jar. Multiple JRE installs are not needed for each Java application.

For Julia, I want to be able to send over the system image as a MyApp.jle. When the user double clicks on it, it runs julia -e "main(ARGS)" -J MyApp.jle, and then my application runs. Multiple Julia runtimes are not needed for each Julia application.

Java jar files usually do not contain native code. The problem for Julia is that a system image is native code and it is usually specific to a particular version of Julia.

While some Java applications ship with a JRE, such as Fiji.sc, within an organization you can often just ship the jar file. You can even download a non-JRE image of Fiji.sc above.

joa-quim · April 28, 2024, 12:01am

I wish Julia try, if possible, to keep its size under more control. (no disk size is NOT free on laptops)

Windows here

dir /s Julia-1.9

...
     Total Files Listed:
            1826 File(s)    619,604,932 bytes

dir /s Julia-1.10

...
     Total Files Listed:
            1945 File(s)    702,725,493 bytes


dir /s Julia-1.11

...
     Total Files Listed:
            2164 File(s)    911,601,200 bytes

dir /s Julia-1.12

...
     Total Files Listed:
            2203 File(s)    880,076,048 bytes

mikmoore · April 29, 2024, 2:23pm

I’ve long wanted LinearAlgebra to be an actual library and not something that’s loaded anyway and merely missing its exports. The big issue here is that many matrix operations are done via the use of Base functions (e.g., Base.:*, Base.sqrt) on Base.Array, so defining it anywhere outside Base is piracy.

To fix this, we could have proper linear algebraic types. Ideally, we could reclaim the (Abstract)Vector/(Abstract)Matrix aliases for this purpose and rename the current (Abstract)Array{T,N} aliases to something else. Then linear algebraic operations would only operate on these types and would fail for generic Array. Slightly more “simply”, we could accept that this will be handled via piracy and one would need to import LinearAlgebra to pirate functions to operate on AbstractArray.

Either is breaking. The linear-algebra types method is quite breaking (especially if it were to steal the Vector/Matrix aliases). The piracy method is less breaking, because a import LinearAlgebra anywhere in the code (i.e., it’s one line to add or might already be present) would pirate functions to restore current behavior. But it’s piracy, which has its own intrinsic cost and it sets a bad precedent.

Neither seems palatable in Julia v1.X and they might even be too contentious on a hypothetical v2.

Palli · April 29, 2024, 3:19pm

The non-default I had in mind for 1.x would yes be slightly breaking, but since non-default, would be not breaking at all, by default as is (as fast, or if people ok with it slower).

I.e. for it you could dousing LinearAlgebra and then you restore full 1.x compatibility (and speed). But you would have to do that for the non-default, and some ENV var Julia1 could control it for you, if not set, or to true, then this would happen implicitly.

Currently OpenBLAS is loaded even before you do using LinearAlgebra, to get access to basic operators. But we can at least drop OpenBLAS keeping full compatibility, meaning same semantics and syntax, just use slower naive matmul until you do using LinearAlgebra that would bring in OpenBLAS (or at least libblastrampoline.so).

What could be implemented, and in what phases or order (some are independent):

Drop OpenBLAS, that’s a low-hanging fruit, easy to do, wouldn’t really shrink the sysimage itself, nor really startup in a big way (though RAM used).
The non-default: drop LinearAlgebra semantics, by default, we would still parse arrays, and store, i.e. everything in the parser still relevant. This shrinks the sysimage, and improves thus startup speed.
Drop libmpfr.so, another low-hanging fruit, most don’t need it. libgfortran.so is actually larger, I assume drop it, just forget what’s it for… when and libstdc++.so would be great to drop, since even larger, just unclear easily possible, since LLVM is usually needed, and it’s hard to drop it (only possible for some compiled apps).

Right, the simplest thing is simply dropping 91M libLLVM-15jl.so (and I suppose the 65M libjulia-codegen.so.1.10.2) [for compiled apps], and if code gen is needed then simply segfault… I believe actually possible. But yes, the interpreter as a fallback, for those few (or none!) cases when needed at runtime, to keep things running in all cases.

This doesn’t work (but such DOES work for libLLVM.so, which is the symlink to it):

$ mv .julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/libLLVM-15jl.so away

$ julia --compile=min
ERROR: Unable to load dependent library /home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/bin/../lib/julia/libjulia-codegen.so.1.10
Message:libLLVM-15jl.so: cannot open shared object file: No such file or directory

The UPX packer, assuming it packs better, should be a clear win, for Julia, and compiled packages, assuming can work there. This is conceptually low-hanging fruit, but isn’t really since harder to develop/integrate code, than to drop code. But it’s at least fully non-breaking. Maybe this should be nr. 1 or at least can be done concurrently, by different people.

There are things you can do already, such as gc, and you strictly speaking don’t need to keep all the Julia installations around…

But you’re right, Julia has ballooned in size by 2.7x (from 1.0.5) or 3.7x (and the stray 32-bit x86 julia I have is only 17% smaller):

229M .julia/juliaup/julia-0.6.4+0.x64.linux.gnu
314M .julia/juliaup/julia-1.0.5+0.x64.linux.gnu

317M .julia/juliaup/julia-1.1.1+0.x64.linux.gnu
326M .julia/juliaup/julia-1.2.0+0.x64.linux.gnu
340M .julia/juliaup/julia-1.3.1+0.x64.linux.gnu
354M .julia/juliaup/julia-1.4.2+0.x64.linux.gnu
367M .julia/juliaup/julia-1.5.3+0.x64.linux.gnu
369M .julia/juliaup/julia-1.5.4+0.x64.linux.gnu
319M .julia/juliaup/julia-1.5.4+0.x86.linux.gnu
393M .julia/juliaup/julia-1.6.7+0.x64.linux.gnu
425M .julia/juliaup/julia-1.7.3+0.x64.linux.gnu
471M .julia/juliaup/julia-1.8.0+0.x64.linux.gnu
471M .julia/juliaup/julia-1.8.1+0.x64.linux.gnu
465M .julia/juliaup/julia-1.8.2+0.x64.linux.gnu
450M .julia/juliaup/julia-1.8.3+0.x64.linux.gnu
452M .julia/juliaup/julia-1.8.5+0.x64.linux.gnu
499M .julia/juliaup/julia-1.9.0-rc1+0.x64.linux.gnu
500M .julia/juliaup/julia-1.9.4+0.x64.linux.gnu

561M .julia/juliaup/julia-1.10.0-beta1+0.x64.linux.gnu
577M .julia/juliaup/julia-1.10.0-rc1+0.x64.linux.gnu
579M .julia/juliaup/julia-1.10.2+0.x64.linux.gnu
478M .julia/juliaup/julia-1.10.2+0.x86.linux.gnu
847M .julia/juliaup/julia-1.11.0-beta1+0.x64.linux.gnu

823M .julia/juliaup/julia-nightly

Individual packages can also be very large like Makie 89 MB and Wakame 185 MB:
du -s .julia/packages/* |sort -h
22848 .julia/packages/RDatasets
29548 .julia/packages/CSV
31048 .julia/packages/Interpolations
34396 .julia/packages/OrdinaryDiffEq
51648 .julia/packages/GR
90352 .julia/packages/Makie
188424 .julia/packages/Wakame
266376 .julia/packages/Groebner

Note, misleading for some packages, e.g. 261 MB for Groebner, since it’s seems redundantly stored:

du -hs .julia/packages/Groebner/*
127M .julia/packages/Groebner/BFrEm
6,4M .julia/packages/Groebner/fcANv
128M .julia/packages/Groebner/GRCIz

Note, Groebner/fcANv is oldest, then the other got some large additions:

252K .julia/packages/Groebner/GRCIz/experimental/linear-algebra
940K .julia/packages/Groebner/GRCIz/experimental/runge-kutta
76M .julia/packages/Groebner/GRCIz/experimental/vectrref [containing many large matrices in .jld2 files]
…

[Linear algebra is clearly costly, not just in core Julia…]

(@v1.10) pkg> gc
      Active manifest files: 22 found
      Active artifact files: 356 found
      Active scratchspaces: 31 found
     Deleted 48 package installations (51.067 MiB)
     Deleted 16 artifact installations (1.081 GiB)
     Deleted 1 scratchspace (0.000 byte)

Grobner is still there after, with 3 installations…

DNF · April 29, 2024, 3:31pm

I use linear algebra semantics absolutely all the time, but would be very happy with separating vector/matrix from Array. I always found that mix a bit ‘messy’ anyway, and having to do using LinearAlgebra (or import ) would be perfectly fine.

(This is v2.0, of course)

joa-quim · April 29, 2024, 4:00pm

I have a 4 TB disk, so no problems for me but I see the struggle of some students when they try to install Julia+packages in their laptops.

And you are not even looking at the compiled dirs where multi GB are often found.

Palli · April 29, 2024, 4:24pm

You’re right (such as for me largest single package):
-rwxrwxrwx 1 pharaldsson pharaldsson 155957688 okt 29 2023 .julia/compiled/v1.10/OrdinaryDiffEq/DlSvy_6EN87.so
-rw-r–r-- 1 pharaldsson pharaldsson 6025190 nóv 6 11:47 .julia/compiled/v1.10/OrdinaryDiffEq/DlSvy_6EN87.ji
-rwxrwxr-x 1 pharaldsson pharaldsson 162222968 des 12 10:42 .julia/compiled/v1.10/OrdinaryDiffEq/DlSvy_vWmI1.so
-rw-r–r-- 1 pharaldsson pharaldsson 6025328 des 12 10:42 .julia/compiled/v1.10/OrdinaryDiffEq/DlSvy_vWmI1.ji

and for LoopVectorization:
-rw-r–r-- 1 pharaldsson pharaldsson 7752044 mar 8 2023 4TogI_9LtBo.ji
-rw-r–r-- 1 pharaldsson pharaldsson 7754210 mar 8 2023 4TogI_ahEnU.ji
-rw-r–r-- 1 pharaldsson pharaldsson 13449204 mar 8 2023 4TogI_3SlYI.ji
-rwxrwxrwx 1 pharaldsson pharaldsson 15486592 okt 29 2023 4TogI_6EN87.so
-rw-r–r-- 1 pharaldsson pharaldsson 634065 nóv 6 11:47 4TogI_6EN87.ji
-rwxrwxr-x 1 pharaldsson pharaldsson 16670288 des 12 10:39 4TogI_vWmI1.so
-rw-r–r-- 1 pharaldsson pharaldsson 634203 des 12 10:40 4TogI_vWmI1.ji
-rwxrwxr-x 1 pharaldsson pharaldsson 15646664 apr 18 14:20 4TogI_1XyXR.so
-rw-r–r-- 1 pharaldsson pharaldsson 635022 apr 18 14:29 4TogI_1XyXR.ji

That’s for 1.10, and the trend seems to be a .so compiled for a .ji, paired as expected, but why .so missing for some .ji files? And are these pairs redundant multiples which could be dropped, and why didn’t gc do that? That would be a plausible first step then think of UPX packing of .so after that…

Julia and Python have Base64 de/encoding available, and Python 3.13 is adding more space-efficient Z85, similar but better than Ascii85, encoding:

Rather than just drop the Base64 module, likely too breaking, for little benefit, I think Julia could and should(?) add this new encoding. I’m looking into where it could be used (or Base64 is now used) and possibly in Serialization (indirectly helping where, in .jld2, e.g. in some packages); also by Markdown, but seemingly that can’t change.

gitboy16 · April 29, 2024, 5:07pm

Did anyone manage to use upx successfully? I have tried on Windows but it made Julia un-usable (i.e. a lot of error when I start julia.exe?

Palli · April 29, 2024, 10:21pm

Yes, I have now. Success! Seemingly (move/take a backup of the relevant files before testing). It works well for executable files, julia, juliaup, julialauncher, without slowdown, but not as well for sys.so, maybe better for other .sos (might be great for e.g. LLVM one if you need to have it but not fast startup or always available, could be deferred to start it or some others)?

Before, startup was 193.7 ms, and I DO get compression (the compression itself relatively fast under a min.):

$ time ./upx-4.2.3-amd64_linux/upx /home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2024
UPX 4.2.3       Markus Oberhumer, Laszlo Molnar & John Reiser   Mar 27th 2024

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
 241252240 -> 169848900   70.40%   linux/amd64   sys.so

but it’s 90% slower to start (32-bit Julia had only 30% slowdown if I recall), based on min time, a one-time cost, ok some:

$ hyperfine '/home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/bin/julia -e ""'
Benchmark 1: /home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/bin/julia -e ""
  Time (mean ± σ):     416.5 ms ±  33.4 ms    [User: 370.3 ms, System: 156.5 ms]
  Range (min … max):   373.0 ms … 466.5 ms    10 runs

I did try non-default options: --brute (took 13 min.):

 241252240 -> 162406468   67.32%   linux/amd64   sys.so

$ /home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/bin/julia -e ""
Trace/breakpoint trap (core dumped)

and --ultra-brute same result, likely took 20+ min. to compress… and watching many progress bars, apparently going through 48(?) methods, all of them I guess, and then hopefully getting minimum size:

 241252240 -> 162336836   67.29%   linux/amd64   sys.so

On least amount of compression with -1 I get (and still slower startup at 407.6 ms):

 241252240 -> 183201860   75.94%   linux/amd64   sys.so

Maybe some other combination minimizes slowdown, or actually gets startup to be faster…

  10792568 ->   3632708   33.66%   linux/amd64   juliaup

32-bit Julia starts faster…:

$ hyperfine '/home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x86.linux.gnu/bin/julia -e ""'
Benchmark 1: /home/pharaldsson/.julia/juliaup/julia-1.10.2+0.x86.linux.gnu/bin/julia -e ""
  Time (mean ± σ):     211.0 ms ±  18.5 ms    [User: 208.5 ms, System: 106.5 ms]
  Range (min … max):   178.5 ms … 232.8 ms    12 runs

mkitti · April 29, 2024, 10:35pm

@vchuravy tried recently:

I’m not sure if piracy is a blocker in this case.

Benny · April 29, 2024, 10:40pm

I thought that import happened somewhere immediately because parentmodule(*, (Matrix{Float64}, Matrix{Float64})) in a fresh session returns LinearAlgebra, and it is in values(Base.loaded_modules). I just don’t know how to find it, I don’t know how to make and plot a modules tree and @edit doesn’t seem to track where else a file is included.

Palli · April 29, 2024, 10:53pm

I also did it in case you’re ok with slightly breaking:

github.com/JuliaLang/julia

Radically smaller sysimage; excise Pkg/REPL from it, and maybe LinearAlgebra (or at least openBLAS)

opened 01:16PM - 08 Aug 23 UTC

closed 02:18PM - 27 Oct 23 UTC

PallHaraldsson

performance

PR here https://github.com/PallHaraldsson/julia/pull/4, can skip to final commen…t, over (then) work-in-progress text, most of below here at the top may be invalid: When I comment out `:REPL` from sysimg.jl and build Julia (again) apparently nothing bad happens, and the sysimage still shrinks, and with it Julia startup, which is more-or-less linear in the size of the sysimage. Probably most of REPL is brought in (by why not all?) because of Pkg that depends on it. I can now excise Pkg, if I keep in all its dependencies, including :p7zip_jll, which is missing in sysimg.jl. I realize that excision alone isn't a big win (note, the REPL still operates so why is it there?), so I'm investigating excising more, e.g. the big-ticket LinearAlgebra, and others, only dropping it makes the sysimage 4% smaller. With it additionally commented out I can still start Julia and I don't run into trouble (it does break matrix multiply until I do `using LinearAlgebra` so such a simple change would be be a breaking change for 2.0, but I'm looking at what can be done in 1.x). Another way we could reduce the sysimage, is excise all doc/helps texts from it, 1.5% of it, 2.6 MB (and growing, possibly an overestimate of the gain): ``` $ strings -n 40 usr_lib_julia/sys.so |wc 38268 261726 2768701 ``` I _do_ get a smaller sysimage, with excising all but Pkg and LinearAlgebra, and while I think a PR could be made, it's only 0.05% smaller sysimage, though strangely 0.74% faster startup. The big, but breaking, change comes with excising LinearAlgebra additionally (I believe it can though be excised in a non-breaking but that's morework than I like to do, one way would be dropping BLAS and using naive matmul until `using LinearAlgebra`), or likely (and then non-breaking) if Pkg is excised. Or both, or some of Base additionally. I got a 1-161.2/174.0 = 7.4% faster startup (for min) with a sysimage just (or rather with it and what it may have brought in) with only Pkg in (I'm not fully sure of my timing, I thought I had disabled the web browser, but I had only susbended the wrong process, not the master process): ``` $ hyperfine './julia -e ""' Benchmark 1: ./julia -e "" Time (mean ± σ): 182.4 ms ± 16.7 ms [User: 113.5 ms, System: 69.0 ms] Range (min … max): 161.4 ms … 200.9 ms 14 runs ``` <s>I believe my machine is not loaded (web browser suspended), so variation in min and mean a bit worrying:</s> ``` $ hyperfine './julia -e ""' Benchmark 1: ./julia -e "" Time (mean ± σ): 199.0 ms ± 3.4 ms [User: 117.7 ms, System: 81.1 ms] Range (min … max): 194.2 ms … 204.1 ms 14 runs ``` Seemingly I can't excise Pkg, but I would really like to be able to (and its dependencies, e.g. Downloads, that I believe should go in 2.0, and strangely I can excise Pkg's dependencies). I'm unclear of why it fails (it seems to hang), while e.g. excising REPL is ok. With my past excision of I got: Base64 ─────────── 5.963757 seconds .. Pkg ────────────── 20.987424 seconds Note, these timings change depending on what else is excised. I'm currently excising _all_ but Pkg, which is the problem, including most importantly :LinearAlgebra which I think accounts for almost all the reduction in sysimage size, because in effect Pkg is holding the reduction back. The reduction, as is, isn't as large as I hoped for, what would it take to reduce to massively? I think Base could also be made smaller (in the sysimage, but I'm not sure how). Base 48.235718 seconds Pkg ─ 38.298616 seconds Stdlibs total 38.307447 seconds Sysimage built. Summary: Base ──────── 48.235718 seconds 55.734% Stdlibs ───── 38.307447 seconds 44.2624% Note, the timing for Pkg just goes up, the more I excise its dependencies, i.e. what I think is happening it forces them effectively into the sysimage. But I don't want Pkg in the sysimage (nor its dependencies). It's not needed most of the time. More radical, i.e. excising **ALL** stdlibs, meaning additionally Pkg, gets me a much better timing (for building) but doesn't work: ``` Sysimage built. Summary: Base ──────── 47.605037 seconds 99.9933% Stdlibs ───── 0.000005 seconds 9.92475e-6% Total ─────── 47.608228 seconds JULIA usr/lib/julia/sys-o.a Collecting and executing precompile statements └ Collect (Basic: ◒ , REPL 0/0: ◒ 382) => Execute ◒ 33┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] │ exception = ErrorException("Required dependency Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] failed to load from a cache file.") └ @ Base loading.jl:1986 └ Collect (Basic: ✓ 913, REPL 0/0: ◐ 402) => Execute ◐ 506┌ Warning: Failed to precompile expression │ form = precompile(Tuple{typeof(InteractiveUtils.include), String}) │ exception = UndefVarError(:InteractiveUtils) └ @ nothing nothing:0 └ Collect (Basic: ✓ 913, REPL 0/0: ◓ 402) => Execute ◓ 562 (1 failed)┌ Warning: Failed to precompile expression │ form = precompile(Tuple{typeof(Base.convert), Type{Function}, InteractiveUtils.var"#2#3"{Bool, InteractiveUtils.var"#8#22", String}}) │ exception = UndefVarError(:InteractiveUtils) └ @ nothing nothing:0 ┌ Warning: Failed to precompile expression │ form = precompile(Tuple{typeof(InteractiveUtils.define_default_editors)}) │ exception = UndefVarError(:InteractiveUtils) └ @ nothing nothing:0 ┌ Warning: Failed to precompile expression │ form = precompile(Tuple{typeof(Core.kwcall), NamedTuple{(:wait,), Tuple{Bool}}, typeof(InteractiveUtils.define_editor), Function, Array{String, 1}}) │ exception = UndefVarError(:InteractiveUtils) └ @ nothing nothing:0 ┌ Warning: Failed to precompile expression │ form = precompile(Tuple{typeof(Core.kwcall), NamedTuple{(:wait,), Tuple{Bool}}, typeof(InteractiveUtils.define_editor), Function, Array{Base.Regex, 1}}) │ exception = UndefVarError(:InteractiveUtils) └ @ nothing nothing:0 └ Collect (Basic: ✓ 913, REPL 0/0: ◑ 402) => Execute ◑ 659 (5 failed) ``` @KristofferC, It seemingly hangs there, so I can't excise all, and Pkg is the reason (not Base64 as I first thought), anyone knows why it might hang? I can actually try to excise Pkg, but keep some of its dependencies in: I'm now down to, more packages in, with better timing and hopefully smaller sysimage and faster startup (if I get to there...): Base ───────────── 47.120218 seconds Markdown ───────── 6.928026 seconds Base64 ─────────── 0.000132 seconds InteractiveUtils ─ 0.745049 seconds Stdlibs total ──── 7.682920 seconds Sysimage built. Summary: Base ──────── 47.120218 seconds 85.9757% Stdlibs ───── 7.682920 seconds 14.0183% JULIA usr/lib/julia/sys-o.a Collecting and executing precompile statements └ Collect (Basic: ✓ 718, REPL 0/0: ◒ 229) => Execute ◒ 477 Seemingly hangs there (at some other time I got hang at 500 not 477, in case it's helpful info). Which of these (and possibly more) are dependencies of Pkg?

It’s now 30.2% faster and 37.5% smaller

There numbers were without LinearAlgebra if I recall.

foobar_lv2 · April 30, 2024, 9:38am

But isn’t a big point for libraries that they can get mmapped by the OS?

Such that multiple julia sessions or programs that use the same library can all be served from the same physical memory, and such that they can be swapped out in a pinch?

DRAM is much more precious than disk / flash.

Additionally, as far as I can see UPX uses standard compression algorithms. I think the ideal way would be transparent filesystem-level compression (i.e. the kernel decompresses on page-fault), not userland shennenigans. This doesn’t work if special-purpose compression algorithms are needed, but UPX doesn’t supply them anyway?

Or am I seeing this wrong somehow? Is the point that users can’t configure their OS for appropriate disk compression so we must work around that?

Also, users in corporate environments will probably hate this (malware scanning / packer detection).

Palli · April 30, 2024, 10:06am

I just know special methods can be used for executable code, so I assumed UPX used those. Maybe it’s not its (main) point (could it be preprocessing for code, then use standard methods?), I see it e.g. has this option:

  --lzma              try LZMA [slower but tighter than NRV]

it does get me almost best compression, but then fails when running Julia:
 241252240 -> 163610692   67.82%   linux/amd64   sys.so  

Trace/breakpoint trap (core dumped)

Yes, I was mainly thinking compression on disk might lead to faster startup (not less RAM in the end), I think you’re saying the .sos are reused across different Julia’s, or well UPX would block that, block mmap. I don’t know, seems plausible (I took a look see mmap still in its source code).

Probably, if any “virus detection” false-positives, I’m just testing for now on Linux, and I think not a problem there, only on Windows? UPX states “any relevant Security/Antivirus software is able to peek inside UPX compressed apps to verify them”.

Palli · April 30, 2024, 10:48am

I did try rather to compress libLLVM, the next-largest .so and:

  94690784 ->  37315104   39.41%   linux/amd64   libLLVM-15jl.so

It compresses better but I get even worse startup at 589.2 ms and if the sys.so is also compressed I get 774.7 ms…

The fact that you can compress 1 or more .so files might be helpful to some, even more if there would be a way to decompress them concurrently, to at least limit the slowdown of startup to the largest .so.

Some issues and solutions:

Not all software is in a situation where VM page sharing between multiple-instances is an issue though, and some compressors allow skipping of shared sections. PECompact skips them by default, for instance. […]

As the author of a commercial PE compressor (PECompact), I can say that every packer should support DEP fully - including UPX […]

github.com/upx/upx

Windows 64-bit executable crashes with "Access denied" - Windows 10, UPX 3.91 and 3.94

opened 01:04PM - 05 Dec 17 UTC

apankrat

enhancement help wanted os:Windows

## What's the problem (or question)? Got a report from a customer, can't repr…oduce, but the symptoms are interesting. A 64-bit executable - C++, built with Visual C++ 2013, v120_xp. Been compressing it with UPX 3.91 since October 2013. The installation base is at least in 10s of thousands. Never had a single UPX-related issue reported until now. A couple of days ago a customer complained that the program no longer starts. The OS is Windows 10. Ruled out the program's own crash as a cause, also ruled out any antivirus interference (there are no 3rd party AV installed). The event log shows: > Faulting application name: ***, version: ***, time stamp: 0x5a1c2d50 > Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000 > Exception code: **0xc0000005** > Fault offset: 0x000000000028ae40 > Faulting process id: 0x3d85c > Faulting application start time: 0x01d36c3fe2462f40 > Faulting application path: *** > Faulting module path: unknown > Report Id: *** > Faulting package full name: > Faulting package-relative application ID: Switched from UPX 3.91 to 3.94 - the issue remained. Removed UPX compression - the issue went away. Also of relevance - this gentleman has been running an older version of the program (not dramatically different and also UPX-3.91 compressed) on the same machine and OS for over a year. I've asked for additional details, including if he perhaps was "tightening up" his OS setup in some form. Once I hear back, I'll update the ticket. I can ask for any details required and, probably, can ask to run diagnostics, collect logs, etc. Just need some guidance from you guys as to what to ask for. ## What should have happened? Program should've launched ## Do you have an idea for a solution? No ## How can we reproduce the issue? Can't tell ## Please tell us details about your environment. * UPX version used - 3.91 and 3.94 * Host Operating System and version: Windows 8.1 * Host CPU architecture: 64-bit * Target Operating System and version: Windows 10 * Target CPU architecture: 64-bit

Fixing this issue soon is unlikely. The priorities are Android shared libraries and iOS. UPX Team also has less experience and knowledge on MS Windows.
What could hasten a fix: a crisp identification of the problem(s). […]

Palli · April 30, 2024, 2:14pm

PCRE2 is only 0.7 MB, so not most important to get rid of for size reasons, but

is a much faster regex engine, so if someone wants to experiment getting it to work, then at some point it could replace PCRE2, or as a first step add it through a package, then PCRE2 can be dropped by default (with a possibility to enable it for compatibility if needed).