Tiny executables with all capabilities. was: PackageCompiler.jl a bad name(?) And why can't modules and scripts not be compiled?

Community
, ,
1

People want compiled binary executable for a few reasons, some good, some bad:

  • To distribute just one file, already possible, but it’s not small. It can also be tiny with my idea.
  • To have everything needed in that file, and it being an .exe. There’s strictly no need that everything is in that file, it just needs to feel that way, give that illusion.
  • To hide (all) the source code. Already possible. Would still be possible. Or you could choose to have source and/or binaries for some files, there’s really no need to hide the source code of e.g. open-source dependencies.
  • To include all dependencies in that one file. That’s just something NOT needed, a (bad) convention, not shared by executables in all operating systems, nor for all on the same operating system, e.g. on Windows.
  • Not need an internet connection after you get the file. Do we really still live in the floppy disk world?

What I propose is that every .jl file you run compiles to .ji binary file, and if you run a .jl file then the cached code in .ji file is used if the timestamp is newer. If there’s a missing corresponding .jl file then it might be an error, maybe be default, or not.

You still get lots of files this way, you could distribute any combination of .ji, or jl files, or both (for all files e.g. if you use GPL) for your whole project. Julia would provide an archive possibility into one file, similar to "[Windowed] Python ZIP Applications”, i.e. .pyz and .pyzw extensions. Are those common? They were new to me, so I thought not.

You COULD if you wish distribute source code and/or binaries of your dependencies, if you wish, e.g. if you think they might go away. On of those dependencies is Julia itself, and it may be the largest one. It’s not hugely important, even much better to skip Julia; and all other dependencies, for smaller files.

But then your one Julia “executable” is not complete.But who cares? You distribute the Manifest.toml file with so you know all of them, and SHAs of all of them. You can download them from all the relevant places that have them available, or if you’re paranoid, from your own website. You need to do that, but only on first use.

Instead of using .jlz similar to Python, which isn’t a registered file type, nor Python’s file types, so you can’t run the file… it will be a self-extracting .exe file, which means in practice a standard ZIP file, but with x86 prefix code to self-extract AND additionally download all binaries. That file could be recognized even on e.g. Linux, ignoring the header (somehow), and if would download everything for Linux for the dependencies. And recompile your code (would assume you do not strip out your source code; theoretically, well very much in practice too, it could just emulate the x86 code).

Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6

.pyz and .pyzw extensions as “Python ZIP Applications” and “Windowed Python ZIP Applications”, and providing some simple tooling to manage the format.

Python-Version: 3.12

When should extensions be linked against libpython?

Short answer
Yes, on Windows. No on POSIX platforms, except Android, Cygwin, and other Windows-based POSIX-like platforms.

Old post:
I felt I needed to explain the name PackageCompiler:
https://www.reddit.com/r/Julia/comments/15x1u10/comment/jxp4cx9/

@kristoffer.carlsson
I don’t know the full history of it, but would you consider renaming the package, may just to Compiler.jl?

What do people think, are people put off by the name, or overlook it, since it implies something it doesn’t (only)?

By now as of Julia 1.9 packages are actually (pre)compiled by default. There’s not huge need for a tool with a name doing that only, or implying it does it only.

What people want is an [app]compiler, i.e. the already existing one.

What I would want for Julia, not an external package, is for even scripts compiled to .ji, as I explained at Reddit. Any reason why it wouldn’t be an easy addition?

1 Like
2

I do work with machines that don’t have internet connection. I deploy long running simulations on these machines.
These machines have a different OS than the one I use. While I can install Julia from one ma hine to the other, I can’t install anything that require an internet connection.

If I could cross compile on one machine and deploy on an another one, it would make my life easier.

For some (aerospace, military grade embedded system…), disk space and memory is a luxury, so if the binaries are not small…it is a no no.

Dll are also important if you want to extend existing applications or libraries written in Python, R …etc. Some libraries have a lot of these dlls, so it is important to be able to update 1 dll without having to update all the others.

And if you want to distribute them via package manager, server…etc in remote location with not so great connection…then small binaries are preferable.

1 Like
3

Well, I still use PackageCompiler mainly do compile a system image that contains all packages I am using… It cuts down startup time for me from 20s to 5s… With Julia 1.10.0-beta I still have a startup time (including package load time) of 10s. So for me creating a custom system image with PackageCompiler is still worth it, and the name fits.

It can also compile apps, but I dont’t think renaming it to app compiler is worth the effort.

1 Like
4

I think most of this is on the roadmap of Julia, but it will require years of development to get closer to this goal.

Some things are already possible, e.g. small binaries if you are not using the Julia runtime: GitHub - tshort/StaticCompiler.jl: Compiles Julia code to a standalone library (experimental)

5

What I’m trying to solve, and my proposal would(?).

That doesn’t apply to most, so a little inconvenience for those users seems ok. You would just run your program on a different similar (or if not cross-compile) machine to get all the .ji files, and you could simply copy over from that one with s/cp (and packages such into one file by default would also be ok, just no longer a tiny binary). If you only have .jl files and all of them, then it would also work out of the box, and still be a small download.

Do you need small AND precompiled on first use (for all the included code)?

DLLs are an implementation detail, also only on Windows. I’m for now mainly thinking of the general thing, Julia-only code, and then you would get .ji files, DLL substitute. There’s nothing inherently wrong about bundling DLLs too, solving the library problem, to allow other languages to call Julia without its runtime, but not a priority for me, there’s nothing wrong for other languages e.g. Python to call Julia and its runtime. The main problem with that approach was bundling the (relatively) huge Julia runtime (which is the same for all .exe, otherwise, so why duplicate it in all?), but you don’t need to do it for every app download.

You can still keep on using PackageCompiler.jl, with that name. My first post was about it and the name, changing it, not the functionality. Then I changed my original post, and now it’s about changing Julia, you would get the c. 20s startup time, on first use, then 5s or better on all later without messing with PackageCompiler.jl (you could even mix and match the approaches if needed, with a custom sysimage, but I’m not concentrating on that since not needed), just with a default (new and upgraded) Julia. Or at least I think it’s very possible. StaticCompiler is trying to get rid of the runtime, for tiny binaries, and to make libraries. It has many limitations, but my approach has none really if you have internet connectivity, and I think would take far from one year to implement.