Compiling and building binaries from your julia code

General Usage
41

I’m also interested in real-time environments (for live audio processing), but it’s a tough nut to crack.

Even if your code is compiled statically it doesn’t change the memory allocation/GC model, so the code will still not be real-time safe. If you can get away with “mostly safe” then you can get pretty far by avoiding as much heap allocation as possible so the GC doesn’t have much to do.

In ages past there was some exciting work towards making the GC incremental with a time-bounded pause times, but I think it was never ready for prime time and didn’t get merged.

42

I think I could get away with “mostly safe”, certainly if there was a path towards stronger safety guarantees down the road. I haven’t read anything about Julia yet that indicates that it shouldn’t be possible in principle to have some subset of Julia programs that could be real time safe, in some future hypothetical version of Julia.

43

I also noticed that it fails when using some packages (e.g. Gtk compiles but segfault, Cairo doesn’t compile, …) do we know the potentials reasons why some packages fail ? I guess calling external libraries in particular can go wrong in several ways. Or in other words if I want to make Cairo compile-compatible, what should I look for ?

44

I agree, and there have even been recent changes that allow more things to be stack-allocated, further reducing the GC pressure.

45

It will have lesser startup overhead, but for eliminating most of the startup overhead, we will need something like Tim Holy’s SnoopCompile. Jameson recently fixed it to work on 0.6/0.7 - so that would be the thing to further investigate. Completely dynamic things will still invoke the jit, but the goal post 1.0 should be to have tools that help understand these things and simplify them.

GC will be present. The much longer term plan should be to have either a DSL or a restricted subset of Julia for real-time controllers. I think it would be a great research project at a university, or for someone with the right skill and enough time to put into something like this. There have been many discussions (offline), but nothing that has been written up.

Threading in Julia is still early, but it should work in these binaries as expected. Same for distributed computing stuff. It will be a bit more work to build further tooling to make all this easy - but if it doesn’t work, it would be good to get issues on the static-julia repo.

-viral

2 Likes
46

Ability to deploy a package binary at some point is very important, as well as an ability to write Julia in such a way that all inferences about the code structure/types can be made at compile time. The latter will also enable the same or better experience that users of “Intellisense” and C# enjoy. Please please make sure that everything needed will be baked in 1.0 :slight_smile:

47

I just recently put snoopcompile and the static-compilation script into one package: https://github.com/SimonDanisch/PackageCompiler.jl - so that should make things already a bit easier.

@jonathanBieler

After making a fairly large package (Makie) and it’s dependencies precompilable, I came up with the following list:

  • non const globals are problematic, or globals defined in functions - removing those got me to 90% of making the package save for static compilation

  • type unstable code had some inference issues (around 2 occurrence, where I’m still not sure what was happening) - both cases happened with dictionaries… Only way to find those was investigating the segfaults with gdb, but then it was relatively easy to just juggle around the code a bit. - The non const globals might be related since they introduce type instabilities, not sure though…

  • some generated functions needed reordering of the functions they call ( actually, even for normal code all functions that get called in a generated function should be defined before it)

  • I uncovered one out of bounds issue, that somehow was not coming up without static-compilation

  • I used julia-debug to uncover those bugs, but actually, the last errors I was trying to uncover where due to using julia-debug - so Makie just works correctly statically compiled with a release build right now!

  • you’re pretty much on your own and need to use gdb to find the issues and I still don’t know what the underlying julia issues are and when they will get fixed :wink: See: https://github.com/JuliaLang/julia/issues/24533

20 Likes
48

I am so excited to try this over the weekend. Thank you very much for this effort!

49

The solution:

  • use the mingw-w64 compiler from http://mingw-w64.org/doku.php/download/mingw-builds with default settings
  • set the PATH to cc = is_windows() ? raw"c:\Program Files\mingw-w64\x86_64-7.2.0-win32-seh-rt_v5-rev1\mingw64\bin\x86_64-w64-mingw32-gcc.exe" : "gcc" (I’ve used the absolute path purposely); or set system PATH
  • copy all required *.dll files to the hello.exe folder from c:\Program Files\mingw-w64\x86_64-7.2.0-win32-seh-rt_v5-rev1\mingw64\bin\ and YOUR_JULIA_INSTALL\Julia-0.6.0\bin\
  • run the hello.exe in Unicode friendly console; I use Visual Studio Code
shell> d:/work/J-CPP/static-julia-master/builddir/hello.exe
hello, world
f() = 0.369760359689711
       ┌────────────────────────────────────────┐
   100 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠎│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⠁⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠊⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠔⠁⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠔⠁⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠔⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠤⠒⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡠⠔⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       │⠀⠀⠀⠀⠀⠀⢀⣀⠤⠔⠊⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
     0 │⣀⠤⠤⠔⠒⠉⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
       └────────────────────────────────────────┘
       1                                       10
3 Likes
50

Please notice that static-julia was merged with PackageCompiler.jl, any further development will happen there.

6 Likes