Compiling and building binaries from your julia code

I put together a repository on how to compile binaries and shared libraries from your user code - with lots of help from Jameson.

https://github.com/JuliaComputing/static-julia

The instructions are in the README, and are hopefully straightforward. The juliac.jl script is also simple and easy to inspect and modify.

https://github.com/JuliaComputing/static-julia/blob/master/README.md

The hello.jl in this repo gets compiled into a hello binary, and can be executed like any other OS binary. I have tested all of this largely only on Linux at the moment, although with little tweaks it should work elsewhere.

My goal is to announce it here, get some feedback, and eventually get a juliac script included in the default julia distribution as the way to compile julia programs for various purposes. Also, I would love to have the documentation in the manual, and some form of CI (although build times are fairly large).

While the pieces have always been there, I feel that a complete example has been missing. Hope this helps.

43 Likes

Thanks for shaing @viralbshah! This project can be very useful in certain situations.

We should probably link here from some of the many older threads where people asked for static compilation.

4 Likes

This could be great! I opened two issues while playing around with it:

Last yearā€™s Static and Ahead of Time blog post mentioned:

However, once the user defines a local scope (for example, a function definition, let block, for loop), only static constructs can be used, with three exceptions provided for user flexibility: eval, macros, and generated functions. This is important, because it means that if the programmer avoids using these three dynamic constructs for the application logic, it is possible to statically analyze and compile the program generated as a result of running the userā€™s code file.

Iā€™m assuming these limitations still apply? If so, that should be noted clearly in the readme.

While this is based on the same work, this is not for AOT compilation by going through C. For example, I have included Gadfly and thus all its dependencies in hello.jl. Thus, I donā€™t think these issues apply.

The issues that are not covered in the script so far are bindeps related. Packages that may need shared libraries will need to be provided separately on LD_LIBRARY_PATH or equivalent. We may need some work to conveniently package it altogether eventually. Maybe even adopt the whole Electron framework to build complete applications with UIs and installers and such. One step at a timeā€¦

2 Likes

Oh, I see, I was confused. Thanks.

Seems to work just fine now for some test code involving my package, subject to the issues I opened and the bindeps caveat.

So how does this relate to https://github.com/dhoegh/BuildExecutable.jl?

2 Likes

It is no longer using the userimg.jl method and compiling a system image by recompiling all of base. You can see how the script works, and I feel it is more robust. The rest of the infrastructure in BuildExecutable.jl should be reusable and this can be fit within BuildExecutable with very little effort by someone who knows it well. In fact we should probably at least have an issue there.

-viral

1 Like

Blockquote[quote=ā€œviral, post:1, topic:5687ā€]
I put together a repository on how to compile binaries and shared libraries from your user code - with lots of help from Jameson.
[/quote]

Should we be concerned about the input of your helper :wink:

healyp https://discourse.julialang.org/u/healyp
September 3

Blockquote[quote=ā€œviral, post:1, topic:5687ā€]
I put together a repository on how to compile binaries and shared libraries
from your user code - with lots of help from Jameson.
[/quote]

Should we be concerned about the input of your helper :wink:

Yes

Thanks for this!

Iā€™m trying to sell Julia use for a project, and this will be helpfulā€¦particularly if Iā€™d be able to compile a reasonably sized (non commercial) shared library callable from python and matlab. Would that be possible with this method?

Yes, it should be possible to call the library from anything that can call a C function from a shared library.

-viral

1 Like

Hi,
does the workflow expect some preparation steps before running the juliac.jl script? Iā€™ve tried to compile the example exactly according to the readme instructions, but failed because of missing sys.so library (and may be lotā€™s of other prerequisities that didinā€™t run at all).
Iā€™m using the binnary win64 JuliaPro 0.6.0.1.

c:\Users\phlavenk\Documents\Python_Scripts\static-julia>c:\JuliaPro-0.6.0.1\Juli
a-0.6.0\bin\julia.exe juliac.jl hello.jl c:\JuliaPro-0.6.0.1\Julia-0.6.0 c:\Juli
aPro-0.6.0.1\pkgs-0.6.0.1
Program File : hello.jl
Julia Install Path: c:\JuliaPro-0.6.0.1\Julia-0.6.0
ERROR: could not load library "c:\JuliaPro-0.6.0.1\Julia-0.6.0\lib\julia\sys.so"

The specified module could not be found.

ERROR: LoadError: failed process: Process(`'c:\JuliaPro-0.6.0.1\Julia-0.6.0\bin\
julia.exe' -Cx86-64 '-Jc:\JuliaPro-0.6.0.1\Julia-0.6.0\lib\julia\sys.dll' --comp
ile=yes --depwarn=yes '-Jc:\JuliaPro-0.6.0.1\Julia-0.6.0\lib\julia\sys.so' --sta
rtup-file=no --output-o hello.o -e '
         vers = "v0.6"
         const DIR_NAME = ".julia"
         push!(Base.LOAD_CACHE_PATH, abspath("c:\\JuliaPro-0.6.0.1\\pkgs-0.6.0.1
", "lib", vers))
         include("hello.jl")
         empty!(Base.LOAD_CACHE_PATH)
         '`, ProcessExited(1)) [1]
Stacktrace:
 [1] pipeline_error(::Base.Process) at .\process.jl:682
 [2] run(::Cmd) at .\process.jl:651
 [3] compile(::String, ::String, ::String) at c:\Users\phlavenk\Documents\Python
_Scripts\static-julia\juliac.jl:27
 [4] include_from_node1(::String) at .\loading.jl:569
 [5] include(::String) at .\sysimg.jl:14
 [6] process_options(::Base.JLOptions) at .\client.jl:305
 [7] _start() at .\client.jl:371
while loading c:\Users\phlavenk\Documents\Python_Scripts\static-julia\juliac.jl,
 in expression starting on line 61

I opened an issue: https://github.com/dhoegh/BuildExecutable.jl/issues/45.

This is great!

@viralbshah can you please comment:
How is your experience with it , with regard to load time and first execution time?

Can it e used as the system image of a new Julia session?

Given that there has not been any response to the BuildExecutable.jl issue and the fact that the last commit to that repo by the original author was in February, I think it might be worth exploring other options for turning ā€˜static-juliaā€™ into a real utility.

I actually think that having this as a well-supported feature could be a sufficiently big draw for new Julia users that it might be best not to rely on package authors external to Julia Computing.

I think itā€™s because Windows is not supported yet (it should look for sys.dll rather than sys.so). I really hope this script will support Windows soon.

Yes, that is correct. On windows you would need a dll, while on linux so would do.

Ok, I found out how to compile on Windows.

Change line 9 of juliac.jl from:

SYS_LIB = joinpath(julia_install_path, "lib", "julia", "sys.so")

to:

SYS_LIB = joinpath(julia_install_path, "lib", "julia", "sys.$(Libdl.dlext)")

then change line 31 from:

run(`x86_64-w64-mingw32-gcc -m64 -fPIC -shared -o $(SO_FILE) $(O_FILE) $(ldflags) $(ldlibs)`)

to:

run(`x86_64-w64-mingw32-gcc -m64 -fPIC -shared -o $(SO_FILE) $(O_FILE) $(ldflags) $(ldlibs) -Wl,--export-all-symbols`)

Now everything compiles and I get hello.o, libhello.dll and hello.exe, however unfortunately the executable does not print anythingā€¦ any clues?

(Iā€™m using Julia version 0.6.0 and x86_64-w64-mingw32-gcc version 7.1.0 x86_64-posix-seh-rev0)

Thanks

3 Likes