You should edit your post and add a link to the video.
Thanks. I just did.
Regards,
Jan
With respect, if static compilation/linking is the key feature
for someone, why would they want to use an essentially
dynamic language?
I will assume the question by Tamas_Papp to be a genuine one
rather than an arrogant top-down question belittling the use
case of people.
I will explain my own use case - this may be different to the
threadstarter, but remember the question here was posed
in a general manner aka “why would xyz want to use a dynamic
language”.
I am a heavy ruby guy. I wrote a “package” manager that is
actually a crappy build system; horrible quality. Don’t
anyone use it!
Anyway. I am tracking almost 4000 different packages so
far - glibc, gcc, binutils etc… - you name it. All of
KDE5, all of GNOME3, mate-desktop. Literally most of
the currently used software stack. And I keep on adding
to this.
I try to be independent, as much as possible, from
distributions - we could see the nightmare of IBM Red
Hat force-abusing everyone via systemd (and arrogant
maintainers such as on debian refusing to give people
a choice here, leading to the devuan-fork; anyway,
this is a separate topic. I wanted to give a background.)
The key here is that I am independent.
Now - I am stuck, though, because I need a clean environment
aka chroot. I can setup most of the chroot, but I do not
understand everything really. In particular there are
hardcoded paths but also glibc being annoying to no ends.
And … ruby is not working in the chroot. Which is
annoying. But a statically compiled bash is working as-is.
So I could use bash scripts. But I hate them.
Compiling a static ruby is difficult. It is easy with
mruby but … you need to know C. I don’t know C. I don’t
want to invest xxx hours into mastering C.
I just want my scripts to work.
As it is, ruby does not work, python does not work, and
getting lua to compile statically is ANNOYING to. So
I was looking whether julia fits the bill.
It would be simpler for me to re-create the CORE aspects
of my scripts (all the packages are actually stored in
yaml files, so I have the information as-is; I can
easily use another language too, which is different
if you look at e. g. homebrew) in another language.
So right now I don’t know whether I can use a minimal
julia, something simple, like --mini or --minijulia
and --enable-static or something like that. That would
be sweet.
No clue if that is possible, but my aim here was to
answer the above question.
So my use case is not related to distributing a static
binary to anyone - I just want my set of tools to work
as it is. (I already tried to make it simpler for ruby
to be compiled statically, and for python too but it
just does not work for me … bash works fine, grep
too, sed, awk … lots of packages work fine statically.
I only need it for the initial bootstrap-part by the
way. Once that works fine, I can bootstrap all the other
parts on a linux system fine.)
Especially if someone doesn’t already have julia installed,
installation times with download, precompilation, etc
easily range between 10-60 minutes right now.
There is no question that this needs to be addressed,
especially if we want to reach a wider audience
Different people have different use cases. IMO, ideally,
in particular the “scripting” languages should be as
flexible and dynamic as possible, to allow for people
to use them in many different scenarios. Sure, C and
C++ will always be faster, but I rather write scripts
in simpler languages, and maintain them, then go and
learn C and C++ properly because they are just less
fun compared to, say, ruby. Or python too. (I can not
use perl because I hate the syntax.)
By the way, for my use cases Go is not an option. I am
aware of many people jumping to the Google empire
but it is unlikely that I will - although this has less to do
with Google per se, and more that I don’t feel Go is
a “scripting” family really, whereas Julia fits a lot more
there, IMO.
There will be a talk about scripting in Julia at JuliaCon2020 (@fredrikekre might tell you more).
1 Like
I am actually kind of confused to what you’re trying to do. The original post is about statically compile Julia code and distribute an application written in Julia. Which is not what you’re trying to achieve.
If my understanding is correct, you’re trying to use Julia as a package manager, not as a package that is managed. (idk, something like apt-get except you build them from source?)
My confusion is essentially that, if your tool of choice lives on the host environment, why would Julia (or anything) needs to be statically compiled? If you just need some julia code to ‘just run’, use the official binary and run your core code once maybe? not sure if you have looked at Yggdrasil, which is a collection of scripts for building, well, softwares. Is this close to what your ruby manager does?
tl;dr - I understand if you need to build julia using some package manager, static linking can be a problem, but if you’re just using the software (given julia, python, or whatever) to write a package manager, I’m not sure what’s the difficulty truly is.
I have read through your post twice and still don’t understand your use case.
Also, please don’t revive topics 2 years later — a lot has changed in the meantime. If you have a question, just open a new topic.
Hey Tamas,
I wanted to answer your original question of why people might want to do static compilation of Julia programs. Julia is attempting to solve the two language problem. That is the same language for fast development and releasing fast performing code. Code that doesn’t need to be rewritten for a production environment.
In a production environment you typically want to release a statically linked binary. It’s simply a lot safer. That’s one huge use case for having support for statically linked binaries.
1 Like
Welcome to the Jula community. I think you’ll find what you need or want, with Julia as is or with non-default options (for e.g. binaries/hiding source code, while the binaries are currently non-small).
It’s just not true, that it’s safer (in general). You may want to provide binaries anyway, and it’s possible. PackageCompiler.jl helps, using it gets you a systemimage/binary, in some sense similar to Java, not fully compiled to assembly (while looks and feels that way), and like Java, I believe it’s as safe, safer than binary (statically-linked or not) machine code.
Static Linking Considered Harmful
http://web.archive.org/web/20080612143346/http://people.redhat.com/drepper/no_static_linking.html
From a security standpoint, static linkage has one advantage and one weakness:
[…]
Library injection requires local access, while known-address attacks tend to be remote, so on the whole, I’d say that static linkage of server binaries is a net negative to security, in addition to its other drawbacks.
1 Like
Aha we are talking about two different uses of the word safer. You are talking about safer from a security point of view. I’m talking about safety from a static environment point of view. In the view I’m taking, the completely static environment of a docker container would be a logical conclusion.
Deploying statically linked binaries in a standardized server/computing environment is an important and regular use case. It reduces dependencies and is comparatively much safer than a dynamically linked system.
I agree package_compiler.jl is part of the solution for this use case. I wanted to explain to Tamas, why people might care for package_compiler.jl in the first place.
You can use Docker with Julia, there are official Julia docker images out there (I’ve not used Docker, it shouldn’t require PackageCompiler to additionally use your Jown Julia code with?). But your claim vs. Julia’s Package/Artifact system may not be true. It’s at least supposed to nail everything down including all pure Julia code dependencies, or foreign language binaries (library or otherwise) used with (or I suppose Julia’s if wanted). There’s a loophole (related to old binary code from before the Artifact system, recent binaries should avoid that trap), not a big concern I think, but at least with Julia-only code it should be foolproof.
I think you mean easier for installation, less brittle, and that applies to many other languages, just to less or no extent to Julia. The Julia dependency system has been described as “a mini Linux distro” if I recall by Karpinski, and I e.g. noticed you can use it to install Perl (and Ruby), a specific version (Readme · Perl_jll.jl), that happens to be more up-to-date than the one in my Ubuntu 18.04. Not yet, say PostgreSQL (meaning the full database, the driver LibPQ.jl is there), and I see value in your logic for such dependencies (and config files for such), while you often wouldn’t have a database on the same machine.
There’s nothing inherently forbidding the system installing a full database (that one or at least SQLite, seemingly already done with JuliaHub as appropriate for an embedded database) for you. I’ve done for JuliaDB.jl, a relational database written in pure Julia (it’s just not an SQL one, is not meant to replace multi-user DB). Relational-AI also does something similar.
Deploying Julia applications in a self-contained manner and reducing compilation/startup time are both reasonable requests, and some form precompilation is part of the solution, but what people understand as static compilation/linking in C-like languages may be a red herring in the context of Julia.
Again, a lot has happened since the above discussion. Instead of reviving it, please start a new one, focusing on goals (“I want to deploy my Julia application, and this is whatI need help with”) instead of means (“I want static linking, whatever it means for Julia”). Cf XY problem.
7 Likes
Hey,
I wasn’t saying Julia doesn’t support static compilation. I was commenting why even in a dynamic language like Julia such a feature would be useful. I was justifying package_compiler.jl’s existence.
Having said that, you provided some interesting context and items to read, thank you.
I’m checking them out!
2 Likes
Yes, and no. If all your types are known you can compile to (even small, in theory) binaries, but in general you can’t! I mean, in general, the binary needs to include the compiler (and/or interpreter).
Think of the function:
mul(a, b) = a * b
Do you want to compile only for the integers, or floating point, or string concatenation or the infinite possible types, as Julia is generic? In general you could @eval at runtime, and solving this runs into the halting problem, theoretically not possible. Restricted Julia without being limited to being generic is still as powerful as e.g. C (you could e.g. type all your functions).
You can solve it by defining the elements at the entry point to the function in the library.
For instance, I create a Julia function as entry point and define its entry point are 2 double arrays and 2 scalars parameters.
From there I’d like the Julia Coder to infer all the elements down the road.
If there is an ambiguity somewhere, and it can infer the options which are limited, then do its magic. If not, return an error and ask the user to define those specific cases.
It might require some decorations or specific style of coding. But I think it is doable (Or at least this is not the main issue, no real need to support all cases or add the compiler).
3 Likes
Regarding generating small self-contained binaries, you might also want to check @tshort’s early experimental work: GitHub - tshort/StaticCompiler.jl: Compiles Julia code to a standalone library (experimental)
which is an interesting tree-shaking approach IIUC. I guess this is in principle possible for all type-stable functions.
5 Likes
StaticCompiler.jl is much closer to what @Mayank_Gupta is describing, although dynamic linking still occurs, but it’s much closer to a standard C program in that there is no JIT. However, if you had the Julia code to replace libc, BLAS, and whatever else your code pulls in, you could technically compile a fully-static Julia binary with StaticCompiler, with a bit of work.
@RoyiAvital is correct that in order to be able to do this, you’ll need to define all “entry points” with their input types, and ensure that you: never do dynamic dispatch, replace dynamic dispatch with if-else, or implement a custom dynamic dispatch that doesn’t use a JIT. This is really the fundamental problem that needs solving, since dynamic dispatch comes up in a lot of places. Exceptions are also a similar problem that need to be solved by some similar mechanism, since they require allocations and non-linear control flow. Solving these sorts of problems would probably require generating a custom Julia runtime library that implements the basics of what the user’s program requires.
1 Like