Reluctance to switch to Julia; PyPy and Cython

[Liso]

Could you (or anybody else) explain in example how could changed context invalidate precompiled file please?

[antoine-levitt]

Precompiling on add is typically the kind of things that would be good to expose as a global setting people could set in their .juliarc.jl.

[StefanKarpinski]

# package A.jl version 1

module A
    export x
    x = "one"
end

# package A.jl version 2

module A
    export x
    x = "two"
end

# package B.jl version 1
# requires A (any version)

module B
    export y
    import A
    y = A.x
end

What is the value of B.y for B version 1?

[Per]

What if Julia automatically precompiled packages in the background? This could happen in a separate process and only run when the Julia REPL is idle.

[Elrod]

What about people using laptops?
If someone on a laptop has a lot of packages, and updates regularly, they may then suddenly realize that running Julia rapidly drains their battery, while regular internet browsing, R, and Python do not.

While that would be cool, I’d still prefer it to be optional.

I also feel like automatically re-pre-compiling with every update sort of discourages installing or trying many packages. Every package you install adds to that compile time, whether or not you’re using it regularly. Although (judging from my colleagues using R), most users may almost never actually update packages, so that isn’t a major concern for them.

[Per]

People that are computer savvy enough to use Julia are probably aware that running any kind of update will consume energy. But yes, of course there would be a possibility to opt out.

[tamasgal]

I have to comment on this quickly.

@Thomas_Benardo

• {limited} Python can be statically compiled using shed-skin(faster than Julia, almost anytime)
• {fully-compatible} Python can be compiled using Cython for a enormous speed up(faster than Julia if used wisely)
• {limited} There is RPython metacompiler which is also very fast(comparable to Julia)

Thomas, I have no idea how you did your measurements, but I spent a few years of my life (fine)tuning and optimising Python code for a neutrino experiment and no Cython or PyPy approach could compete with the C++ counterpart anyhow (we have an expert team for both worlds). The only bits of modules which came near to the C performance were the numba JITted functions in Python which completely avoided the usage of dicts (not supported in numba, thus no lookup tables like std::map! this introduced very ugly coding practices) or any custom classes. We were basically writing algorithms like in Fortran…

In Python you simply have so many caveats. Also the performance of PyPy was more close to Stefans 1/5th of C.

We started to use Julia a while ago and it was unbelievable how you had almost all the time C speed without thinking to much about implementation details. All this using our own custom types…

That’s it.

[Tamas_Papp]

I think that there are at least 3 issues mixed up in this topic, namely

1. delivery of some project as a self-contained application,
2. reducing compilation time for predictable code paths,
3. reducing compilation time in general.

In traditional statically compiled languages, 2 and 3 are mostly the same, and 1 is trivially available, and all the solutions involve compilation.

So it is somewhat understandable that potential users think that (pre)compilation is the solution to their problems in Julia. Understanding the implications of AOT, multiple dispatch, and parametric rypes even at the user level takes time.

[Elrod]

That would rule out everyone else in my graduate program from potential Julia-users.

[Liso]

Couldn’t be this problem of linker instead of (pre)compiler?

I am thinking about something like:

ccall((:sqrt, "libm.so.6"), Cdouble, (Cdouble,), 4)

where we don’t need to recompile libm.

If it is problem to change language - we could probably have in intermediate time something like this? →

module B
    export y
    import A
    y = jucall("A.ji.1", :x) 
end

[dfdx]

At the very least late compilation lets compiler to do more optimizations such as constant propagation or function inlining. In fact, the later you compile, the more information you have and the more optimized code you can get.

What I’m missing in this thread is why compilation on the first module load is a problem at all. I can get argument about code obfuscation, but it doesn’t seem to be your concern, right?

[MA_Laforge]

Re: Precompilation:

@Tamas_Papp: I think that there are at least 3 issues mixed up in this topic, namely […]

There is at least another very important reason people want pre-compiled code: Not wanting to give away the IP that brings you revenue.

This might not be obvious to those trying to solve complex scientific problems. However: I can see that many people would really like to use Julia to develop their own solutions, but don’t see it as a real possibility because if you have to distribute the code - it is too easy to steal.

All the big companies I know of use managers like flexlm to control software use - because they know that people will inevitably “steal/abuse” licenses if they don’t do this… And surely even these solutions do not completely eliminate “theft”.

If your solution requires you to distribute code, it is almost guaranteed that it will be stolen… presumably in a big way. Not everybody honours the licenses attached to your code.

A possible solution?

Mabe a solution for those people would be just a bit of code lowering/name mangling/“mechanization”… Basically making code impractical for humans to read… but also difficult for a program to restore to a human-readable state.

Basically, you would want your distributed code to be as difficult to reverse-engineer as possible.

I just don’t know if this obfuscation would be simple enough to warrant its own implementation, or if we are just better off putting energy on the real compilation module.

[MA_Laforge]

FYI: I think there is one aspect of Julia that is significantly understated in the Julia vs Language X debate: We talk about individual features, but we don’t really emphasize how quick and natural the development process itself really is.

I noticed @ChrisRackauckas does indeed mention (implicitly) that one of Julia’s advantages is that it has “generics”, and a “parametric type system”… but what’s missing seems to be the difficulty to convey how much this really helps.

It is a bit blurry for me now since I have not really been developping in other languages for a while, but I do somewhat remember when I started using Julia:

• You can build complex Vectors & Arrays of pretty much anything you want - without even thinking about it (little wasted time/effort).
• You can perform complex mathematical operations on relatively complex data structures… without getting lost in the details of it.
• You can use Dicts, parametric keywords, and optional keywords without a huge overhead of setting them up.
• You can splat a Vector or Tuple as an array index instead of looping through each index to create a subarray, etc.
• You can use Strings seamlessly, and perform interpolations to make your code more readable/tractable even at the time you are writing it.
• A side effect of multi-dispatch means you don’t have to pause to come up with a naming scheme when you overload functions: funcF32, funcI8, funcF64, …
• There is very little overhead in creating functions (ex: f(x)=3x). You often don’t even have to translate your thoughts from “math” to “computer”.
• There is just so little overhead in creating a new type. I really can’t explain what it is about this… It just feels alot simpler than other languages. I don’t think it is all in type creation either - I think it is how you use it/extend it.

The worst part (to me) is, I know there are plenty of other tiny little advantages which I am forgetting (because you just don’t have to think about them). And yes, individually these aspects are not that important, but together, these tiny little advantages make a huge difference when you are developing with Julia.

Of course, it is not really just that these tools are built-in. The reason why Julia is so easy to use is because all of these built-in tools derive from very simple, very well thought out constructs that just work the way you would expect (or at least better than the other solutions I have used).

And I know Python does implement some of these tools, and does somewhat of a good job, but Julia really takes it to a level where things are noticeably more harmonious.

…So please: let’s not underplay what an important role this entire infrastructure has to the ease-of-use & “speed boost” you experience during the development process itself. Julia makes it easier/quicker to build throw-away code to mock up your ideas, thus enabling you to experiment more. It even makes it faster to get fully up & running than with anything else I have seen.

[StefanKarpinski]

@Liso

where we don’t need to recompile libm.

This is why I said that precompilation is not like separate compilation of shared libraries in C.

[ChrisRackauckas]

You may want to read an in-depth discussion on how Julia tackles a much larger problem space here:

Why Numba and Cython are not substitutes for Julia - Stochastic Lifestyle