I am new to Julia and I love it. However, there are several issues I cannot fix.
At the moment this is the most annoying one:
I decided using Julia because of its infamous speed.
I’m writing scientific software which aims at solving large sparse eigenvalue systems.
For that reason I end up with very large *.jl files. By large I mean several kB up to a few MB file size.
Though these files are lengthy they basically define a long chain of matrix-vector -products and vector-vector-additions.
The key point here is that compilation takes very long while using the compiled function is actually fairly quick.
In order to avoid Julia compiling the function any time a new workspace was set up, I wrapped the code in a module contained in a custom julia package.
The module structure is kind of nested. It looks pretty much like this:
__precompile__()
module mymodule
__precompile__()
module submoduleA
#some code
end
__precompile__()
module submoduleB
importall ..submoduleA
#the very very long bunch of code
#containing the function which is crucial to be compiled
end
end
Now, when I load the package by “using” or “import” etc. after I did some changes to the source code, there appears a notification in the console that the package is going to be precompiled and a corresponding .*ji file is created in ~/.julia/lib . Seemingly, this is exactly what I want. However, when I import the relevant function, the first time it is invoked takes several minutes. Obviously, this is due to compilation. A behavior I intended to avoid by declaring the modules as precompiled.
What is it I am doing wrong?
Some more details:
I am using Julia 0.6 on an 64-bit ubuntu 17.04 system.
The large files are algorithmicly created by an auxiliary julia function which automatically generates the necessary julia code and saves it to a *.jl file. I know that julia has meta-programming features which should be able to do the same job without creating a lengthy source code file explicitly. However, as I am a beginner the current approach is more convenient for me.
Try giving it a precompile hint for the function you call. However, this may not solve your issue as per-compilation stops at package boundaries. Thus, for instance all Base functions which your function calls will not get compiled (some are of course already compiled).
I will definitely start using the Revise.jl-functionallity to improve my workflow.
However, the code I wrote is meant to be shared with some colleauges (and finally the entire julia community).
Maybe you could outline, what you mean by “giving a precompile hint”. I thought this is what one is doing by stating __precompile__() before the declaration of a module.
From what I got from the documentation precompilation is a pretty basic idea in julia. I am an early adopter of julia in my department. For me it is, therefore, pretty important to keep things as simple as possible. Otherwise I’ll trigger the I-told-you-so-attitude of my colleagues by sending them code together with a long list of additional packages to install and actions to be taken, just in order to keep the code running.
I’m definitely not an expert, but I can at least provide some direction. First of all, I’m pretty sure you only need __precompile__() once at the top of your file. But the reason your function is still slow the first time is that __precompile__() does not fully compile your module, rather it instructs Julia to parse and lower the code and cache that result. But the final steps of compilation (like actually generating machine code) still happen when your function is actually called. That’s what you’re seeing. If you want to know everything there is to know about this stuff, I’d suggest Jameson’s talk from this year’s JuliaCon: JuliaCon 2017 | AoT or JIT: How Does Julia Work? | Jameson Nash - YouTube
It is definitely possible that using metaprogramming to generate only the specialized code that you need for a particular function call, rather than generating megabytes of Julia code manually, may help reduce your time spent parsing, lowering, and compiling code. In particular, @generated functions are designed to help with tasks like unrolling loops and generating optimized code for a very particular set of inputs. They’re used in packages like StaticArrays.jl to create optimized code on-the-fly for arrays with known sizes.
Precompilation only pre-compiles functions which are either executed or which are hinted with precompile:
__precompile__()
module A
f(x) = 1
g(x) = 2
g(5) # this will get g compiled for input type Int
precompile(f, (Float64,)) # this will get f compiled for input type Float64 (without executing the function)
# and, for example, f("a") is not precompiled at all.
end
Concerning my statement “per-compilation stops at package boundaries”, consider
__precompile__()
module A
function f(x)
a = x+1
return sin(a)
end
precompile(f, (BigFloat,)) # this will get f compiled for input type BigFloat
end
This will only pre-compile the part of f which belongs to module A. But for instance the sin will not get compiled for arguments BigFloat, as sin is part of another module, namely Base. Thus if most of the total compilation time is spent on compiling functions from other packages then __precompile__ is not helping.
a long list of additional packages to install
You should make your code into a Package which has a REQUIRE file in its top directory. When installing this with Pkg.clone("git...") it will automatically install the packages and its dependencies listed in REQUIRE (note though, that REQUIRE can only install officially registered packages, but that should cover 9 out of 10 packages).