Does adding dependencies increase compilation time?

I am writing a Julia Package and I feel like compilation time of the package is longer than It was in the past. Does it have to do with the added dependencies?

Should I only import those functions that my package uses? Instead of using PyPlot, should I write import PyPlot: imshow.

I tried changing all using's for import's in my code and I ended up with conflicts between Gadfly.plot and Plots.plot. In these cases, the second imported function is just ignored.

I though that I could create a “minimal module” for each dependency, and import within these modules, to avoid conflicts in Main.

module minGadfly
    import Gadfly: plot
end

module minPlots
    import Plots: plot
end

Does any of this make any sense? I wanted to ask before committing to it.

Short answer: yes, adding dependencies will increase compilation time.

Unfortunately, trying to import only individual functions from a module won’t help. So doing import PyPlot: imshow won’t be any faster than using PyPlot. Folks with more knowledge of the compiler could perhaps give a more detailed answer but I believe it just boils down to the fact that a function in a given module might call other functions in that same module and it’s hard determine what exactly to compile when do using Foo: bar

However, it’s important to point out that I’m using the term “compile” pretty loosely above. The last paragraph is about what happens when you do using Foo or import Foo. That includes things like parsing of source code and lowering to an internal intermediate format but not the full pass through LLVM to generate machine code. When you actually call a function at runtime with concrete inputs, then julia can fully compile that function for the given input arguments. I don’t think that bit should be affected by adding extra dependencies.

On a final note, there’s no performance difference between using and import. The difference is just in how they bring names into scope.

2 Likes

I don’t think there’s any reason to expect this to have a performance impact.

You might, however, find the new @optlevel macro in Julia 1.5 useful: https://github.com/JuliaLang/julia/blob/master/HISTORY.md#new-language-features

Thanks for the answer! I don’t know that much about the compilation details of importing. However, I have tested that doing import Gadfly: plot does not suffice for running Gadfly.plot( Coord.Cartesian( xmin=0, xmax=1 ), more_arguments... ), because Gadfly.Coord needs to be imported explicitly. The compiler does not do it by itself.

Therefore, it seems that the compiler does not necessarily try to resolve classes that are used by the imported function. Your point about functions in a module calling other functions still stands, I don’t know how the compiler deals with this.

Thanks for the answer and the reference to @optlevel. I am still in Julia1.3 though, it’s a hard to keep up with the version updates :sweat_smile:

This is just a matter of Gadfly.Coord not being brought into scope. It’s orthogonal to how much work is being done by the using and import statements. I don’t have Gadfly installed right now but here’s an experiment you can do to demonstrate this:

@time import Gadfly: plot
@time import Gadfly.Coord: Cartesian

then restart julia and run the same two lines in reverse order, you should find that both times, the first line to run takes about the same amount of time and is much slower than the second.

1 Like

I see, I have a better idea now of what goes on under the hood ( and what doesn’t ). thanks

the experiment conforms to your expectations, by the way: 10s for the first import and 1s for the second, regardless of the imported members.