Will Julia ever fix its "using ..." latency problems?

General Usage Performance
1

The “using …” command is extremely slow:

julia> @time using Plots
12.731850 seconds (17.25 M allocations: 976.326 MiB, 3.64% gc time)

julia> @time using DifferentialEquations
34.780873 seconds (41.90 M allocations: 2.456 GiB, 2.63% gc time)

julia> @time using Atom
3.137487 seconds (2.95 M allocations: 184.199 MiB, 1.27% gc time)

This makes Julia nearly unusable. Does anyone know why Julia is so much slower here than other languages and does anyone know if this will ever be fixed?

I thought julia wanted to be faster, not slower than Python.

3 Likes
2

What version of Julia are you on? 1.5 is about 2x faster than 1.4, and 1.6 is much faster than 1.5.

1 Like
3

$ julia --version
julia version 1.5.3

4

You can alleviate some of this using Revise.jl and create a custom sysimage with PackageCompiler.jl

https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/

This issue is known, we have seen significant improvements over the last few versions.

4 Likes
5

To be fair you’ve picked out like the worst of the worst here. Most packages have rather small load times.

8 Likes
6

On 1.6 (not yet released) I’m getting 2.6 and 4.5 seconds for Plots and DifferentialEquations. So this is still not perfect, but is much better than 1.5.3. 1.6 should be out soonish (December or January)

7 Likes
7

This is a well-known problem, and there have been dozens of commits to address the root issue. As a result, there’ve been significant improvements since v1.4:

julia> @time using DifferentialEquations # v1.4
 42.003165 seconds (70.47 M allocations: 3.891 GiB, 3.65% gc time)

julia> @time using DifferentialEquations # v1.6
 10.538916 seconds (14.14 M allocations: 1.121 GiB, 4.64% gc time, 27.26% compilation time)
5 Likes
8

Do you know why julia is slower here than other languages?

9

This is news to me, I use Julia every day.

Package loading times are being actively worked on as others have mentioned.

If you care about startup time instead of runtime, then indeed you should use Python. If you later decide you want to do something with those packages after you load them, Julia will be waiting for you.

56 Likes
10

There are a couple reasons for this. The first is that Julia needs to do some compilation when you load packages. This is necessary because any package can add methods to functions that change how other functions perform. The good news is that this is expected to continue to speed up (there are already ideas for how to make 1.7 faster than 1.6 currently is).

15 Likes
11

Thank you

12

Do you know why julia is slower here than other languages?

From the manual:

Large modules can take several seconds to load because executing all of the statements in a module often involves compiling a large amount of code.

In simplified terms, Julia is a compiled language while Python is an interpreted language (the reality is much more complicated than that). If you really want to understand why, you’re going to have to invest some time in learning about how Julia and Python work under the hood. A thorough answer to your question would be quite long.

2 Likes
13

I know that Julia is compiled, but I did not know that it needs to be (partly) recompiled every time you relaunch julia, because of what Oscar_Smith said:

This is necessary because any package can add methods to functions that change how other functions perform.

3 Likes
14

I could reproduce that:

julia> @time using Plots
3.541552 seconds (5.94 M allocations: 426.191 MiB, 3.80% gc time, 18.98% compilation time)

julia> @time using DifferentialEquations
6.540690 seconds (12.49 M allocations: 1.001 GiB, 6.18% gc time, 22.92% compilation time)

5 Likes
15

With a sysimage I managed to get the startup time to usuable speeds. I know that many people don’t care for startup time, but I can’t use the REPL, because I haven’t found any IDE workflow that is acceptable.
Thank you.

2 Likes
16

If you haven’t found an IDE you like, you can just use a Jupyter notebook as a REPL-like experience (but with persistence and multimedia)…

2 Likes
17

I know about the jupyter notebook, but

  1. You cannot split your code into multiple files.
  2. Jupyter notebook can’t work with *.jl files, but all libraries are *.jl files
  3. Backtraces will not show the correct file number
  4. There is no “go to definition” feature like in vs code.

What I actually just tried is writing in vscode and having a jupyter notebook that is just one line:
include(“main.jl”)

Same with pluto notebook

This works pretty ok.

18

Yes you can, using NBInclude.

(But if you are at the point of making lots of files, you should probably be creating modules and doing structured programming, and then calling your modules from your notebooks. In fact, with NBInclude you can actually use notebook files as part of your Julia modules if you want, but personally I would tend to use notebooks only for interactive exploration code and keep long-term code in modules.)

Jupyter notebook can’t work with *.jl files, but all libraries are *.jl files

include and import work just fine, as does Revise.jl for interactively running code as you edit modules.

Or do you mean that notebooks can’t be used to edit .jl files? I thought we were talking about REPL replacements here and interactive work? If you want an IDE or an editor, use vsCode or JupyterLab or …

I’m confused about whether you are talking about writing “libraries” (modules/packages) or about writing interactive scripts.

Backtraces will not show the correct file number

Backtraces should work fine in notebooks. For statements in the running notebook, they will give you line numbers like In[35]: line 15, i.e. line 15 of input cell In[35].

6 Likes
19

Check jupytext, you can use a .jl file as a notebook. That is super usefull for version control too.

Also in vscode you can have a similar interacting experience using the Shift + Enter workflow and separating each peace of code you wants to run (“cell”) between ##s directly in a .jl file

3 Likes
20

You asked pretty much the same thing a week ago, in a topic precisely about this, and got more or less the same answers.

Julia is developing very rapidly, but asking this every week is unlikely to result in new information.

18 Likes