Will Julia ever fix its "using ..." latency problems?

[josePereiro]

NOTE: Im not an expert
But It works with jupyter in general, among other stuff, It allows you to have a textual working representation of a notebook, I have use it for julia and python to produce .py and .jl runnable scripts. It just add a bunch of comments lines to store important info such that where a cell starts and ends or whether it is a markdown/code cell. After that you can use the .jl/.py/etc file alone to start a new session or in synchronization with a .pynb file. You can even drop the .pynb completely and work just using the common text files, but you lose the cell output of previous sessions.

Here and example

# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     cell_metadata_filter: -all
#     formats: jl,ipynb
#     text_representation:
#       extension: .jl
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.3.2
#   kernelspec:
#     display_name: Julia 1.1.0
#     language: julia
#     name: julia-1.1
# ---

# +
println("How you doooing?")
# -

This is a .jl file equivalent to a .pynb with a single cell with the print statement.

I primarily use it because the .pynb is too noise for version control.

[Volker_Weissmann]

@Palli

I can see your point, while you put them to a very high standard, higher than I guess any proprietary software and most open source.

I put every software at a very high standard. I can rant about every software except for maybe Rust.

Julia is one of the better languages.

[Henrique_Becker]

@FedericoStra

On the other hand, if you don’t accept these outcomes, then no, redefining const is not truly UB in the sense of C.

I accept these outcomes.

I am not sure this discussion is relevant anymore, when I said “undefined behavior” I meant in both the general sense and in the C sense, because for me they are the same. I have read a manual of C99 (made for people that wanted to understand the nuances of the language enough to write a compiler for it) and I have never understood undefined behavior the same way you do. All these spurious examples of undefined behavior fall under the umbrella of undefined behavior in the general sense, and while many times they are mentioned in the context of C they never happened in any compiler that I know of. They are ridiculous just to be easier to imprint the dangers of undefined behavior on young programmers’ minds. The undefined behaviour C will display by breaking rules similar to this one will be similar to the undefined behavior displayed by Julia. The definition of C keeps the possibilities open to anything just to protect compiler maintainers free from user complaints.

[FedericoStra]

@Henrique_Becker

I accept these outcomes.

Ok, then in this case tell me of what use it is the possibility to redefine a const in an interactive session, if from the moment we try to do so our program instantly loses any meaning whatsoever.

What I’m desperately trying to say is that there are two alternatives:

1. Redefining a const is UB.
2. Redefining a const c means that occurrences of c can have any possible value that it has ever had during the entire session. This is not UB; it i closer to undefined value, unpecified value, unspecified behavior, whatever you want to call it.

If 1. is true, then we accept that const c = 0; c = 1 can format the hard disk.

The documentation, in its current state, seems to lean more towards 2.

I agree that all this discussion has become quite useless.

[Tamas_Papp]

@FedericoStra

Is it allowed to print 42 ? What about "hello" ? What about printing nothing? What about never halting?

None of these outcomes are ruled out by the API as defined at the moment. Please read this again:

this can produce unpredictable behavior or corrupt the state of your program

Once the image is corrupted, anything goes.

[FedericoStra]

@Tamas_Papp

None of these outcomes are ruled out by the API as defined at the moment. Please read this again:

this can produce unpredictable behavior or corrupt the state of your program

The problem lies there. What is “unpredictable” behavior? Having any arbitrary value previously assigned is “unpredictable behavior” too, but is not undefined behavior. That’s why I’m saying that from the docs is not clear at all whether reassigning is undefined behavior or more like unspecified value. They are dramatically different concepts.

---

@Tamas_Papp

None of these outcomes are ruled out by the API as defined at the moment.

This might very well be the case, and it is quite unfortunate to be honest. In this case, doing

const c = 0
const c = 1

at the REPL is instant nonsense, you might just as well restart the session (and wait 40 second for using Plots, DifferentialEquations…).

[Henrique_Becker]

@FedericoStra

then in this case tell me of what use it is the possibility to redefine a const in an interactive session

One possible outcome of undefined behavior include the code working as you would expect if the variable was not const. This is clearly intended for REPL use (the documentation says so). The basic idea is: instead of closing and opening the REPL again to redefine a const variable, I can try my luck and just redefine it to make a fast test I wanted to do; if it behaves as I expected this change to behave, then I had luck in my favor and this saved me a few seconds, if something else happens then I did not have luck on my side and will have to restart to adequately test what I wanted. Simple as that. It is a luck/chance/probability/bayesian thing. You have the possibility of play dice with the compiler and get some not-entirely-trustworthy information.

@FedericoStra

Redefining a const c means that occurrences of c can have any possible value that it has ever had during the entire session. This is not UB; it i closer to undefined value, unpecified value, unspecified behavior, whatever you want to call it.

I am not sure if the possible changes in behavior are restricted to just to that one. I understand your point now, if this is the only possible behavior (i.e., the code may or may not act as any of the previous values was yet assigned to the variable) then you could infer more things and not worry about having your disk formatted, XD. It would be better to inquire someone from the Julia internals development about this. In particular, I am not sure if changing the const variable to an object of another type will always error, or it may accept it and end up with a segmentation fault when the variable is referenced, as the object address may not be inlined but the type information can be, for example.

[FedericoStra]

@Henrique_Becker

The basic idea is: […], I can try my luck […] It is a luck/chance/probability/bayesian thing. You have the possibility of play dice with the compiler and get some not-entirely-trustworthy information.

I get what you mean, and I agree that it might be “sufficient” in some circumstances, but please consider that this might affect some tests that silently pass instead of failing. I don’t particularly like this chance-based programming, and I would not like to set foot on an airplane whose fly-by-wire system has been tested this way.

---

@Henrique_Becker

I am not sure if the possible changes in behavior are restricted to just to that one.

No no, definitely that one is not the only possiblility: it was just an example of an alternative to full UB.

---

@Henrique_Becker

In particular, I am not sure if changing the const variable to an object of another type will always error, or it may accept it and end up with a segmentation fault when the variable is referenced, as the object address may not be inlined but the type information can be, for example.

This would be particularly nasty. If reassigning to a const is UB (especially if the type is different), I don’t see any reason not to make it a hard error instead, since anything that follows it would be completely meaningless (actually, in C, UB is “retroactive” too, so also what precedes the invalid operation becomes meaningless: see this example).

---

In any case, I’ve opened this issue, where I’m not expecting plenty of spam regarding what UB is or isn’t (as witnessed in this thread), but rather a clarifying answer from the core developers.

[Palli]

@Volker_Weissmann DifferentialEquations is a bit unfair, but also getting a lot faster.*

Several plotting packages have sub-sec. startup (but I’m otherwise checking same packages you did/the abstraction plotting packages; what are comparable package/s in Python, i.e. with backends?):

julia> @time using PGFPlotsX
  0.413068 seconds (464.33 k allocations: 29.720 MiB, 10.62% compilation time)

same plotting package you get with Plots.jl, without the abstraction:

julia> @time using GR
  0.815509 seconds (819.21 k allocations: 47.663 MiB, 1.71% gc time, 86.12% compilation time)

Fastest plotting library?

If using PGFPlotsX for multiple figures, make sure you are not generating one figure per pdf but instead put multiple figures into the same pdf. As an example: x = range(-1; stop = 1, length = 51); axises = [@pgf Axis( { xmajorgrids, ymajorgrids, }, Plot( { no_marks, }, Coordinates(x, 1 ./ (x.-i)) ) ) for i in 1:10]; @time for (i, axis) in enumerate(axises) pgfsave("file_$i.pdf", axis) end # ~ 5 seconds td = TikzDocument…

@tbeason

To be fair you’ve picked out like the worst of the worst here. Most packages have rather small load times.

There’s been good progress made from 12.7 sec (on your machine), now 70% faster (on my loaded machine, compared to yours):

in 1.9-DEV:

julia> @time using Plots
  3.871938 seconds (5.66 M allocations: 393.934 MiB, 4.29% gc time, 22.05% compilation time: 0% of which was recompilation)

after:
[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
 80.083775 seconds (5.59 M allocations: 389.394 MiB, 0.21% gc time, 1.14% compilation time: 0% of which was recompilation)

vs in 1.7.0:

julia> @time using Plots
  6.400478 seconds (9.10 M allocations: 642.353 MiB, 5.92% gc time, 20.97% compilation time)

[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
151.895940 seconds (9.11 M allocations: 643.639 MiB, 0.48% gc time, 1.11% compilation time)

I’m testing Plots.jl since you complained, and to your numbers, but the better package by now seems to be Makie.jl, and there are other simpler with very fast startup.

You can actually get instantaneous startup, I understand, of (these/these?) [plotting] packages, by putting in your sysimage, still showing for 1.7.0 without that:

julia> @time using CairoMakie
  8.230858 seconds (12.80 M allocations: 891.117 MiB, 6.84% gc time, 11.23% compilation time)

julia> @time using AlgebraOfGraphics
  8.557753 seconds (13.31 M allocations: 922.143 MiB, 6.76% gc time, 12.52% compilation time)

[the times do not add up for both of above, you only get extra 0.3 sec for the latter after usinging the former, and I believe you can use them together.]

or if you need 3D/OpenGL other backends:

julia> @time using GLMakie
 10.075012 seconds (14.93 M allocations: 1021.257 MiB, 8.08% gc time, 9.91% compilation time)

julia> @time using RPRMakie
  8.130355 seconds (12.83 M allocations: 892.046 MiB, 6.78% gc time, 10.59% compilation time)

* DifferentialEquations.jl also has 70% faster startup (on 1.7.0) at 10 sec. Maybe not great, but likely the slowest package in the Julia ecosystem to start (but fastest at runtime compares to any other language). Since you compare to Python, and it or no language compares for this, not too helpful to look at the slow (by now tolerable?) startup time. Also use can use it from Python, but since you’re then calling Julia, I assume the startup is the same.

With 1.9 not out, it’s of course not yet supported on 1.9, and it might be also much faster after that upgrade, as Plots.jl did, when this fixed:

julia> @time using DifferentialEquations
[ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
ERROR: LoadError: Sundials is not properly installed. Please run Pkg.build("Sundials")

@Volker_Weissmann

The “using …” command is extremely slow:

julia> @time using Plots
12.731850 seconds (17.25 M allocations: 976.326 MiB, 3.64% gc time)

julia> @time using DifferentialEquations
34.780873 seconds (41.90 M allocations: 2.456 GiB, 2.63% gc time)

julia> @time using Atom
3.137487 seconds (2.95 M allocations: 184.199 MiB, 1.27% gc time)

[I get actually slightly slower timing for Atom, but it could be my machine, and since Atom is in maintenance mode, VS Code is what’s important now (and is fast for me).]

I realized there’s one minor regression on master where I do not get this (convenient) prompt, unlike on 1.7.0 [EDIT: Issue filed, see next comment]

[ianshmean]

@Palli

I realized there’s one minor regression on master where I do not get this (convenient) prompt, unlike on 1.7.0

Just filed an issue Missing package install prompt doesn't work when load is behind a macro · Issue #45082 · JuliaLang/julia · GitHub