Nice workflows for using and developing in Julia 1.9+

[cstjean]

TIL what shared environments are. Sorry everyone for spreading confusion!

So to summarize:

• Activating a shared environment is fine
• Stacking is generally fine, but can be dangerous
• The default environment (aka (@v1.8)) is stacked by default (via the LOAD_PATH), so beware of accidentally using packages from it.

[Barget]

This might help: https://github.com/JuliaLang/Pkg.jl

[Nathan_Boyer]

@Barget

This might help: https://github.com/JuliaLang/Pkg.jl

A “shared” environment is simply an environment that exists in ~/.julia/environments. The default v1.8 environment is therefore a shared environment.

Shared environments have a @ before their name in the Pkg REPL prompt.

This documentation is confusing though because user-generated shared environments do not work the same way as the default shared environment (unless you explicitly alter the LOAD_PATH) as lmiq says:

@lmiq

That applies to the “main” shared environment only. You cannot load packages from other shared environments without explicitly activating them first, and in that sense they behave the same as any other environment, as far as I understand.

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

(@DataFramesTest) pkg> activate --temp
  Activating new project at `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz`

(jl_nosSqz) pkg> st
Status `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz\Project.toml` (empty project)

julia> using DataFrames # Local environment does not stack with user-generated shared environment.
 │ Package DataFrames not found, but a package named DataFrames is available from a registry.
 │ Install package?
 │   (jl_nosSqz) pkg> add DataFrames
 └ (y/n/o) [y]:

Also user-generated shared environments still stack with the default shared environment. (This is probably a good thing, but it is confusing.) You cannot escape the default (1.8) environment:

(@v1.8) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\v1.8\Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.2
  [f68482b8] Cthulhu v2.8.0
  [5903a43b] Infiltrator v1.6.3
⌃ [5fb14364] OhMyREPL v0.5.14
  [14b8a8f1] PkgTemplates v0.7.32
  [295af30f] Revise v3.5.1
Info Packages marked with ⌃ have new versions available and may be upgradable.

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

julia> using BenchmarkTools #Works fine even though not in current shared environment.

[Barget]

Ok now I get why there’s some confusion. Shared environments have nothing special, except being “activable” from anywhere.

But, because some shared environments are defined in JULIA_LOAD_PATH (namely, ["@", "@v#.#", "@stdlib"], as per julia doc), they looks “special” … while they are not

Conversely, it’s totally fine to set a “regular” environment in JULIA_LOAD_PATH, as the same documentation shows (some paragraphs later) … which will behave as, say @v1.8 for example.

Hope this clears things up

[nilshg]

Why is this confusing? The docs just state - correctly - that the default environment is a shared environment. That doesn’t imply that therefore any shared environment must have all characteristics of the default environment (such as automatically being included in the load path)?

[carstenbauer]

I’ve said this before, but I wonder if it wouldn’t have been a better choice to keep the concept of a default environment (the one that is activated when you start julia without any particular project/environment options) and the concept of a “stack” environment (the one that is on LOAD_PATH by default) separate. Currently, v1.X plays both of these roles, which, among other reasons, is unfortunate because it is way to easy to (accidentally) inflate the “stack” environment (and not even know about it being stacked on top of other environments). If the “stack” environment would be separate, one would have to be very explicit about adding packages to it and for the naive user it would just be empty.

[tbeason]

@carstenbauer

If the “stack” environment would be separate, one would have to be very explicit about adding packages to it and for the naive user it would just be empty.

Had never thought much about environments (except for using --temp often for throwaways) but I like that opinion.

[lmiq]

@carstenbauer

Currently, v1.X plays both of these roles, which, among other reasons, is unfortunate because it is way to easy to (accidentally) inflate the “stack” environment (and not even know about it being stacked on top of other environments).

This has been discussed in other threads, and could be solved by starting Julia by default in a temporary environment. Then, adding the shared/stacked packages to a new “stack” environment, explicitly, that is in the path by default and shared. I like that idea.

[Emmanuel-R8]

Also, I also often use Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true; before activating a temporary env. This avoid updating the package repository when creating the environment. Just saves time for something temporary.

[fer2tdt]

Hi. I’m ussing Jupyterlab over 1.8.5 Julialang server. I can adjust the “linewidth’s” plot in a totaly confortable way without “Revise pkg.” Even more, givent that my 1.8.5 Julialang server is running in a Virtual Machine using notebooks is the easier way for me.
I understood that a script (and VSCode of course) would be ussesfull if I` ll have to generate a dozen (o more) plots after have found the “nicer” linewidth. Then I wouldn´t need “Revise” any more, and I’ ll run the script in a batch mode.
But I don´t know if this was the “central” idea of the article or I have loose something …
Is there a IJulia version (with Jupyterlab) running in Julia 1.9+ ?

[lmiq]

I am not very familiar with development with notebooks (less still with Jupyter), so I cannot really say what are the advantages or disadvantages for one specific use.

I’m not a huge fan of notebooks, for instance in a small test here I tried using Plots in an empty Jupyter notebook and after a couple of minutes waiting I got “unable to connect to kernel”:

I’m sure that everything being setup correctly the experience must be better, but I usually don’t like the hidden extra layers of complexity that those higher level interfaces imply.

The example of the plot is one example that applies to any other development: using Revise you can develop any function with that very interactive workflow. VSCode is not even really required, you can have just a script in an open file in any editor and an open Julia section in a terminal window, where you have includet() the script.

(the reason I adopted VSCode is mostly because of its integration with github, but this is a point not directly related to the OP)

[stevengj]

@lmiq

I’m not a huge fan of notebooks, for instance in a small test here I tried using Plots in an empty Jupyter notebook and after a couple of minutes waiting I got “unable to connect to kernel”:

That has nothing to do with Plots — if you got that error then it would fail for 1+1. Jupyter/IJulia is misconfigured on your machine. You need IJulia in your default environment (at least, for the default kernel).

[fer2tdt]

Hi again,
1.- Notebooks are very good to “try” and “test”.Also are a good tool to “share” with other people.
I alsi think we are needed of a pure Julia notebooks tool.
The error that "unable to connect " is almost sure for a bad configuration.
2.- … Yes I know I not need VSCode editor but I have tested others and VSCode gives me the best experience at the moment.

Than you.

[lmiq]

@stevengj

That has nothing to do with Plots — if you got that error then it would fail for 1+1. Jupyter/IJulia is misconfigured on your machine. You need IJulia in your default environment (at least, for the default kernel).

@fer2tdt

The error that "unable to connect " is almost sure for a bad configuration.

I´m sure that it is a bad configuration. I didn´t want to imply anything else. My point there is only that there is a (small) configuration barrier for using notebooks that is not much different from other workflows. That said, I’m not against anyone using and preferring them for any task. Its just not my preference.

@fer2tdt

I alsi think we are needed of a pure Julia notebooks tool.

Pluto is one such notebook, with very nice features. I use it to some things, presentations in particular, but I don’t like it as a practical tool for everyday experimental coding.

[fer2tdt]

[fer2tdt]

JupyterLab running your nice example in a Julia_1.8.5 on Virtual_Machine.

[fer2tdt]

@lmiq

@time my_plot(data)

Here we have the same example in Pluto.
We can see that in Pluto, only the first “run” has a significant cost. Then it suggests us that the can make the first run like a test with few points (by minimizing problem “size”). Then, after we have the “recompilation done” we could launch the definitive run.

[lmiq]

That’s always like that. In 1.9 the first run is faster if you use environments (and Pluto does).

[fer2tdt]

OK thanks.

[sairus7]

I’d also add workflow for module reloading, since it is brittle and there are some corner cases:
(AFAIK, Revise just automates this process, binding it to the next REPL command.)

• All type definitions should be placed inside module in a separate file.
• Script files should have include("mymodule.jl") and import .MyModule1.
• First change definitions within module, then re-run script files.
• export’ed names are not realoaded (this point is missing from docs, so I’ve spent some hard time to find out this), so don’t pollute your global namespace with using and write import MyModule1 as my and make all calls with my.Foo
• All above is unnecessary if you are already working only inside the module.