Nice workflows for using and developing in Julia 1.9+

TIL what shared environments are. Sorry everyone for spreading confusion! :face_with_open_eyes_and_hand_over_mouth:

So to summarize:

  • Activating a shared environment is fine
  • Stacking is generally fine, but can be dangerous
  • The default environment (aka (@v1.8)) is stacked by default (via the LOAD_PATH), so beware of accidentally using packages from it.
4 Likes

This might help: https://github.com/JuliaLang/Pkg.jl

A ā€œsharedā€ environment is simply an environment that exists in ~/.julia/environments. The default v1.8 environment is therefore a shared environment.

Shared environments have a @ before their name in the Pkg REPL prompt.

This documentation is confusing though because user-generated shared environments do not work the same way as the default shared environment (unless you explicitly alter the LOAD_PATH) as lmiq says:

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

(@DataFramesTest) pkg> activate --temp
  Activating new project at `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz`

(jl_nosSqz) pkg> st
Status `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz\Project.toml` (empty project)

julia> using DataFrames # Local environment does not stack with user-generated shared environment.
 │ Package DataFrames not found, but a package named DataFrames is available from a registry.
 │ Install package?
 │   (jl_nosSqz) pkg> add DataFrames
 ā”” (y/n/o) [y]:

Also user-generated shared environments still stack with the default shared environment. (This is probably a good thing, but it is confusing.) You cannot escape the default (1.8) environment:

(@v1.8) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\v1.8\Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.2
  [f68482b8] Cthulhu v2.8.0
  [5903a43b] Infiltrator v1.6.3
⌃ [5fb14364] OhMyREPL v0.5.14
  [14b8a8f1] PkgTemplates v0.7.32
  [295af30f] Revise v3.5.1
Info Packages marked with ⌃ have new versions available and may be upgradable.

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

julia> using BenchmarkTools #Works fine even though not in current shared environment.

Ok now I get why there’s some confusion. Shared environments have nothing special, except being ā€œactivableā€ from anywhere.

But, because some shared environments are defined in JULIA_LOAD_PATH (namely, ["@", "@v#.#", "@stdlib"], as per julia doc), they looks ā€œspecialā€ … while they are not :slight_smile:

Conversely, it’s totally fine to set a ā€œregularā€ environment in JULIA_LOAD_PATH, as the same documentation shows (some paragraphs later) … which will behave as, say @v1.8 for example.

Hope this clears things up :wink:

3 Likes

Why is this confusing? The docs just state - correctly - that the default environment is a shared environment. That doesn’t imply that therefore any shared environment must have all characteristics of the default environment (such as automatically being included in the load path)?

3 Likes

I’ve said this before, but I wonder if it wouldn’t have been a better choice to keep the concept of a default environment (the one that is activated when you start julia without any particular project/environment options) and the concept of a ā€œstackā€ environment (the one that is on LOAD_PATH by default) separate. Currently, v1.X plays both of these roles, which, among other reasons, is unfortunate because it is way to easy to (accidentally) inflate the ā€œstackā€ environment (and not even know about it being stacked on top of other environments). If the ā€œstackā€ environment would be separate, one would have to be very explicit about adding packages to it and for the naive user it would just be empty.

8 Likes

Had never thought much about environments (except for using --temp often for throwaways) but I like that opinion.

This has been discussed in other threads, and could be solved by starting Julia by default in a temporary environment. Then, adding the shared/stacked packages to a new ā€œstackā€ environment, explicitly, that is in the path by default and shared. I like that idea.

Also, I also often use Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true; before activating a temporary env. This avoid updating the package repository when creating the environment. Just saves time for something temporary.

4 Likes

Hi. I’m ussing Jupyterlab over 1.8.5 Julialang server. I can adjust the ā€œlinewidth’sā€ plot in a totaly confortable way without ā€œRevise pkg.ā€ Even more, givent that my 1.8.5 Julialang server is running in a Virtual Machine using notebooks is the easier way for me.
I understood that a script (and VSCode of course) would be ussesfull if I` ll have to generate a dozen (o more) plots after have found the ā€œnicerā€ linewidth. Then I wouldnĀ“t need ā€œReviseā€ any more, and I’ ll run the script in a batch mode.
But I donĀ“t know if this was the ā€œcentralā€ idea of the article or I have loose something …
Is there a IJulia version (with Jupyterlab) running in Julia 1.9+ ?

I am not very familiar with development with notebooks (less still with Jupyter), so I cannot really say what are the advantages or disadvantages for one specific use.

I’m not a huge fan of notebooks, for instance in a small test here I tried using Plots in an empty Jupyter notebook and after a couple of minutes waiting I got ā€œunable to connect to kernelā€:

I’m sure that everything being setup correctly the experience must be better, but I usually don’t like the hidden extra layers of complexity that those higher level interfaces imply.

The example of the plot is one example that applies to any other development: using Revise you can develop any function with that very interactive workflow. VSCode is not even really required, you can have just a script in an open file in any editor and an open Julia section in a terminal window, where you have includet() the script.

(the reason I adopted VSCode is mostly because of its integration with github, but this is a point not directly related to the OP)

That has nothing to do with Plots — if you got that error then it would fail for 1+1. Jupyter/IJulia is misconfigured on your machine. You need IJulia in your default environment (at least, for the default kernel).

2 Likes

Hi again,
1.- Notebooks are very good to ā€œtryā€ and ā€œtestā€.Also are a good tool to ā€œshareā€ with other people.
I alsi think we are needed of a pure Julia notebooks tool.
The error that "unable to connect " is almost sure for a bad configuration.
2.- … Yes I know I not need VSCode editor but I have tested others and VSCode gives me the best experience at the moment.

Than you.

IĀ“m sure that it is a bad configuration. I didnĀ“t want to imply anything else. My point there is only that there is a (small) configuration barrier for using notebooks that is not much different from other workflows. That said, I’m not against anyone using and preferring them for any task. Its just not my preference.

Pluto is one such notebook, with very nice features. I use it to some things, presentations in particular, but I don’t like it as a practical tool for everyday experimental coding.

JupyterLab running your nice example in a Julia_1.8.5 on Virtual_Machine.


Here we have the same example in Pluto.
We can see that in Pluto, only the first ā€œrunā€ has a significant cost. Then it suggests us that the can make the first run like a test with few points (by minimizing problem ā€œsizeā€). Then, after we have the ā€œrecompilation doneā€ we could launch the definitive run.

That’s always like that. In 1.9 the first run is faster if you use environments (and Pluto does).

OK thanks.

I’d also add workflow for module reloading, since it is brittle and there are some corner cases:
(AFAIK, Revise just automates this process, binding it to the next REPL command.)

  • All type definitions should be placed inside module in a separate file.
  • Script files should have include("mymodule.jl") and import .MyModule1.
  • First change definitions within module, then re-run script files.
  • export’ed names are not realoaded (this point is missing from docs, so I’ve spent some hard time to find out this), so don’t pollute your global namespace with using and write import MyModule1 as my and make all calls with my.Foo
  • All above is unnecessary if you are already working only inside the module.
1 Like