Nice workflows for using and developing in Julia 1.9+

TIL what shared environments are. Sorry everyone for spreading confusion! :face_with_open_eyes_and_hand_over_mouth:

So to summarize:

  • Activating a shared environment is fine
  • Stacking is generally fine, but can be dangerous
  • The default environment (aka (@v1.8)) is stacked by default (via the LOAD_PATH), so beware of accidentally using packages from it.
4 Likes

This might help: https://github.com/JuliaLang/Pkg.jl

A ā€œsharedā€ environment is simply an environment that exists in ~/.julia/environments. The default v1.8 environment is therefore a shared environment.

Shared environments have a @ before their name in the Pkg REPL prompt.

This documentation is confusing though because user-generated shared environments do not work the same way as the default shared environment (unless you explicitly alter the LOAD_PATH) as lmiq says:

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

(@DataFramesTest) pkg> activate --temp
  Activating new project at `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz`

(jl_nosSqz) pkg> st
Status `C:\Users\nboyer.AIP\AppData\Local\Temp\jl_nosSqz\Project.toml` (empty project)

julia> using DataFrames # Local environment does not stack with user-generated shared environment.
 ā”‚ Package DataFrames not found, but a package named DataFrames is available from a registry.
 ā”‚ Install package?
 ā”‚   (jl_nosSqz) pkg> add DataFrames
 ā”” (y/n/o) [y]:

Also user-generated shared environments still stack with the default shared environment. (This is probably a good thing, but it is confusing.) You cannot escape the default (1.8) environment:

(@v1.8) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\v1.8\Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.2
  [f68482b8] Cthulhu v2.8.0
  [5903a43b] Infiltrator v1.6.3
āŒƒ [5fb14364] OhMyREPL v0.5.14
  [14b8a8f1] PkgTemplates v0.7.32
  [295af30f] Revise v3.5.1
Info Packages marked with āŒƒ have new versions available and may be upgradable.

(@v1.8) pkg> activate @DataFramesTest
  Activating project at `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest`

(@DataFramesTest) pkg> st
Status `C:\Users\nboyer.AIP\.julia\environments\DataFramesTest\Project.toml`
  [a93c6f00] DataFrames v1.5.0 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [08abe8d2] PrettyTables v2.2.2

julia> using BenchmarkTools #Works fine even though not in current shared environment.

Ok now I get why thereā€™s some confusion. Shared environments have nothing special, except being ā€œactivableā€ from anywhere.

But, because some shared environments are defined in JULIA_LOAD_PATH (namely, ["@", "@v#.#", "@stdlib"], as per julia doc), they looks ā€œspecialā€ ā€¦ while they are not :slight_smile:

Conversely, itā€™s totally fine to set a ā€œregularā€ environment in JULIA_LOAD_PATH, as the same documentation shows (some paragraphs later) ā€¦ which will behave as, say @v1.8 for example.

Hope this clears things up :wink:

3 Likes

Why is this confusing? The docs just state - correctly - that the default environment is a shared environment. That doesnā€™t imply that therefore any shared environment must have all characteristics of the default environment (such as automatically being included in the load path)?

3 Likes

Iā€™ve said this before, but I wonder if it wouldnā€™t have been a better choice to keep the concept of a default environment (the one that is activated when you start julia without any particular project/environment options) and the concept of a ā€œstackā€ environment (the one that is on LOAD_PATH by default) separate. Currently, v1.X plays both of these roles, which, among other reasons, is unfortunate because it is way to easy to (accidentally) inflate the ā€œstackā€ environment (and not even know about it being stacked on top of other environments). If the ā€œstackā€ environment would be separate, one would have to be very explicit about adding packages to it and for the naive user it would just be empty.

8 Likes

Had never thought much about environments (except for using --temp often for throwaways) but I like that opinion.

This has been discussed in other threads, and could be solved by starting Julia by default in a temporary environment. Then, adding the shared/stacked packages to a new ā€œstackā€ environment, explicitly, that is in the path by default and shared. I like that idea.

Also, I also often use Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true; before activating a temporary env. This avoid updating the package repository when creating the environment. Just saves time for something temporary.

4 Likes

Hi. Iā€™m ussing Jupyterlab over 1.8.5 Julialang server. I can adjust the ā€œlinewidthā€™sā€ plot in a totaly confortable way without ā€œRevise pkg.ā€ Even more, givent that my 1.8.5 Julialang server is running in a Virtual Machine using notebooks is the easier way for me.
I understood that a script (and VSCode of course) would be ussesfull if I` ll have to generate a dozen (o more) plots after have found the ā€œnicerā€ linewidth. Then I wouldnĀ“t need ā€œReviseā€ any more, and Iā€™ ll run the script in a batch mode.
But I donĀ“t know if this was the ā€œcentralā€ idea of the article or I have loose something ā€¦
Is there a IJulia version (with Jupyterlab) running in Julia 1.9+ ?

I am not very familiar with development with notebooks (less still with Jupyter), so I cannot really say what are the advantages or disadvantages for one specific use.

Iā€™m not a huge fan of notebooks, for instance in a small test here I tried using Plots in an empty Jupyter notebook and after a couple of minutes waiting I got ā€œunable to connect to kernelā€:

Iā€™m sure that everything being setup correctly the experience must be better, but I usually donā€™t like the hidden extra layers of complexity that those higher level interfaces imply.

The example of the plot is one example that applies to any other development: using Revise you can develop any function with that very interactive workflow. VSCode is not even really required, you can have just a script in an open file in any editor and an open Julia section in a terminal window, where you have includet() the script.

(the reason I adopted VSCode is mostly because of its integration with github, but this is a point not directly related to the OP)

That has nothing to do with Plots ā€” if you got that error then it would fail for 1+1. Jupyter/IJulia is misconfigured on your machine. You need IJulia in your default environment (at least, for the default kernel).

2 Likes

Hi again,
1.- Notebooks are very good to ā€œtryā€ and ā€œtestā€.Also are a good tool to ā€œshareā€ with other people.
I alsi think we are needed of a pure Julia notebooks tool.
The error that "unable to connect " is almost sure for a bad configuration.
2.- ā€¦ Yes I know I not need VSCode editor but I have tested others and VSCode gives me the best experience at the moment.

Than you.

IĀ“m sure that it is a bad configuration. I didnĀ“t want to imply anything else. My point there is only that there is a (small) configuration barrier for using notebooks that is not much different from other workflows. That said, Iā€™m not against anyone using and preferring them for any task. Its just not my preference.

Pluto is one such notebook, with very nice features. I use it to some things, presentations in particular, but I donā€™t like it as a practical tool for everyday experimental coding.

JupyterLab running your nice example in a Julia_1.8.5 on Virtual_Machine.


Here we have the same example in Pluto.
We can see that in Pluto, only the first ā€œrunā€ has a significant cost. Then it suggests us that the can make the first run like a test with few points (by minimizing problem ā€œsizeā€). Then, after we have the ā€œrecompilation doneā€ we could launch the definitive run.

Thatā€™s always like that. In 1.9 the first run is faster if you use environments (and Pluto does).

OK thanks.

Iā€™d also add workflow for module reloading, since it is brittle and there are some corner cases:
(AFAIK, Revise just automates this process, binding it to the next REPL command.)

  • All type definitions should be placed inside module in a separate file.
  • Script files should have include("mymodule.jl") and import .MyModule1.
  • First change definitions within module, then re-run script files.
  • exportā€™ed names are not realoaded (this point is missing from docs, so Iā€™ve spent some hard time to find out this), so donā€™t pollute your global namespace with using and write import MyModule1 as my and make all calls with my.Foo
  • All above is unnecessary if you are already working only inside the module.
1 Like