Globally Available Environments and State of Package Compilation

General Usage Performance
,
1

1. Environments in Julia

I recently started using Conda environments for Python in my Class and realized how simple and efficient it is to use them.
Unlike how I do things in Julia - for each new project, I just create a new environment, add the required packages, and start working - in Anaconda, you define a globally available set of environments that you can choose at runtime.

This reduces the redundant environments I make for projects that use the same packages. I don’t need to create a new environment, copy the package name list and install all packages again. I do not need another copy of the same environment. Instead, I can just choose the environment I want to work with from a list of environments that are available as a choice in all my projects.

How do I do this in Julia? Specifically in VSCode.

I know you can simply add the interpreter path to open the Julia environment at a particular location. But in VSCode, we can only see environments defined within our folder in hand - not a global set of environments. Is there a way to not have to navigate to my global environments folder each time to select an environment?

Would I need to create some folder of my own, say Julia-Environments, create all my environments there, and then add its path to environment variables in Windows? Or would I need to add the path somewhere in VSCode? Or both? Is that how that may work?

2. State of Package Compilation

To speed up the load-time of my environments, I was looking into a way to store the compiled machine code that Julia creates each time on the first run. I simply want a way to initialize my environment each time with a specific plot that I can then work on further.

I came across a few compilation packages - PackageCompiler and CompilerTools (I swear I could see its documentation until a few days ago, but now it seems to be discontinued. Maybe I was viewing some other package. But I know there was some other package that had in its description that it is akin to PackageCompiler.jl but has a different method of saving the compilation - not images).

I tried working with PackageCompiler.jl, trying to compile the Example in their documentation, which just plots a handful of points and displays it.

@time using Plots
@time p = plot(rand(10), rand(10));
@time display(p);

ran create_sysimage(["Plots"], sysimage_path="sys_plots.so", precompile_execution_file="precompile_plots.jl")
And it still hasn’t been compiled.

Assuming it does in the next few minutes, how would I even run things in VSCode with this? I read online that they got rid of the sysimage support from Julia Extention in VSCode, saying not many people use it now.

What do people do? Does Julia actually natively store the compiled code? Is Julia faster now? Is PackageCompiler unnecessary? How do I speed things up on the first runtime surely, each time I open a new session, the same plotting code is compiled again, taking a minute each time I try to start working?

2

Update:

(base) PS E:\Programming\Julia-Environments\PackageCompilerTest> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.3 (2024-04-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using Plots
  2.234673 seconds (1.14 M allocations: 80.397 MiB, 6.50% gc time, 2.79% compilation time)

julia> @time p = plot(rand(10), rand(10));
  0.119848 seconds (2.96 k allocations: 186.695 KiB, 92.70% compilation time)

julia> @time display(p);
  2.830213 seconds (94.13 k allocations: 6.562 MiB, 7.38% compilation time: 11% of which was recompilation)

(base) PS E:\Programming\Julia-Environments\PackageCompilerTest> julia --sysimage sys_plots.so
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.3 (2024-04-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using Plots
  0.000972 seconds (286 allocations: 20.992 KiB)

julia> @time p = plot(rand(10), rand(10));
  0.002168 seconds (664 allocations: 55.477 KiB)

julia> @time display(p);
Plot{Plots.GRBackend() n=1}
  0.074793 seconds (153.97 k allocations: 10.953 MiB, 97.54% compilation time)

I don’t know why the display function still compiles on runtime. But well, that is pretty substantial speedup. I still don’t know how to get it to work on VSCode while working in a file and running it.

EDIT:

Interestingly,

julia> display(p)
Plot{Plots.GRBackend() n=1}

julia> display(p);
Plot{Plots.GRBackend() n=1}

julia> p
Plot{Plots.GRBackend() n=1}

I tried all these for the sysimage run, but none actually displayed the plot. It does work for the normal plot display without the sysimage.

(I wonder if that has something to do with localhost during the time of compilation)

3

What you’re looking for is shared environments, which is any environment whose name starts with @: 4. Working with Environment · Pkg.jl. They’re stored in ~/.julia/environments, the same place the base environment is stored. I don’t use VSCode, so can’t help you there, but from the REPL it’s as simple as ] activate @mysharedenv, or, equivalently, ] activate --shared mysharedenv (the --shared switch adds the @ for you). Similarly, from the command line you can start Julia as $ julia --project=@mysharedenv.

5 Likes
4

Omg! Thank you so much! That was exactly what I was looking for in terms of environment management!

2 Likes
5

The question about Package Compilation still holds.
What

6

For managing and using shared environments, you might find ShareAdd.jl helpful.

7

PrecompileTools.jl?

v1.9+ does, but because of the unbounded nature of Julia methods you have to specify call signatures in some way. All of the actively maintained libraries for this are still relevant.

1 Like
8

Yesss!

Do you know how to add the sysimages or other formats that these packages use to be available on VSCode while I run my files?

So, I can directly use the sysimage to run each file.

Can it work somehow with shift + enter to run a particular selection in the terminal?

So for instance, can I run the selected code in a terminal that I ran the sysimage in?

Also, why was the plot not displayed? Should PackageCompiler not be used with display functions?

9

Cached native code is not tied to VSCode settings or source files. Julia’s system images aka sysimages are tied to sessions, package images are tied to packages. We can’t mix sessions together, and we can mix packages within one session, though each can only be loaded once for a session’s lifetime. Within one session, aim to rerun functions on data, not rerun entire files.