Projects, environments: workflow questions

[jdm204]

Hello,

I’m trying to understand the Julia ‘best practices’ around projects/environments with Pkg.

First, are ‘project’ and ‘environment’ synonymous in Julia?

Second, since I’ve separated my packages into environments to avoid incompatibilities, and then activated a specific environment through julia --project foo or ] activate foo, I’m surprised that I still have to import each package (which i assume belies some fundamental misunderstanding). I understand that I might not want to ‘using’ everything because I might want to refer to things through the SomePackage.somefunction syntax, but why would I want to have an environment with access to a specific bunch of packages but have no way to access them? It seems to me that I’d always want to run something like:

import Pkg; for pkg in Pkg.project().dependencies |> keys @eval import $(Symbol(pkg)) end

once I open a REPL in a particular environment, but I’ve never seen anything like that in a startup script. Coming from R, I can access any installed packaged through pkgname::symbol syntax as soon as the REPL launches.

Third, how should packages be organised? I think I read that the base environment (@1.8 currently) should preferably just contain packages that will always be useful, so I have Revise, Match, OhMyREPL in there. Then I have a @stats environment that I use for an R-like batteries-included REPL with GLM, CSV, Makie, DataFrames etc. Then if I have a specific, defined project to complete (for publication for example), I would make a new environment from scratch specifically for that project. Is this roughly the right idea?

Many thanks!

[aplavin]

Auto-import sounds interesting, also haven’t seen this idea before!

[jmair]

Hi, welcome to the forum!

First of all, an environment in Julia just refers to a Project.toml file inside a folder which lists the package dependencies.

A package is slightly different as it has a named module and a specific folder structure. You can add a package as a dependency to another environment, but you can’t add a basic environment to another environment. A package also has a Project.toml file which contains some metadata about the package, like version, name and a GUID. The easiest way to generate a package is with PkgTemplates.jl (in interactive mode). This creates all the folders and starting files you need, and you can copy in any existing code.

@jdm204

I’m surprised that I still have to import each package (which i assume belies some fundamental misunderstanding).

If you have added another package as a dependency, it does not load all the dependencies at once, since this is often unnecessary, and one doesn’t need access to every package all at once. Usually, you say which packages you need at the top of a file. As for imports vs using, using Plots is completely fine as you only need to run import if you really don’t want to pollute the namespace, or there are naming conflicts between two packages.

If you want to autoimport some packages, check out startup.jl (found at ~/.julia/config/startup.jl), which I think does that for you.

Lastly, I think each new project should be in a new folder, with a separate environment. You do this so that you can (1) check it into source control and (2) share it with others (usually via (1)), and they can be up and running instantly. It is also best to be explicit in a script about which packages are being used in a file, so that someone else with a different startup.jl can run the code straight away.

[ederag]

As a complement, @jules wrote a nice and solid introduction to packages and environments.

[digital_carver]

@jdm204

First, are ‘project’ and ‘environment’ synonymous in Julia?

In practice, yeah. Environments are a broader concept in that there are also shared environments like your @stat, which are not exactly projects. But in casual Julia discussion when someone says “activate the project” or “create a project”, they’re usually referring to environments.

@jdm204

Second, since I’ve separated my packages into environments to avoid incompatibilities, and then activated a specific environment through julia --project foo or ] activate foo, I’m surprised that I still have to import each package

When you activate a project, you’re only telling Pkg which Project.toml to add packages to or rm them from. Julia itself uses a stack of environments to load packages from, and the currently activated environment is merely the first among them. Loading all packages from all environments in the stack would load quite a lot of (mostly unnecessary) things.

(Also, there are cases like FileIO.jl backends, where you want the backend package installed, but not loaded - but that’s pretty rare afaik.)

Of course, it would be a possible feature that Pkg could import a package as soon as we add it - I’ve sometimes wished for that - but now we have a similar feature coming from the other way around: importing a package that hasn’t been added prompts you to add it to the current environment.

@jdm204

Third, how should packages be organised?

This is pretty personal depending on your workflow, other than keeping the base @v1.8 environment minimal. I started out with an organization like yours, with a minimal base environment and a few different shared environments, for eg. @MLJ for machine learning packages, @Sym for symbolic libraries, etc., but found that in practice I rarely used those shared environments.

Creating new projects with the particular set of packages we currently need is pretty easy and cheap in Julia, so I always ended up with such specific folder-based environments.

The one exception is an environment I call @InteractiveEnv, that has packages like Revise, Plots, BenchmarkTools, Eyeball, etc. I add this to the end of my LOAD_PATH in my startup.jl within atreplinit, so that I can keep my base @v1.8 environment absolutely minimal while still having access to these packages in the REPL.