Create an implicit environment with package directories

[francis-gagnon]

The manual mention that it’s possible to create an implicit environnement with package directories (i.e. directories without Project.toml files). Like @plafer in this post, I tried to replicate the example in the manual with this file structure:

Aardvark/
    src/Aardvark.jl:
        import Bobcat
        import Cobra

Bobcat/
    Project.toml:
        [deps]
        Cobra = "4725e24d-f727-424b-bca0-c4307a3456fa"
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Bobcat.jl:
        import Cobra
        import Dingo

Cobra/
    Project.toml:
        uuid = "4725e24d-f727-424b-bca0-c4307a3456fa"
        [deps]
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Cobra.jl:
        import Dingo

Dingo/
    Project.toml:
        uuid = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Dingo.jl:
        # no imports

and I have exactly the same error:

ERROR: could not find project file in package at `Aardvark` maybe `subdir` needs to be specified

It seems that this part of the manual was written before the addition of Pkg and is missing some details, am I wrong ?

Thanks!

[ffevotte]

@francis-gagnon

missing some details

I think this only works if you explicitly put the “package directory” in the LOAD_PATH, e.g. using something like:

pushfirst!(LOAD_PATH, ".")

@francis-gagnon

It seems that this part of the manual was written before the addition of Pkg

I’m not sure when this part of the manual was written, but I’d say the current consensus these days would be to avoid this kind of project organization. Explicit dependencies management in Project.toml/Manifest.toml is more robust IMO, and you can make it work for such a “local ecosystem” in at least two ways that know of:

• by carefully Pkg.developing each dependency into its dependent (can become tedious), or
• be registering all projects in a LocalRegistry.

[francis-gagnon]

Thank you, it does work by calling pushfirst!(LOAD_PATH, ".") beforehand:

julia> pushfirst!(LOAD_PATH, ".")
4-element Vector{String}:
 "."
 "@"
 "@v#.#"
 "@stdlib"

shell> ls
Aardvark

julia> using Aardvark

julia> 

If it’s really mandatory, it should be mentioned in this part of the manual, no?

But, from my understanding, it should work without this call, since the @ entry in LOAD_PATH means current active environment:

and:

julia> LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

(@v1.10) pkg> activate .
  Activating new project at `~/Documents/Code/test`

(test) pkg> status
Status `~/Documents/Code/test/Project.toml` (empty project)

shell> ls
Aardvark

julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.

Am I missing something?

[fatteneder]

Recently I stumbled upon this too, and I think what is described there should also be supported.

In particular, for personal projects (or call them mono repos) I can’t see any benefit in

• having to mess with Pkg.develop, a top-level Project.toml and several Project.tomls in every subfolder I want to have.
• maintaining a LocalRegistry just for the sake of versioning something that does not need to be versioned in separation.

Why would a single top-level Project.toml not be sufficient to declare that all code that is loaded in this environment should stick to exactly those versions?

[ffevotte]

@francis-gagnon

But, from my understanding, it should work without this call, since the @ entry in LOAD_PATH means current active environment:

My current understanding is that @ always refers to a “project environment” as opposed to a “package directory”, but I’m not actually 100% sure it is the case (and if so I agree that it could be clearer in the documentation).

[ffevotte]

@fatteneder

for personal projects (or call them mono repos) I can’t see any benefit in

I was merely trying to explain how I understand things work, and what I believe the current consensus is among the community. But I would personally tend to agree with you. I actually think there would be room for better tooling, especially as far as monorepos are concerned. For example an automated tool could look at a tree of julia projects/packages and

1. build a list of all packages (characterized by the presence of a valid Project.toml at the root of each of them),
2. build a map linking each package UUID to its path,
3. use these information to Pkg.develop the local packages into each of their dependants.

I don’t think it would be too complicated to build such a tool, but I never really had a need for it, so I never did it myself.

EDIT:

@fatteneder

Why would a single top-level Project.toml not be sufficient to declare that all code that is loaded in this environment should stick to exactly those versions?

I think part of the issue here is: when you look at the code in Aardvark/src/Aardvark.jl, how do you know what Bobcat refers to? This is why I say explicit dependencies declared in a Project.toml file are more robust IMO: you always know at least what you need to look for. (And if we had automated tools to populate the Manifest.toml to know where to look, that’s all the better).

[fatteneder]

@ffevotte

I think part of the issue here is: when you look at the code in Aardvark/src/Aardvark.jl, how do you know what Bobcat refers to?

Yes, if I want Aardvark to be a standalone pkg, I will have to declare inside Aardvark/Project.toml where Bobcat is coming from.

But if Aardvark only makes sense inside my mono repo, which might look like

monorepo|⇒ tree
.
├── Aardvark
│   └── src
│       └── Aardvark.jl
├── Bobcat
│   └── src
│       └── Bobcat.jl
├── Cobra
│   └── src
│       └── Cobra.jl
├── Dingo
│   └── src
│       └── Dingo.jl
└── Project.toml

then it should be straightforward to figure out: Bobcat is either another monorepo local pkg, or it is an external pkg and as such needs to be declared in the Project.toml.

---

EDIT: With this directory setup, running

julia> pushfirst!(LOAD_PATH, ".")

julia> Aardvark

does not work!
I guess that is the behavior that is mentioned in the second paragraph of the linked docs.
Only if you remove Project.toml, it works that way.
So this means my desired monorepo layout above is not supported!

[ffevotte]

Even if Aardvark only makes sense in the local monorepo, it could still require external (registered) dependencies, no? These would need to be declared in a Project.toml. Or, in the case of the directory structure you propose, would you declare all external dependencies for all sub packages in the same top-level Project.toml?

@fatteneder

I wonder whether this can be automated.

I don’t know of any pure-julia way of doing this, but maybe tools like direnv can help you with this. See for example this blog post:

Project specific Julia configuration with direnv | Fredrik Ekre

[francis-gagnon]

@fatteneder

EDIT: With this directory setup, running

julia> pushfirst!(LOAD_PATH, ".")

julia> Aardvark

does not work!
I guess that is the behavior that is mentioned in the second paragraph of the linked docs.
Only if you remove Project.toml, it works that way.

Exactly! Now I’m really lost. It works only if there is no Project.toml in the parrent dir. It does not work anymore if the file is there, even without activating any environnement:

[francis@thinkpad-x1 test]$ ls
Aardvark
[francis@thinkpad-x1 test]$ julia --quiet
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]
julia> push!(LOAD_PATH, ".")
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "."

julia> using Aardvark

julia> exit()
[francis@thinkpad-x1 test]$ >Project.toml
[francis@thinkpad-x1 test]$ ls
Aardvark  Project.toml
[francis@thinkpad-x1 test]$ julia --quiet
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]
julia> push!(LOAD_PATH, ".")
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "."
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]

[fatteneder]

@ffevotte

Or, in the case of the directory structure you propose, would you declare all external dependencies for all sub packages in the same top-level Project.toml?

Yes, because I should only be able load one version of any external pkg at a time. I personally would not put two subprojects that depend on the same pkg, but different versions, into the same monorepo.

[fatteneder]

@ffevotte’s link above also discusses how LOAD_PATH works. Sorry, did not read that first.

---

So the ‘problem’ (or intended behavior) is that julia does not literally take what is in LOAD_PATH for loading, but instead it normalizes the entries first. The result you can see with Base.load_path():

shell> ls
Aardvark  Bobcat  Cobra  Dingo

julia> pushfirst!(LOAD_PATH, ".")
4-element Vector{String}:
 "."
 "@"
 "@v#.#"
 "@stdlib"

julia> Base.load_path()
3-element Vector{String}:
 "/tmp/jltest/monorepo/"
 "/home/flo/.julia/environments/v1.10/Project.toml"
 "/home/flo/.julia/juliaup/julia-" ⋯ 20 bytes ⋯ "ux.gnu/share/julia/stdlib/v1.10"

shell> touch Project.toml

shell> ls
Aardvark  Bobcat  Cobra  Dingo	Project.toml

julia> Base.load_path()
3-element Vector{String}:
 "/tmp/jltest/monorepo/Project.toml"
 "/home/flo/.julia/environments/v1.10/Project.toml"
 "/home/flo/.julia/juliaup/julia-" ⋯ 20 bytes ⋯ "ux.gnu/share/julia/stdlib/v1.10"

[ffevotte]

@fatteneder

So this means my desired monorepo layout above is not supported!

Perhaps something like this is close enough?

├── Aardvark
│   └── src
│       └── Aardvark.jl  # using Bobcat, Example
├── Bobcat
│   └── src
│       └── Bobcat.jl
└── env                  # dedicated subdir for the project env
    ├── Project.toml     # declares dep on Example
    └── Manifest.toml

▷ JULIA_LOAD_PATH="$PWD:@:@v#.#:@stdlib" julia --project=env
julia> using Aardvark
[ Info: Precompiling Aardvark [top-level]

julia> Aardvark.Bobcat
Bobcat

julia> Aardvark.Example
Example

[fatteneder]

That’s a neat idea.

And now put that unwieldy command into a bash script julia.sh

#!/usr/bin/env bash
thisdir=$(dirname -- "$0")
JULIA_LOAD_PATH="$thisdir:@:@v#.#:@stdlib" julia --project="$thisdir/env" $@

and make it exectuable, chmod +x julia.sh.

I think I can live with that.

[francis-gagnon]

@ffevotte

JULIA_LOAD_PATH="$PWD:@:@v#.#:@stdlib" julia

Do you think something similar could be achieved by using the .vscode/settings.json file instead? I tried adding this setting:

{
   "julia.executablePath": "JULIA_LOAD_PATH=".:@:@v#.#:@stdlib" julia"
}

but it did not work.

[ffevotte]

@francis-gagnon

Do you think something similar could be achieved by using the .vscode/settings.json file instead?

You could put the relevant command in a shell script, and put the path to that script into the julia.executablePath VSCode setting.

But then I’m not sure whether the LanguageServer will correctly support such a project organization. Again, although it’s a bit more work to setup initially, I really think that in the long run you’d be better off with an explicit project environment.

[francis-gagnon]

The script in executablePath did not work, but I found another solution that seems to work well. Simply add this line in .vscode/setting.json:

  "julia.additionalArgs": [  
      "-e pushfirst!(LOAD_PATH, pwd())"
  ],

With this, a fresh REPL session in VS Code give:

shell> ls
Aardvark  env

julia> using Aardvark

julia>