Create an implicit environment with package directories

The manual mention that it’s possible to create an implicit environnement with package directories (i.e. directories without Project.toml files). Like @plafer in this post, I tried to replicate the example in the manual with this file structure:

Aardvark/
    src/Aardvark.jl:
        import Bobcat
        import Cobra

Bobcat/
    Project.toml:
        [deps]
        Cobra = "4725e24d-f727-424b-bca0-c4307a3456fa"
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Bobcat.jl:
        import Cobra
        import Dingo

Cobra/
    Project.toml:
        uuid = "4725e24d-f727-424b-bca0-c4307a3456fa"
        [deps]
        Dingo = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Cobra.jl:
        import Dingo

Dingo/
    Project.toml:
        uuid = "7a7925be-828c-4418-bbeb-bac8dfc843bc"

    src/Dingo.jl:
        # no imports

and I have exactly the same error:

ERROR: could not find project file in package at `Aardvark` maybe `subdir` needs to be specified

It seems that this part of the manual was written before the addition of Pkg and is missing some details, am I wrong ?

Thanks!

I think this only works if you explicitly put the “package directory” in the LOAD_PATH, e.g. using something like:

pushfirst!(LOAD_PATH, ".")

I’m not sure when this part of the manual was written, but I’d say the current consensus these days would be to avoid this kind of project organization. Explicit dependencies management in Project.toml/Manifest.toml is more robust IMO, and you can make it work for such a “local ecosystem” in at least two ways that know of:

  • by carefully Pkg.developing each dependency into its dependent (can become tedious), or
  • be registering all projects in a LocalRegistry.
2 Likes

Thank you, it does work by calling pushfirst!(LOAD_PATH, ".") beforehand:

julia> pushfirst!(LOAD_PATH, ".")
4-element Vector{String}:
 "."
 "@"
 "@v#.#"
 "@stdlib"

shell> ls
Aardvark

julia> using Aardvark

julia> 

If it’s really mandatory, it should be mentioned in this part of the manual, no?

But, from my understanding, it should work without this call, since the @ entry in LOAD_PATH means current active environment:


and:

julia> LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

(@v1.10) pkg> activate .
  Activating new project at `~/Documents/Code/test`

(test) pkg> status
Status `~/Documents/Code/test/Project.toml` (empty project)

shell> ls
Aardvark

julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.

Am I missing something?

Recently I stumbled upon this too, and I think what is described there should also be supported.

In particular, for personal projects (or call them mono repos) I can’t see any benefit in

  • having to mess with Pkg.develop, a top-level Project.toml and several Project.tomls in every subfolder I want to have.
  • maintaining a LocalRegistry just for the sake of versioning something that does not need to be versioned in separation.

Why would a single top-level Project.toml not be sufficient to declare that all code that is loaded in this environment should stick to exactly those versions?

3 Likes

My current understanding is that @ always refers to a “project environment” as opposed to a “package directory”, but I’m not actually 100% sure it is the case (and if so I agree that it could be clearer in the documentation).

I was merely trying to explain how I understand things work, and what I believe the current consensus is among the community. But I would personally tend to agree with you. I actually think there would be room for better tooling, especially as far as monorepos are concerned. For example an automated tool could look at a tree of julia projects/packages and

  1. build a list of all packages (characterized by the presence of a valid Project.toml at the root of each of them),
  2. build a map linking each package UUID to its path,
  3. use these information to Pkg.develop the local packages into each of their dependants.

I don’t think it would be too complicated to build such a tool, but I never really had a need for it, so I never did it myself.


EDIT:

I think part of the issue here is: when you look at the code in Aardvark/src/Aardvark.jl, how do you know what Bobcat refers to? This is why I say explicit dependencies declared in a Project.toml file are more robust IMO: you always know at least what you need to look for. (And if we had automated tools to populate the Manifest.toml to know where to look, that’s all the better).

Yes, if I want Aardvark to be a standalone pkg, I will have to declare inside Aardvark/Project.toml where Bobcat is coming from.

But if Aardvark only makes sense inside my mono repo, which might look like

monorepo|⇒ tree
.
├── Aardvark
│   └── src
│       └── Aardvark.jl
├── Bobcat
│   └── src
│       └── Bobcat.jl
├── Cobra
│   └── src
│       └── Cobra.jl
├── Dingo
│   └── src
│       └── Dingo.jl
└── Project.toml

then it should be straightforward to figure out: Bobcat is either another monorepo local pkg, or it is an external pkg and as such needs to be declared in the Project.toml.


EDIT: With this directory setup, running

julia> pushfirst!(LOAD_PATH, ".")

julia> Aardvark

does not work!
I guess that is the behavior that is mentioned in the second paragraph of the linked docs.
Only if you remove Project.toml, it works that way.
So this means my desired monorepo layout above is not supported!

1 Like

Even if Aardvark only makes sense in the local monorepo, it could still require external (registered) dependencies, no? These would need to be declared in a Project.toml. Or, in the case of the directory structure you propose, would you declare all external dependencies for all sub packages in the same top-level Project.toml?

I don’t know of any pure-julia way of doing this, but maybe tools like direnv can help you with this. See for example this blog post:

1 Like

Exactly! Now I’m really lost. It works only if there is no Project.toml in the parrent dir. It does not work anymore if the file is there, even without activating any environnement:

[francis@thinkpad-x1 test]$ ls
Aardvark
[francis@thinkpad-x1 test]$ julia --quiet
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]
julia> push!(LOAD_PATH, ".")
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "."

julia> using Aardvark

julia> exit()
[francis@thinkpad-x1 test]$ >Project.toml
[francis@thinkpad-x1 test]$ ls
Aardvark  Project.toml
[francis@thinkpad-x1 test]$ julia --quiet
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]
julia> push!(LOAD_PATH, ".")
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "."
julia> using Aardvark
ERROR: ArgumentError: Package Aardvark not found in current path.
[...]

Yes, because I should only be able load one version of any external pkg at a time. I personally would not put two subprojects that depend on the same pkg, but different versions, into the same monorepo.

1 Like

@ffevotte’s link above also discusses how LOAD_PATH works. Sorry, did not read that first.


So the ‘problem’ (or intended behavior) is that julia does not literally take what is in LOAD_PATH for loading, but instead it normalizes the entries first. The result you can see with Base.load_path():

shell> ls
Aardvark  Bobcat  Cobra  Dingo

julia> pushfirst!(LOAD_PATH, ".")
4-element Vector{String}:
 "."
 "@"
 "@v#.#"
 "@stdlib"

julia> Base.load_path()
3-element Vector{String}:
 "/tmp/jltest/monorepo/"
 "/home/flo/.julia/environments/v1.10/Project.toml"
 "/home/flo/.julia/juliaup/julia-" ⋯ 20 bytes ⋯ "ux.gnu/share/julia/stdlib/v1.10"

shell> touch Project.toml

shell> ls
Aardvark  Bobcat  Cobra  Dingo	Project.toml

julia> Base.load_path()
3-element Vector{String}:
 "/tmp/jltest/monorepo/Project.toml"
 "/home/flo/.julia/environments/v1.10/Project.toml"
 "/home/flo/.julia/juliaup/julia-" ⋯ 20 bytes ⋯ "ux.gnu/share/julia/stdlib/v1.10"
1 Like

Perhaps something like this is close enough?

├── Aardvark
│   └── src
│       └── Aardvark.jl  # using Bobcat, Example
├── Bobcat
│   └── src
│       └── Bobcat.jl
└── env                  # dedicated subdir for the project env
    ├── Project.toml     # declares dep on Example
    └── Manifest.toml
▷ JULIA_LOAD_PATH="$PWD:@:@v#.#:@stdlib" julia --project=env
julia> using Aardvark
[ Info: Precompiling Aardvark [top-level]

julia> Aardvark.Bobcat
Bobcat

julia> Aardvark.Example
Example
2 Likes

That’s a neat idea.

And now put that unwieldy command into a bash script julia.sh

#!/usr/bin/env bash
thisdir=$(dirname -- "$0")
JULIA_LOAD_PATH="$thisdir:@:@v#.#:@stdlib" julia --project="$thisdir/env" $@

and make it exectuable, chmod +x julia.sh.

I think I can live with that. :slight_smile:

Do you think something similar could be achieved by using the .vscode/settings.json file instead? I tried adding this setting:

{
   "julia.executablePath": "JULIA_LOAD_PATH=".:@:@v#.#:@stdlib" julia"
}

but it did not work.

You could put the relevant command in a shell script, and put the path to that script into the julia.executablePath VSCode setting.

But then I’m not sure whether the LanguageServer will correctly support such a project organization. Again, although it’s a bit more work to setup initially, I really think that in the long run you’d be better off with an explicit project environment.

1 Like

The script in executablePath did not work, but I found another solution that seems to work well. Simply add this line in .vscode/setting.json:

  "julia.additionalArgs": [  
      "-e pushfirst!(LOAD_PATH, pwd())"
  ],

With this, a fresh REPL session in VS Code give:

shell> ls
Aardvark  env

julia> using Aardvark

julia>