Package directories query

New to Julia
1

This question is based from information present in this documentation page:

Specifically, under the Package directories section, the following is written:

Which dependencies a package in a package directory can import depends on whether the package contains a project file

If it does not have a project file, it can import any top-level package—i.e. the same packages that can be loaded in Main or the REPL

My interpretation might be slightly wrong, however my understanding is like this:

  • A “Package directory” is a directory containing a set of 1 or more Julia Packages
  • A Julia Package is a directory named X containing either of the following files: X.jl, X/src/X.jl or X.jl/src/X.jl
  • A “top-level package” is a package within the “Package directory”

It might be that my understanding of the last point is incorrect.

I tried to build a Package directory with two packages, one of which depends on the other.

To do this, I created a folder, changed into this directory and ran

pkg> generate PackageA
pkg> generate PackageB

I then changed the contents of PackageB/src/PackageB.jl:

module PackageB

using PackageA

greet() = print("Hello World!")

end # module PackageB

I tried to run this with Julia.

$ julia PackageB/src/PackageB.jl

and obtained the following error

ERROR: LoadError: ArgumentError: Package PackageA not found in current path.

Julia does not know what PackageA is. I have three questions:

  1. Is this the expected behaviour?
  2. What did I misunderstand?
  3. What should I have done to make this work? (To add a dependency between PackageA and PackageB.)
2

It’s unfortunately not mentioned explicitly in the manual, but you need to manually add the package directory to the LOAD_PATH. Take a look at this post: Using or import search current directory? - #66 by danielwe

1 Like
3
  1. Yes, that’s expected behavior.
  2. Maybe I’ll come back to this later.
  3. PackageB doesn’t know anything about PackageA, so you have to find a way to tell it.

Modify LOAD_PATH

I often use this technique in Pluto notebooks that use unpublished libraries that are sitting on my local machine.

julia> push!(LOAD_PATH, "/path/to/PackageA/Project.toml")

julia> using PackageA

Add a dependency to PackageB

This is another way to tell PackageB about the existence of PackageA. Note that PackageB has to be activated in your REPL for this to work.

julia> ]

(PackageB) pkg> add "/path/to/PackageA.jl"

<BackSpace>

julia> using PackageA
1 Like
4

add only works when PackageA is a full-blown package, that is, a git repository containing a Project.toml with a name and UUID. The point of package directories (the section of the manual @world-peace is linking to) is to support loading code from locations that don’t contain all this boilerplate, even just a single julia file. For that, you need to modify LOAD_PATH manually.

2 Likes
5

That is true, and it’s good to make that explicitly known.

6

I would recommend against using package directories at all. The feature was included in Julia 1.0 mainly because the completely redesigned package manager was new and it wasn’t clear how the whole project+manifest thing would pan out, and I wanted people who already been using Julia’s previous package system to have a smooth transition. As it turns out the project+manifest thing works great and people have no real need for package directories at this point.

What you want to do instead is use the Pkg dev feature. After generating the packages, do this:

  • shell> cd PackageB
  • pkg> activate .
  • pkg> dev ../PackageA

After that you can edit PackageA and if you’re using Revise, it will pick up the changes as you make them. Doing pkg> add ../PackageA works similarly, but that only works if PackageA is a git repo and it makes a static snapshot of PackageA that’s stored in ~/.julia/packages/PackageA and referenced by git tree-SHA1. (It’s actually treating the path ../PackageA as a git URL, which is why it has to be a git repo. IMO this is kind of a misfeature and you should be required to use a file:/// URL if you want to do that.)

9 Likes
7

Thanks for your suggestion.

I thought that it was against best practices to modify the environment variables such as JULIA_LOAD_PATH, JULIA_DEPOT_PATH, etc.

Or, are you suggesting modifying LOAD_PATH another way?

I’m not sure I like the concept of “bootstrapping it” from within every other file by manually hacking this variable with (potentially a long list of) push!(LOAD_PATH, ...) calls.

If I have 50 packages to load, it’s not ideal to have to put 50 lines of push!(LOAD_PATH, ...) at the top of each file.

Surely there’s got to be a better way?

8

To add clarification here, my project directory structure looks like this:

my-repo-name/
  .git
  PackageA/
  PackageB/
  ... etc ...

So it is part of a Git repository, however, each of the individual packages themselves do not correspond 1:1 with a Git repository, in other words, each of the package directories are not themselves a single Git repo.

Hope that helps. It’s a “Package Directory” structure, meaning a folder, which is a Git repo, with multiple Packages inside.

9

Thanks for your helpful response. This is a productionize app, meaning it gets deployed using a Docker container to a server somewhere.

I’m pretty sure therefore the is no advantage to using Revise.jl and I can just skip that part without affecting anything else? Thought it worth to check.

The problem with this approach is I don’t know how I would productionize this.

I could explicitly call those commands as part of a Docker container build, but that doesn’t feel like the right way to do it.

The reason is that the dependency relationships are then not part of the code repository but described in a Dockerfile which is stored elsewhere. (Different repo.)

This kind of cross-dependencies thing is less than ideal.

I will think if there is something slightly different which can be done here. Do you have any thoughts on this?

10

Maybe a better question to ask is what does dev ../PackageA actually do?

11

Not sure where you got this idea. It’s a quite frustrating to receive this series of very narrow, technical questions without greater context of what you’re trying to do, spread out across many posts. More context seems to be coming out here than you’ve availed us of before, but I think I’ve hit my limit engaging with this. Maybe someone else has the time and energy. My recommendation if you want effective help: give the big picture of what you’re trying to do and why and be receptive to alternate ways to accomplish your end goal.

3 Likes
12

I’m not sure it makes a difference here but in contexts where Julia packages relate to git, it is not required that they are at the top of the repository but can be in any subdirectory. For some operations you need to specify the location with a subdir argument.

13

There isn’t a “bigger picture” than what is written in the first post.

To summarize the situation, I have an existing repository which I am trying to restructure into a repository of multiple Packages. Hence the example involving PackageA and PackageB, which is just a “minimal working example” of the “bigger picture”.

Nothing is being hidden from you, the details have just been abstracted to make the question manageable.

14

You just gave more of the picture later on, so that’s not actually the case. The pattern here is: “Here’s a problem, please help.” Which you get an answer to, but your reply is “Oh, but I can’t do that because of these other things I didn’t tell you before.” You’re effectively asking people to solve a problem and then re-solve the problem again each time you add another constraint. You also never seem particularly satisfied with any answers, which may be because we can’t actually solve your real problem since we don’t know the full context.

4 Likes
15

If you’re not interested in helping, there is no need to continue to comment on the thread.

Certainly it is pointless and unreasonable to try and somehow frame the situation as if I am at fault here.

If I realize I did not provide enough information initially, I will provide more details. This is just how the world works.

You provided some suggestions, and perhaps in some context your suggestions would have been a satisfactory solution.

For example, for the (probably) average (?) Julia developer, who does some data science, and runs everything from their REPL, there would be nothing wrong with what you have suggested.

However, in this context, although your suggestion will work, it also has shortcomings. This is why I provided you with further details, so that you could understand the shortcomings and problems which would be caused by your idea.

There is no need to be defensive about it. It’s fine to have an idea, and then later learn that idea does not work well in certain contexts due to assumptions you have made which did not happen to apply here.

16

I’m trying to explain why you’re not getting the kind of help you want and why this is also frustrating for people trying to help you. This is a nearly textbook example of the XY problem that @stevengj mentioned in another thread. Your actual goal seems to be to develop an application in a git monorepo with component packages in that same repo (is that it or are there more constraints?). You seem to have concluded that package directories are the way to do this, so you’re asking detailed technical questions about how to do various increasingly awkward things with package directories. Which people can answer but the gist of most answers is “You probably don’t want to use package directories, what are you actually trying to do?” Anyway, there are many people who have developed applications in git monorepos without using package directories, so you could try asking about that instead.

4 Likes
17

The problem here is that you want me to ask a different question to the question which has been asked. What you mention here is asking a totally separate question.

I am happy to have another conversation elsewhere and debate the pros-and-cons of multiple packages vs some other structure for a repository. But this is not what I am asking.

No, it is not an example of an XY problem. It is an example of an unwillingness to read problem.

18

When you do add ../PackageA you get an entry like this in your Manifest file:

[[deps.PackageA]]
git-tree-sha1 = "31604fd54e5a488ba22fcca12602405740a2ae59"
repo-rev = "main"
repo-url = "../PackageA"
uuid = "c5788b5b-9078-464d-8306-3cfa814b54ce"
version = "0.1.0"

There’s a repo-url entry with ../PackageA as its value, indicating that’s where the package “upstream repo” can be found. There’s also a uuid indicating the identity of the package and a git-tree-sha1 giving the tree hash of the code to use. The actual code is found by looking in ~/.julia/packages/PackageA/Wd1MK (the last part is computed as a function of uuid and git-tree-sha1) which is where the code is installed after cloning a copy of the repo found at ../PackageA.

When you do dev ../PackageA you instead get an entry like this:

[[deps.PackageA]]
path = "../PackageA"
uuid = "c5788b5b-9078-464d-8306-3cfa814b54ce"
version = "0.1.0"

Instead of a repo-url entry, there’s a path entry indicating that the actual source of the package can be found at ../PackageA. There is no git-tree-sha1 entry since the content at the path location is expected to change, so any tree hash would become outdated. This is what a dev entry generally looks like.

What you probably want to do is have a root project file for your application and dev all of your internal package dependencies like this and check that into the repo. If you insist on using a package directory instead, there’s no need to push all the dependency directories into the LOAD_PATH—in fact, that won’t work—you only need to push the one directory that contains all the packages into the LOAD_PATH. But again, that’s not recommended—it’s better to use the dev mechanism. There’s nothing wrong with modifying your load path or depot path, that’s why the variables exist. It’s just not necessary for what you’re trying to do.

2 Likes
19

First let me say thank you for your extensive comments. This is very helpful and I appreciate it.

On this point, unfortunately I don’t quite know enough to follow this. Sorry for my ignorance, I just don’t know enough yet to fully understand this.

Allow me to ask a few questions -

  • By a root project file, my understanding is this would be a Project.toml file in the root of the whole project directory. This is the same directory where each of the packages would live. So it looks like
my-project-root/
  Project.toml
  PackageA/
  PackageB/
  • dev all internal packages just means run julia --project=. in the my-project-root directory, enter the pkg REPL, and do dev ./PackageA, dev ./PackageB
  • If you insist on using a package directory instead - this is where I get lost. What I have described above is a Package Directory, AFAIK ?
  • no need to push all the dependency directories into the LOAD_PATH - again a bit lost here - are you essentially saying not to follow the suggestion from @g-gundam ?
  • in fact, that won’t work—you only need to push the one directory that contains all the packages into the LOAD_PATH - I (think I) understand this. LOAD_PATH would be a directory containing each of the packages. So this is sufficient, rather than descending into each of them individually.
  • *** But again, that’s not recommended—it’s better to use the dev mechanism*** - fine, it seems we agree on this point? Better to avoid modifying JULIA_LOAD_PATH, JULIA_DEPOT_PATH and LOAD_PATH if possible?
  • It’s just not necessary for what you’re trying to do. - Ok fine, I agree a local environment with dev seems like a better solution.
20

Yes, that’s right. And it could be laid out like that. But one of the advantages of the dev approach is that it doesn’t matter where you put the dependencies—you can structure it however you want to as long as various path entries point to the correct locations.

Yes, but you can also activate the packages themselves and add dependencies between them in the same way. For example, if PackageB depends on PackageA then you would cd into PackageB, activate it and do pkg> dev ../PackageA in order to add a dependency on PackageA.

No, a “Package Directory” is a directory containing package that is put into the load path. In this approach you do not modify the load path.

You were talking about doing this:

push!(LOAD_PATH, "PackageA")
push!(LOAD_PATH, "PackageB")

and so on for each dependency. This will not work because it would cause code loading to look for packages inside of the PackageA and PackageB directories, which is not where you would find them. What you would do if you were using this approach is simply push!(LOAD_PATH, ".") because the top-level directory is the “Package Directory”, ie a directory that contains packages (which are directories, typically).

Exactly.

Yes, in which case you are not using the Package Directory mechanism, you are putting the path to each dependency explicitly in each manifest that needs it.

Here’s the full layout of an example I generated:

MyProject:
Manifest.toml  PackageA/  PackageB/  Project.toml

MyProject/PackageA:
Manifest.toml Project.toml  src/

MyProject/PackageA/src:
PackageA.jl

MyProject/PackageB:
Manifest.toml  Project.toml  src/

MyProject/PackageB/src:
PackageB.jl

Example contents of various project and manifest files…

MyProject/Project.toml:

[deps]
PackageA = "2556dfe6-fba1-45c8-ae61-24ec43e93209"
PackageB = "f533fac8-037c-4ec8-a4c3-8f0aa0ff58a3"

MyProject/Manifest.toml:

# This file is machine-generated - editing it directly is not advised

julia_version = "1.11.2"
manifest_format = "2.0"
project_hash = "bd82cac20d9e29e79f49f82e3d4d848b38c6d981"

[[deps.PackageA]]
path = "PackageA"
uuid = "2556dfe6-fba1-45c8-ae61-24ec43e93209"
version = "0.1.0"

[[deps.PackageB]]
path = "PackageB"
uuid = "f533fac8-037c-4ec8-a4c3-8f0aa0ff58a3"
version = "0.1.0"

MyProject/PackageA/Project.toml:

name = "PackageA"
uuid = "2556dfe6-fba1-45c8-ae61-24ec43e93209"
authors = ["Stefan Karpinski <stefan@karpinski.org>"]
version = "0.1.0"

MyProject/PackageA/Manifest.toml:

# This file is machine-generated - editing it directly is not advised

julia_version = "1.11.2"
manifest_format = "2.0"
project_hash = "da39a3ee5e6b4b0d3255bfef95601890afd80709"

[deps]

MyProject/PackageB/Project.toml:

name = "PackageB"
uuid = "f533fac8-037c-4ec8-a4c3-8f0aa0ff58a3"
authors = ["Stefan Karpinski <stefan@karpinski.org>"]
version = "0.1.0"

[deps]
PackageA = "2556dfe6-fba1-45c8-ae61-24ec43e93209"

MyProject/PackageB/Manifest.toml:

# This file is machine-generated - editing it directly is not advised

julia_version = "1.11.2"
manifest_format = "2.0"
project_hash = "6d002591c1ced36dadf9d723d3ad71052e50f18d"

[[deps.PackageA]]
path = "../PackageA"
uuid = "2556dfe6-fba1-45c8-ae61-24ec43e93209"
version = "0.1.0"
5 Likes