# Could we make first-class support for packages that are "just files" and not repositories?

**URL:** https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931
**Category:** General Usage
**Tags:** packages, package-manager
**Created:** [April 3, 2020, 12:59am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931 "2020-04-03T00:59:17Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)
#### Post date: [April 3, 2020, 12:59am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/1 "2020-04-03T00:59:17Z")

</div>

I keep circling around the same issue from a variety of different vantage points, and I think that having first-class support for local, unregistered packages (just local files, not repositories) would solve a lot of problems that I see in structuring Julia programs.

We have a repo that has our aircraft simulation in it. The current package manager does a great job of managing our dependence on external packages, like HDF5 and YAML. I love it.

We also have local modules for reusable functionality. One module might be, e.g., Aerodynamics. Another might be CommandTypes.

Sometimes, we’re making a huge simulation with tons of complex stuff in it, and we need Aerodynamics in there somewhere. Sometimes, we’re making a little tool to explore our aerodynamics, so it needs Aerodynamics too. Maybe Aerodynamics needs to know about common types as well, like AileronCommand, and lots of other things need to know about the AileronCommand.

If the AileronCommand is part of a local module, then somehow Aerodynamics needs to know where that module is loaded, or it needs to load it from its parent, and this creates weird scope issues. See the developing thread in [Import types "above" a module with ..: for consistency and brevity?](https://discourse.julialang.org/t/import-types-above-a-module-with-for-consistency-and-brevity/36754/).

If the AileronCommand is part of a local, unregistered package, then everything that needs to know about it can just do `using CommandTypes`, where `CommandTypes` is available via `]dev path/to/CommandTypes`. However, if Aerodynamics is also a local, unregistered package, then not only does it need to do `]dev CommonTypes` (which is fine), but anyone that wants to use Aerodynamics _also_ has to do `]dev CommonTypes`. That is, they have to resolve all local dependencies manually. See ["Unsatisfiable requirements" when local packages use local packages (fixed in v1.4)](https://discourse.julialang.org/t/unsatisfiable-requirements-when-local-packages-use-local-packages/36807/). It’s not the end of the world (I just spent all afternoon dealing with Python stuff that was at least this cumbersome), but it is strangely messy for Julia.

If we could just `]add path/to/LocalPackage`, have that work like `]dev`, and just have everything “agree” to use whatever’s in that local path rather than worrying about versions of things that have no versions because they don’t have their own git repos, then I think this would all just work. (I recognize that it’s likely that I’m missing something here, but it sure seems like this would all work, and there seem to be others have similar problems.)

Just to be super-duper clear about this: having tons of packages in different repos is a good workflow for some stuff, and this seems to be the Julian way currently, but it doesn’t make sense for a lot of other stuff. Some projects are going to have everything in one repo, especially when Julia isn’t the only language (Julia is a small minority in our code base _for now_), so in those cases, local, unregistered packages seems like a good way forward.

What do folks think?

---

<div class="post-metadata">

### Author: ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [April 3, 2020, 5:34am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/2 "2020-04-03T05:34:14Z")

</div>

> [@tuckermcclure](#):
>
> just have everything “agree” to use whatever’s in that local path rather than worrying about versions of things that have no versions because they don’t have their own git repos

I am not sure I understand the distinction between local files and git repos here. Both just contain code, and can (should) have a version in the `Project.toml`.

You may be interested in the discussion at

> <https://github.com/JuliaLang/Pkg.jl/issues/810>
>
> It seems impossible currently to safely distribute a registered package with a d…ependence on an unregistered repository. #492 is a "solution" issue, I thought it might be useful to have one that carefully documents "the problem" so that we can link to a concise description from other packages that are currently thinking about this (motivated by https://github.com/JuliaImages/ImageDistances.jl/pull/4).
> 
> Here's an annotated session in which, starting from a fresh package repository, I try to create a new package that depends on an unregistered repository. Here's an English description of the steps I go through:
> 
> 1. Launch julia so that it uses a blank environment, and install \`ColorTypes\` just to populate various directories
> 2. \`generate\` a new package \`Foo\`. (Alternatively this could be an existing package to which we want to add something new.)
> 3. Make \`Foo\` depend on https://github.com/robertfeldt/EarthMoversDistance.jl, which is not registered
> 4. Try to use \`Foo\` in the \*general\* environment. This is the part that fails.
> 5. Show that it works OK inside \`Foo\`'s environment.
> 
> If \`Foo\` is a "workhorse package" that people use in their primary environment, this makes it impossible to introduce the dependency without telling all the users to directly \`add\` or \`dev\` EarthMoversDistance in their primary environment. That's an extra step that requires a trip to the README, something people don't generally have to do for packages that depend only on registered packages.
> 
> \`\`\`julia
> tim@diva:/tmp$ JULIA\_DEPOT\_PATH=/tmp/pkgs julia -q
> (v1.0) pkg\> dev ColorTypes
> Cloning default registries into /tmp/pkgs/registries
> Cloning registry General from "https://github.com/JuliaRegistries/General.git"
> Updating registry at \`/tmp/pkgs/registries/General\`
> Updating git-repo \`https://github.com/JuliaRegistries/General.git\`
> Cloning git-repo \`https://github.com/JuliaGraphics/ColorTypes.jl.git\`
> Updating git-repo \`https://github.com/JuliaGraphics/ColorTypes.jl.git\`
> Resolving package versions...
> Installed FixedPointNumbers ─ v0.5.3
> Updating \`/tmp/pkgs/environments/v1.0/Project.toml\`
> \[3da002f7\] + ColorTypes v0.7.5+ \[\`/tmp/pkgs/dev/ColorTypes\`\]
> Updating \`/tmp/pkgs/environments/v1.0/Manifest.toml\`
> \[3da002f7\] + ColorTypes v0.7.5+ \[\`/tmp/pkgs/dev/ColorTypes\`\]
> \[53c48c17\] + FixedPointNumbers v0.5.3
> \[2a0f44e3\] + Base64 
> \[ade2ca70\] + Dates 
> \[8ba89e20\] + Distributed 
> \[b77e0a4c\] + InteractiveUtils 
> \[76f85450\] + LibGit2 
> \[8f399da3\] + Libdl 
> \[37e2e46d\] + LinearAlgebra 
> \[56ddb016\] + Logging 
> \[d6f4376e\] + Markdown 
> \[44cfe95a\] + Pkg 
> \[de0858da\] + Printf 
> \[3fa0cd96\] + REPL 
> \[9a3f8284\] + Random 
> \[ea8e919c\] + SHA 
> \[9e88b42a\] + Serialization 
> \[6462fe0b\] + Sockets 
> \[8dfed614\] + Test 
> \[cf7118a7\] + UUIDs 
> \[4ec0a83e\] + Unicode 
> 
> shell\> cd pkgs/dev
> /tmp/pkgs/dev
> 
> (v1.0) pkg\> generate Foo
> Generating project Foo:
> Foo/Project.toml
> Foo/src/Foo.jl
> 
> shell\> cd Foo
> /tmp/pkgs/dev/Foo
> 
> (v1.0) pkg\> activate .
> 
> (Foo) pkg\> add https://github.com/robertfeldt/EarthMoversDistance.jl.git
> Cloning git-repo \`https://github.com/robertfeldt/EarthMoversDistance.jl.git\`
> Updating git-repo \`https://github.com/robertfeldt/EarthMoversDistance.jl.git\`
> \[ Info: Assigning UUID e1737d0f-dfce-5227-9318-d7961fac91f7 to EarthMoversDistance
> Resolving package versions...
> Updating \`/tmp/pkgs/dev/Foo/Project.toml\`
> \[e1737d0f\] + EarthMoversDistance v0.0.0 #master (https://github.com/robertfeldt/EarthMoversDistance.jl.git)
> Updating \`/tmp/pkgs/dev/Foo/Manifest.toml\`
> \[e1737d0f\] + EarthMoversDistance v0.0.0 #master (https://github.com/robertfeldt/EarthMoversDistance.jl.git)
> \[2a0f44e3\] + Base64 
> \[8ba89e20\] + Distributed 
> \[b77e0a4c\] + InteractiveUtils 
> \[8f399da3\] + Libdl 
> \[37e2e46d\] + LinearAlgebra 
> \[56ddb016\] + Logging 
> \[d6f4376e\] + Markdown 
> \[9a3f8284\] + Random 
> \[9e88b42a\] + Serialization 
> \[6462fe0b\] + Sockets 
> \[8dfed614\] + Test 
> Building EarthMoversDistance → \`/tmp/pkgs/packages/EarthMoversDistance/LbTWZ/deps/build.log\`
> 
> (Foo) pkg\> activate
> 
> shell\> pwd
> /tmp/pkgs/dev/Foo
> 
> shell\> cd ../..
> /tmp/pkgs
> 
> (v1.0) pkg\> resolve
> Resolving package versions...
> Updating \`/tmp/pkgs/environments/v1.0/Project.toml\`
> \[no changes\]
> Updating \`/tmp/pkgs/environments/v1.0/Manifest.toml\`
> \[no changes\]
> 
> (v1.0) pkg\> dev /tmp/pkgs/dev/Foo
> Resolving package versions...
> ERROR: Unsatisfiable requirements detected for package EarthMoversDistance \[e1737d0f\]:
> EarthMoversDistance \[e1737d0f\] log:
> ├─EarthMoversDistance \[e1737d0f\] has no known versions!
> └─restricted to versions \* by Foo \[d11d1dc6\] — no versions left
> └─Foo \[d11d1dc6\] log:
> ├─possible versions are: 0.1.0 or uninstalled
> └─Foo \[d11d1dc6\] is fixed to version 0.1.0
> 
> shell\> cd dev/Foo
> /tmp/pkgs/dev/Foo
> 
> (v1.0) pkg\> activate .
> 
> julia\> using Foo
> \[Info: Precompiling Foo \[d11d1dc6-ccaf-11e8-00ae-ebbedd466ca6\]
> 
> julia\> Foo.EarthMoversDistance.emd(\[0.2, 0.7, 0.1\], \[0.5, 0.5\], rand(3, 2))
> 0.4461155929675431
> \`\`\`
> 
> Here is the state of various files at the end of this process:
> \`\`\`sh
> tim@diva:/tmp/pkgs$ cat environments/v1.0/Project.toml
> \[deps\]
> ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
> tim@diva:/tmp/pkgs$ cat environments/v1.0/Manifest.toml 
> \[\[Base64\]\]
> uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
> 
> \[\[ColorTypes\]\]
> deps = \["FixedPointNumbers", "Random", "Test"\]
> path = "/tmp/pkgs/dev/ColorTypes"
> uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
> version = "0.7.5+"
> 
> \[\[Dates\]\]
> deps = \["Printf"\]
> uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
> 
> \[\[Distributed\]\]
> deps = \["LinearAlgebra", "Random", "Serialization", "Sockets"\]
> uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
> 
> \[\[FixedPointNumbers\]\]
> deps = \["Pkg", "Test"\]
> git-tree-sha1 = "b8045033701c3b10bf2324d7203404be7aef88ba"
> uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
> version = "0.5.3"
> 
> \[\[InteractiveUtils\]\]
> deps = \["LinearAlgebra", "Markdown"\]
> uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
> 
> \[\[LibGit2\]\]
> uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"
> 
> \[\[Libdl\]\]
> uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
> 
> \[\[LinearAlgebra\]\]
> deps = \["Libdl"\]
> uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
> 
> \[\[Logging\]\]
> uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"
> 
> \[\[Markdown\]\]
> deps = \["Base64"\]
> uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
> 
> \[\[Pkg\]\]
> deps = \["Dates", "LibGit2", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"\]
> uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
> 
> \[\[Printf\]\]
> deps = \["Unicode"\]
> uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
> 
> \[\[REPL\]\]
> deps = \["InteractiveUtils", "Markdown", "Sockets"\]
> uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
> 
> \[\[Random\]\]
> deps = \["Serialization"\]
> uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
> 
> \[\[SHA\]\]
> uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
> 
> \[\[Serialization\]\]
> uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
> 
> \[\[Sockets\]\]
> uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
> 
> \[\[Test\]\]
> deps = \["Distributed", "InteractiveUtils", "Logging", "Random"\]
> uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
> 
> \[\[UUIDs\]\]
> deps = \["Random"\]
> uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
> 
> \[\[Unicode\]\]
> uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
> tim@diva:/tmp/pkgs$ cat dev/Foo/Project.toml 
> name = "Foo"
> uuid = "d11d1dc6-ccaf-11e8-00ae-ebbedd466ca6"
> authors = \["Tim Holy \<tim.holy@gmail.com\>"\]
> version = "0.1.0"
> 
> \[deps\]
> EarthMoversDistance = "e1737d0f-dfce-5227-9318-d7961fac91f7"
> tim@diva:/tmp/pkgs$ cat dev/Foo/Manifest.toml 
> \[\[Base64\]\]
> uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
> 
> \[\[Distributed\]\]
> deps = \["LinearAlgebra", "Random", "Serialization", "Sockets"\]
> uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
> 
> \[\[EarthMoversDistance\]\]
> deps = \["Libdl", "Test"\]
> git-tree-sha1 = "8278e0f422eb8a884e385c74fa015573eb36df8a"
> repo-rev = "master"
> repo-url = "https://github.com/robertfeldt/EarthMoversDistance.jl.git"
> uuid = "e1737d0f-dfce-5227-9318-d7961fac91f7"
> version = "0.0.0"
> 
> \[\[InteractiveUtils\]\]
> deps = \["LinearAlgebra", "Markdown"\]
> uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
> 
> \[\[Libdl\]\]
> uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
> 
> \[\[LinearAlgebra\]\]
> deps = \["Libdl"\]
> uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
> 
> \[\[Logging\]\]
> uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"
> 
> \[\[Markdown\]\]
> deps = \["Base64"\]
> uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
> 
> \[\[Random\]\]
> deps = \["Serialization"\]
> uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
> 
> \[\[Serialization\]\]
> uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
> 
> \[\[Sockets\]\]
> uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
> 
> \[\[Test\]\]
> deps = \["Distributed", "InteractiveUtils", "Logging", "Random"\]
> uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
> tim@diva:/tmp/pkgs$ 
> \`\`\`

and many similar discussions on this forum.

TL;DR: the solution is a local/custom registry. Nice tooling is evolving for it, eg

> **[GitHub - GunnarFarneback/LocalRegistry.jl: Create and maintain local...](https://github.com/GunnarFarneback/LocalRegistry.jl)**
>
> Create and maintain local registries for Julia packages. - GitHub - GunnarFarneback/LocalRegistry.jl: Create and maintain local registries for Julia packages.

---

<div class="post-metadata">

### Author: ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)
#### Post date: [April 3, 2020, 9:13pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/3 "2020-04-03T21:13:16Z")

</div>

Thanks @Tamas_Papp, but I’m confused. For LocalRegistry, it seems like packages still need to be individual git repos, no? I might need to clarify my post a bit:

The individual repos thing doesn’t work for me, or for other people that I talk to in engineering. I need to have all of my code related to the aircraft simulation in one big repo; it’s in there with a bunch of other code (Julia is only a small part of this project). The way packages are brought in in general (e.g., any module anywhere can `using CommonTypes`, and they all get the same types) is very attractive, but I’m going to need the CommonTypes package to just be some code located in some subdirectory of my repo, not its own git repo.

`]dev path/to/CommonTypes` works well, but only for one level. Once you have multiple `]dev`’d packages, you end up having to resolve dependence manually, as mentioned in ["Unsatisfiable requirements" when local packages use local packages (fixed in v1.4)](https://discourse.julialang.org/t/unsatisfiable-requirements-when-local-packages-use-local-packages/36807/).

Maybe what’s not clear in my original post is that by “local, unregistered package” I mean just a subdirectory that’s structured like a package in my overall repo:

```julia
/.git
/analysis # All the other code in the repo...
/drivers
/flight
/sim # The Julia part
  /Aerodynamics # A local package
    /src
    Project.toml # Depend on CommonTypes
    Manifest.toml
  /CommandTypes # Another local package
    /src
    Project.toml
    Manifest.toml
  /HugeSim
    /src
    Project.toml # Directly depends on Aerodynamics, but not on CommonTypes
    Manifest.toml
  /SmallSim
  /StudyOfAerodynamics # etc.
  ...

```

As far as I can tell, using these local paths as packages doesn’t really work, because `]dev LocalPackage` doesn’t resolve dependencies, so, for this example, `HugeSim` would _also_ need to `]dev ../CommonTypes` prior to including the things that depend on CommonTypes, like `]dev ../Aerodynamics`. So, each project moving up has to explicitly write out all of the local dependencies for anything it uses, and for anything that _those_ use, etc.

**If local dependency resolution worked for `]dev`, then this whole thing would work perfectly as far as I can tell.** That’s probably the most important thing to highlight.

For what it’s worth, I also think that using local packages would be an easy way for folks (at least people like me) to learn about creating a program structure that uses multiple packages. It just helps minimize the boilerplate.

Ok, I hope that clears up my intent rather than just drowning everything in detail. Please let me know if this makes sense!

---

<div class="post-metadata">

### Author: ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [April 3, 2020, 10:06pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/4 "2020-04-03T22:06:35Z")

</div>

> [@Tamas\_Papp](#):
>
> I am not sure I understand the distinction between local files and git repos here. Both just contain code, and can (should) have a version in the `Project.toml` .

Strictly speaking, a package does not need to be a git repository : in its most basic form (as created by `Pkg.generate`), it is a repository following a specific, predefined structure. In particular it contains a `Project.toml` file storing a bunch of metadata, including a version number. I think this is what Tucker refers to as “local files”.

However, if I understand correctly, as soon as a package wants to be registered, it _needs_ to be a git repository. This is because, as I understand it, the registry cares about versions, and links each registered version with a particular SHA1 identifying the state of the git repository.

In short, unless I’m mistaken, a package that is not a git repository can not be registered. It can define a version in its `Project.toml` file, but I’m not sure that `Pkg` even looks at it. I don’t know of any other way to use such a package than

1. `Pkg.activate`ing it, or
2. `Pkg.develop`ing it (which essentially tells `Pkg` to ignore any notion of version and take the source files as they are)

---

<div class="post-metadata">

### Author: ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [April 3, 2020, 10:10pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/5 "2020-04-03T22:10:53Z")

</div>

How about just do `] dev ../CommandTypes` at `sim/Aerodynamics` and check-in `sim/Aerodynamics/Manifest.toml`? You can then just do `julia --project` in `sim/Aerodynamics/` directory to start working on it. Then `using Aerodynamics` should just work (after `] instantiate`).

Alternatively, if all Julia packages are compatible, having `sim/Manifest.toml` that `dev`s evrything under it may make sense.

---

<div class="post-metadata">

### Author: ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)
#### Post date: [April 3, 2020, 10:44pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/6 "2020-04-03T22:44:59Z")

</div>

Hi @tkf, I hear you, and that’s exactly what I tried, but it doesn’t really work. See ["Unsatisfiable requirements" when local packages use local packages (fixed in v1.4)](https://discourse.julialang.org/t/unsatisfiable-requirements-when-local-packages-use-local-packages/36807/).

(It _may_ work in v1.4; I haven’t verified yet.)

I attribute the fact that it doesn’t work to the fact that “local packages” like this are not first class citizens, and `]dev` is meant as a short-term thing, not something you build your program on.

---

<div class="post-metadata">

### Author: ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [April 3, 2020, 11:00pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/7 "2020-04-03T23:00:14Z")

</div>

Hmm… Relative `dev` trick is working since 1.0 for me. I’m using this for reproducible test environment like this: [https://github.com/tkf/Transducers.jl/blob/2217a84a7a382d9611a7750ae6628681ebb5c667/test/environments/main/Manifest.toml#L366](https://github.com/tkf/Transducers.jl/blob/2217a84a7a382d9611a7750ae6628681ebb5c667/test/environments/main/Manifest.toml#L366)

Quickly skimming the discussion you linked, I wonder if you missed my point on explicit activation of the project:

> [@tkf](#):
>
> You can then just do `julia --project` in `sim/Aerodynamics/` directory to start working on it.

(or you can also use `] activate sim/Aerodynamics/`)

This is to say, I’m not suggesting to do `] dev sim/Aerodynamics/`; just activate it or a project containing it.

---

<div class="post-metadata">

### Author: ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [April 3, 2020, 11:13pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/8 "2020-04-03T23:13:15Z")

</div>

> [@tkf](#):
>
> Relative `dev` trick is working since 1.0 for me

Yes. What did not work until recently is `dev`ing a package that itself `dev`s another package.

In the example shown above, everything is fine with `Aerodynamics`, which depends on `CommonTypes`. As far as I understand, problems start occurring when working with `HugeSim` and trying to make it depend on `AeroDynamics`.

---

<div class="post-metadata">

### Author: ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [April 3, 2020, 11:49pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/9 "2020-04-03T23:49:27Z")

</div>

Why not just do `] dev ../CommandTypes` at `sim/HugeSim`? If you check in `sim/HugeSim/Manifest.toml`, it just has to be done once.

It should also be possible to develop a tool to automatically `dev` relative dependencies. I did something similar in [GitHub - tkf/Rogue.jl](https://github.com/tkf/Rogue.jl) but for `add`.

---

<div class="post-metadata">

### Author: ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)
#### Post date: [April 4, 2020, 4:59am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/10 "2020-04-04T04:59:20Z")

</div>

> [@ffevotte](#):
>
> What did not work until recently is `dev` ing a package that itself `dev` s another package.

Yes.

> [@tkf](#):
>
> Why not just do `] dev ../CommandTypes` at `sim/HugeSim` ?

Yes, but this amounts to having HugeSim manually resolve all dependencies of all packages that it uses, and that they use, and that they use, etc. Even for a small number of packages, this becomes tedious and fragile very quickly.

> [@tkf](#):
>
> It should also be possible to develop a tool to automatically `dev` relative dependencies

Yes, I think that’s what I’m proposing here, and looking at your Rogue package quickly, it does seem similar. Thanks.

---

<div class="post-metadata">

### Author: ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [April 4, 2020, 8:20am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/11 "2020-04-04T08:20:58Z")

</div>

> [@tuckermcclure](#):
>
> I think that’s what I’m proposing here

Ah, sorry, I missed this point. I agree that developing a tool to cover monorepo approach like this would be great.

---

<div class="post-metadata">

### Author: ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [April 4, 2020, 8:25am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/12 "2020-04-04T08:25:59Z")

</div>

In your opinion, would `Rogue` be an adequate package to host such features?

---

<div class="post-metadata">

### Author: ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [April 4, 2020, 10:52pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/13 "2020-04-04T22:52:47Z")

</div>

I think it is. But it’s still kind of a works-for-me package and not in a releasable quality. The main is reason that it relies on my Python package [`vcslinks`](https://github.com/tkf/vcslinks) and so the installation process is not automatic. So I can’t recommend people to use it or contribute to it.

(If people are interested in using Rogue.jl I should probably remove `vcslinks` or make it optional as it is only used for creating nice commit message. Another option is to port `vcslinks` to Julia but it’s kind of a dull task.)

---

<div class="post-metadata">

### Author: ![tuckermcclure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tuckermcclure/32/1187_2.png) [@tuckermcclure](https://discourse.julialang.org/u/tuckermcclure)
#### Post date: [April 9, 2020, 7:10pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/14 "2020-04-09T19:10:11Z")

</div>

One question I have on this front is how precompilation would be handled. When they’re “just files” in `]dev` mode, how does Julia know when to precompile them? Does it always just use fresh, or does it look at time stamps on files or something (I don’t expect there’s much mechanism for this latter idea, but I could be wrong). In my application right now, for instance, this would matter a great deal; compilation times are very long. Does anyone know what level of effort would be involved here?

Do you all think I should make a GitHub issue about this?

---

<div class="post-metadata">

### Author: ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [April 9, 2020, 7:22pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/15 "2020-04-09T19:22:02Z")

</div>

I just wanted to make sure you are familiar with this usage:

```julia
push!(LOAD_PATH,pwd())
import localfile
[Info: Precompiling localfile [top-level]

```

localfile.jl is just a minimal module:

```julia
module localfile
end

```

If you just wanted to deal with files and bypass the whole `Pkg` system, you could just append the directories to `LOAD_PATH` and then import away, precompilation included!

Here is the documentation:

```julia
help?> LOAD_PATH
search: LOAD_PATH

  LOAD_PATH

  An array of paths for using and import statements to consider as
  project environments or package directories when loading code. It is
  populated based on the JULIA_LOAD_PATH environment variable if set;
  otherwise it defaults to ["@", "@v#.#", "@stdlib"]. Entries starting
  with @ have special meanings:

    • @ refers to the "current active environment", the initial
        value of which is initially determined by the JULIA_PROJECT
        environment variable or the --project command-line option.

    • @stdlib expands to the absolute path of the current Julia
        installation's standard library directory.

    • @name refers to a named environment, which are stored in
        depots (see JULIA_DEPOT_PATH) under the environments
        subdirectory. The user's named environments are stored in
        ~/.julia/environments so @name would refer to the
        environment in ~/.julia/environments/name if it exists and
        contains a Project.toml file. If name contains # characters,
        then they are replaced with the major, minor and patch
        components of the Julia version number. For example, if you
        are running Julia 1.2 then @v#.# expands to @v1.2 and will
        look for an environment by that name, typically at
        ~/.julia/environments/v1.2.

  The fully expanded value of LOAD_PATH that is searched for projects
  and packages can be seen by calling the Base.load_path() function.

  See also: JULIA_LOAD_PATH, JULIA_PROJECT, JULIA_DEPOT_PATH, and Code
  Loading.

```

Full documentation here:

[https://docs.julialang.org/en/v1/manual/code-loading/](https://docs.julialang.org/en/v1/manual/code-loading/)

---

<div class="post-metadata">

### Author: ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [April 9, 2020, 7:26pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/16 "2020-04-09T19:26:58Z")

</div>

Regarding your question about “fresh” and when to “precompile”, you may be interested in the Revise.jl package:

> **[GitHub - timholy/Revise.jl: Automatically update function definitions in a...](https://github.com/timholy/Revise.jl)**
>
> Automatically update function definitions in a running Julia session - GitHub - timholy/Revise.jl: Automatically update function definitions in a running Julia session

---

<div class="post-metadata">

### Author: ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [April 12, 2020, 8:30pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/17 "2020-04-12T20:30:06Z")

</div>

> [@tuckermcclure](#):
>
> One question I have on this front is how precompilation would be handled. When they’re “just files” in `]dev` mode, how does Julia know when to precompile them? Does it always just use fresh, or does it look at time stamps on files or something (I don’t expect there’s much mechanism for this latter idea, but I could be wrong).

I don’t know all the details about how this works under the hood, but things should work well here: precompilation is re-done only when the sources have changed (seemingly based on a timestamp, not a hash of the contents).

See for example this little test:

Two projects/packages are created; MyApp `dev`s MyLib:

```julia
shell$ julia -q -e 'using Pkg; pkg"generate MyLib"'
 Generating project MyLib:
    MyLib/Project.toml
    MyLib/src/MyLib.jl

shell$ julia -q -e 'using Pkg; pkg"generate MyApp"'
 Generating project MyApp:
    MyApp/Project.toml
    MyApp/src/MyApp.jl

shell$ julia -q --project="MyApp" -e 'using Pkg; pkg"dev MyLib"'
[ Info: Resolving package identifier `MyLib` as a directory at `/tmp/tst/MyLib`.
Path `MyLib` exists and looks like the correct package. Using existing path.
  Resolving package versions...
   Updating `/tmp/tst/MyApp/Project.toml`
  [f5901e7e] + MyLib v0.1.0 [`../MyLib`]
   Updating `/tmp/tst/MyApp/Manifest.toml`
  [f5901e7e] + MyLib v0.1.0 [`../MyLib`]

```

No precompiled file exists yet:

```julia
shell$ ls -lh ~/.julia/compiled/v1.4/MyLib/
ls: cannot access '/home/francois/.julia/compiled/v1.4/MyLib/': No such file or directory

```

The first time `MyApp` uses `MyLib`, it is precompiled:

```julia
shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
[Info: Precompiling MyLib [f5901e7e-c96d-4cf6-9760-37e60dbc0491]
Hello World!

shell$ ls -l ~/.julia/compiled/v1.4/MyLib/                                                                                                               
total 20
-rw-r--r-- 1 francois francois 2604 Apr 12 22:20 qx7Zn_pzCEQ.ji

```

Subsequent uses of `MyLib` incur no precompilation, since the sources have not changed:

```julia
shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
Hello World!

```

When `MyLib` sources change, however, precompilation has to be performed again:

```julia
shell$ touch MyLib/src/MyLib.jl

shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
[Info: Precompiling MyLib [f5901e7e-c96d-4cf6-9760-37e60dbc0491]

```

  

That being said, if compilation times are long for your application, I would advise against relying _only_ on precompilation: you can get higher benefits with a custom system image built using [`PackageCompiler`](https://github.com/JuliaLang/PackageCompiler.jl). This could be especially useful when you put things into production.

---

<div class="post-metadata">

### Author: ![Ziyuan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ziyuan/32/29269_2.png) [@Ziyuan](https://discourse.julialang.org/u/Ziyuan)
#### Post date: [October 10, 2021, 10:40pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/18 "2021-10-10T22:40:18Z")

</div>

Does it work when we are in a project? In my case it complains that the package cannot be found.

---

<div class="post-metadata">

### Author: ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [October 11, 2021, 2:42pm UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/19 "2021-10-11T14:42:30Z")

</div>

Yes. Could you start another issue? Feel free to ping me on it.

---

<div class="post-metadata">

### Author: ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)
#### Post date: [October 14, 2021, 3:58am UTC](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/20 "2021-10-14T03:58:51Z")

</div>

I’ve had similar struggles at leveraging modules/packages in a way that makes software development simple & straightforward.

Here are some key insights that might help (using Julia v1.6.3).

### Modules: Namespaces for code organization

Julia `module`s mostly correspond to namespaces in C++ (and other languages).

- They they provide mechanisms to avoid name collisions between unrelated code.
- They allow developers to organize their code into separate logical, hierarchical units.
- They are therefore an ideal logical boundary for delimiting what code should be allowed to access the innards of a given software component & mutate its state.
- Julia `module`s therefore provide base capabilities needed to develop _conceptual_ “software modules”.
- But `module`s don’t deal with deployment issues!!!
- (Sub)-module code is directly directly “compiled” in (using `include()` when stored in separate files).
- `import` and `using` is only used to make (sub)-module code available in your namespace (it should already be “compiled”).

### Packages: A way to deploy w/dependencies & manage compile time

Julia packages provide a layer around `module`s. They help with code sharing/deployment in multiple projects:

- Due to practical requirements, packages are themselves wrapped within Julia modules.
- Packages provide Project.toml files to specify requirements (what needs to be installed/accessible).
- Packages are the minimum unit of “pre-compilation” Julia supports at this time (as far as I am aware).
- If a package can make use of fewer dependencies, it should compile faster.
- If multiple packages include the same base dependencies, “shared” code might might still need to be recompiled (as far as I know).
- External packages are loaded using either `import` or `using` - and subsequently “compiled”.

### Adding non-Git-controlled packages

Though your first instinct might be to add a local package using `pkg> add`:

```julia
julia> ]
(MyProjEnv) pkg> add /path/to/MyPackageLib/MyAwesomePkg

```

But this won’t work unless `MyAwesomePkg` is a Git repository (the Git root itself; it is insufficient to be a subdirectory of a Git repo).

- It would appear that `pkg> add` assumes the provided path is a refrence repository that is not to be altered.
- Julia’s package manager therefore _ **clones** _ this repo & generates a working snapshot of it under `~/.julia/packages`.

However, unlike `pkg> add`, `pkg> dev` simply adds packages with a direct link to the supplied path (not a clone):

```julia
julia> ]
(MyProjEnv) pkg> dev /path/to/MyPackageLib/MyAwesomePkg

```

So, this call to `pkg> dev` _ **will** _ work even if `MyAwesomePkg` is _ **not** _ a Git repository.

### Adding “single-file” packages

Unfortunately, Julia (1.6.3) won’t let you `pkg> dev` a “single-file” package:

```julia
pkg> dev /path/to/MyPackageLib/ASingleFilePkg
ERROR: Unable to parse `/path/to/MyPackageLib/ASingleFilePkg` as a package.

```

However, if you have something like:

```julia-auto
/path/to/MyPackageLib
├── MyFolderBasedPackage
| └── src
| └── MyFolderBasedPackage.jl
└── ASingleFilePkg.jl

```

Then, on startup, your project can register this library with `Base.LOAD_PATH` to access package `ASingleFilePkg`:

```julia
# Somewhere in your project's initialization code:
push!(Base.LOAD_PATH, "/path/to/MyPackageLib")

# Further on in your project:
using ASingleFilePkg #Should work

```

### Comments on `LOAD_PATH`

`LOAD_PATH` is what defines your environment stack, and therefore indirectly (1 level) adds packages to your project:

- If a directory in `LOAD_PATH` _ **does not** _ have a `Project.toml` file, Julia recognize {single .jl files} and {subdirectories having proper “package structures”} as packages.
- If a directory in `LOAD_PATH` _ **does have** _ a `Project.toml` file, Julia _ **does not** _ recognize single .jl files or subdirectories as packages. Only the packages specified in `Project.toml` are made available.

More info in Julia docs:

- [Constants · The Julia Language](https://docs.julialang.org/en/v1/base/constants/#Base.LOAD_PATH)

### Might a monolithic, multi-module package be a better solution?

```julia-auto
Aerodynamics/ # The Julia part
  .git/ #Single .git repository
  Project.toml # Dependencies for the whole of Aerodynamics.jl package
  src/
    Aerodynamics.jl
    AerodynamicsBase/ # base definitions (likely its own module)
      definitions.jl
      ...
    CommandTypes/ # Another module
      barrel_roll.jl #Seems kind of wreckless, but still...
      ...
    HugeSim/ # Directly depends on Base, but not on CommandTypes
      flightpaths.jl
      stabilization.jl
      ...
    SmallSim/
      ...

```

Code/files could be included as follows:

```julia
#src/Aerodynamics.jl
module Aerodynamics

module AerodynamicsBase
include("Base/definitions.jl")
#...

#Export symbols defined in one of the above files:
export ReallyImportantBaseType, ReallyImportantBaseFunction
end

module CommandTypes
include("CommandTypes/barrel_roll.jl")
#...
end

end #module Aerodynamics

```

Functions in the `CommandTypes` module could access types & functions using:

```julia
import ..AerodynamicsBase: SomeBaseFunction, SomeBaseType

```

And typical package usage would look something like the following:

```julia
import Aerodynamics #import: Don't pull in export-ed symbols
using Aerodynamics.AerodynamicsBase #using: DO pull in export-ed symbols

SomeBaseFunction() #Now readily available without fully-qualified-path (FQP)

```

### Monolithic, multi-module package: Principal downside

All the code in this `Aerodynamics` package needs to be compiled to use any of its (sub)-modules.

- Might be acceptable/{what you want} in an `Aerodynamics` package.
- Not like the situation where users of `Plots.jl` don’t want to pull in/“compile” ALL plotting backends when they only want/need a single one.

[Next page](https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931.md?page=2)
