Could we make first-class support for packages that are "just files" and not repositories?

General Usage
,
1

I keep circling around the same issue from a variety of different vantage points, and I think that having first-class support for local, unregistered packages (just local files, not repositories) would solve a lot of problems that I see in structuring Julia programs.

We have a repo that has our aircraft simulation in it. The current package manager does a great job of managing our dependence on external packages, like HDF5 and YAML. I love it.

We also have local modules for reusable functionality. One module might be, e.g., Aerodynamics. Another might be CommandTypes.

Sometimes, we’re making a huge simulation with tons of complex stuff in it, and we need Aerodynamics in there somewhere. Sometimes, we’re making a little tool to explore our aerodynamics, so it needs Aerodynamics too. Maybe Aerodynamics needs to know about common types as well, like AileronCommand, and lots of other things need to know about the AileronCommand.

If the AileronCommand is part of a local module, then somehow Aerodynamics needs to know where that module is loaded, or it needs to load it from its parent, and this creates weird scope issues. See the developing thread in Import types "above" a module with ..: for consistency and brevity?.

If the AileronCommand is part of a local, unregistered package, then everything that needs to know about it can just do using CommandTypes, where CommandTypes is available via ]dev path/to/CommandTypes. However, if Aerodynamics is also a local, unregistered package, then not only does it need to do ]dev CommonTypes (which is fine), but anyone that wants to use Aerodynamics also has to do ]dev CommonTypes. That is, they have to resolve all local dependencies manually. See "Unsatisfiable requirements" when local packages use local packages (fixed in v1.4). It’s not the end of the world (I just spent all afternoon dealing with Python stuff that was at least this cumbersome), but it is strangely messy for Julia.

If we could just ]add path/to/LocalPackage, have that work like ]dev, and just have everything “agree” to use whatever’s in that local path rather than worrying about versions of things that have no versions because they don’t have their own git repos, then I think this would all just work. (I recognize that it’s likely that I’m missing something here, but it sure seems like this would all work, and there seem to be others have similar problems.)

Just to be super-duper clear about this: having tons of packages in different repos is a good workflow for some stuff, and this seems to be the Julian way currently, but it doesn’t make sense for a lot of other stuff. Some projects are going to have everything in one repo, especially when Julia isn’t the only language (Julia is a small minority in our code base for now), so in those cases, local, unregistered packages seems like a good way forward.

What do folks think?

9 Likes
2

I am not sure I understand the distinction between local files and git repos here. Both just contain code, and can (should) have a version in the Project.toml.

You may be interested in the discussion at

and many similar discussions on this forum.

TL;DR: the solution is a local/custom registry. Nice tooling is evolving for it, eg

1 Like
3

Thanks @Tamas_Papp, but I’m confused. For LocalRegistry, it seems like packages still need to be individual git repos, no? I might need to clarify my post a bit:

The individual repos thing doesn’t work for me, or for other people that I talk to in engineering. I need to have all of my code related to the aircraft simulation in one big repo; it’s in there with a bunch of other code (Julia is only a small part of this project). The way packages are brought in in general (e.g., any module anywhere can using CommonTypes, and they all get the same types) is very attractive, but I’m going to need the CommonTypes package to just be some code located in some subdirectory of my repo, not its own git repo.

]dev path/to/CommonTypes works well, but only for one level. Once you have multiple ]dev’d packages, you end up having to resolve dependence manually, as mentioned in "Unsatisfiable requirements" when local packages use local packages (fixed in v1.4).

Maybe what’s not clear in my original post is that by “local, unregistered package” I mean just a subdirectory that’s structured like a package in my overall repo:

/.git
/analysis # All the other code in the repo...
/drivers
/flight
/sim # The Julia part
  /Aerodynamics # A local package
    /src
    Project.toml # Depend on CommonTypes
    Manifest.toml
  /CommandTypes # Another local package
    /src
    Project.toml
    Manifest.toml
  /HugeSim
    /src
    Project.toml # Directly depends on Aerodynamics, but not on CommonTypes
    Manifest.toml
  /SmallSim
  /StudyOfAerodynamics # etc.
  ...

As far as I can tell, using these local paths as packages doesn’t really work, because ]dev LocalPackage doesn’t resolve dependencies, so, for this example, HugeSim would also need to ]dev ../CommonTypes prior to including the things that depend on CommonTypes, like ]dev ../Aerodynamics. So, each project moving up has to explicitly write out all of the local dependencies for anything it uses, and for anything that those use, etc.

If local dependency resolution worked for ]dev, then this whole thing would work perfectly as far as I can tell. That’s probably the most important thing to highlight.

For what it’s worth, I also think that using local packages would be an easy way for folks (at least people like me) to learn about creating a program structure that uses multiple packages. It just helps minimize the boilerplate.

Ok, I hope that clears up my intent rather than just drowning everything in detail. Please let me know if this makes sense!

2 Likes
4

Strictly speaking, a package does not need to be a git repository : in its most basic form (as created by Pkg.generate), it is a repository following a specific, predefined structure. In particular it contains a Project.toml file storing a bunch of metadata, including a version number. I think this is what Tucker refers to as “local files”.

However, if I understand correctly, as soon as a package wants to be registered, it needs to be a git repository. This is because, as I understand it, the registry cares about versions, and links each registered version with a particular SHA1 identifying the state of the git repository.

In short, unless I’m mistaken, a package that is not a git repository can not be registered. It can define a version in its Project.toml file, but I’m not sure that Pkg even looks at it. I don’t know of any other way to use such a package than

  1. Pkg.activateing it, or
  2. Pkg.developing it (which essentially tells Pkg to ignore any notion of version and take the source files as they are)
1 Like
5

How about just do ] dev ../CommandTypes at sim/Aerodynamics and check-in sim/Aerodynamics/Manifest.toml? You can then just do julia --project in sim/Aerodynamics/ directory to start working on it. Then using Aerodynamics should just work (after ] instantiate).

Alternatively, if all Julia packages are compatible, having sim/Manifest.toml that devs evrything under it may make sense.

6

Hi @tkf, I hear you, and that’s exactly what I tried, but it doesn’t really work. See "Unsatisfiable requirements" when local packages use local packages (fixed in v1.4).

(It may work in v1.4; I haven’t verified yet.)

I attribute the fact that it doesn’t work to the fact that “local packages” like this are not first class citizens, and ]dev is meant as a short-term thing, not something you build your program on.

1 Like
7

Hmm… Relative dev trick is working since 1.0 for me. I’m using this for reproducible test environment like this: https://github.com/tkf/Transducers.jl/blob/2217a84a7a382d9611a7750ae6628681ebb5c667/test/environments/main/Manifest.toml#L366

Quickly skimming the discussion you linked, I wonder if you missed my point on explicit activation of the project:

(or you can also use ] activate sim/Aerodynamics/)

This is to say, I’m not suggesting to do ] dev sim/Aerodynamics/; just activate it or a project containing it.

8

Yes. What did not work until recently is deving a package that itself devs another package.

In the example shown above, everything is fine with Aerodynamics, which depends on CommonTypes. As far as I understand, problems start occurring when working with HugeSim and trying to make it depend on AeroDynamics.

9

Why not just do ] dev ../CommandTypes at sim/HugeSim? If you check in sim/HugeSim/Manifest.toml, it just has to be done once.

It should also be possible to develop a tool to automatically dev relative dependencies. I did something similar in GitHub - tkf/Rogue.jl but for add.

1 Like
10

Yes.

Yes, but this amounts to having HugeSim manually resolve all dependencies of all packages that it uses, and that they use, and that they use, etc. Even for a small number of packages, this becomes tedious and fragile very quickly.

Yes, I think that’s what I’m proposing here, and looking at your Rogue package quickly, it does seem similar. Thanks.

1 Like
11

Ah, sorry, I missed this point. I agree that developing a tool to cover monorepo approach like this would be great.

2 Likes
12

In your opinion, would Rogue be an adequate package to host such features?

13

I think it is. But it’s still kind of a works-for-me package and not in a releasable quality. The main is reason that it relies on my Python package vcslinks and so the installation process is not automatic. So I can’t recommend people to use it or contribute to it.

(If people are interested in using Rogue.jl I should probably remove vcslinks or make it optional as it is only used for creating nice commit message. Another option is to port vcslinks to Julia but it’s kind of a dull task.)

14

One question I have on this front is how precompilation would be handled. When they’re “just files” in ]dev mode, how does Julia know when to precompile them? Does it always just use fresh, or does it look at time stamps on files or something (I don’t expect there’s much mechanism for this latter idea, but I could be wrong). In my application right now, for instance, this would matter a great deal; compilation times are very long. Does anyone know what level of effort would be involved here?

Do you all think I should make a GitHub issue about this?

15

I just wanted to make sure you are familiar with this usage:

push!(LOAD_PATH,pwd())
import localfile
[ Info: Precompiling localfile [top-level]

localfile.jl is just a minimal module:

module localfile
end

If you just wanted to deal with files and bypass the whole Pkg system, you could just append the directories to LOAD_PATH and then import away, precompilation included!

Here is the documentation:

help?> LOAD_PATH
search: LOAD_PATH

  LOAD_PATH

  An array of paths for using and import statements to consider as
  project environments or package directories when loading code. It is
  populated based on the JULIA_LOAD_PATH environment variable if set;
  otherwise it defaults to ["@", "@v#.#", "@stdlib"]. Entries starting
  with @ have special meanings:

    •    @ refers to the "current active environment", the initial
        value of which is initially determined by the JULIA_PROJECT
        environment variable or the --project command-line option.

    •    @stdlib expands to the absolute path of the current Julia
        installation's standard library directory.

    •    @name refers to a named environment, which are stored in
        depots (see JULIA_DEPOT_PATH) under the environments
        subdirectory. The user's named environments are stored in
        ~/.julia/environments so @name would refer to the
        environment in ~/.julia/environments/name if it exists and
        contains a Project.toml file. If name contains # characters,
        then they are replaced with the major, minor and patch
        components of the Julia version number. For example, if you
        are running Julia 1.2 then @v#.# expands to @v1.2 and will
        look for an environment by that name, typically at
        ~/.julia/environments/v1.2.

  The fully expanded value of LOAD_PATH that is searched for projects
  and packages can be seen by calling the Base.load_path() function.

  See also: JULIA_LOAD_PATH, JULIA_PROJECT, JULIA_DEPOT_PATH, and Code
  Loading.

Full documentation here:

https://docs.julialang.org/en/v1/manual/code-loading/

16

Regarding your question about “fresh” and when to “precompile”, you may be interested in the Revise.jl package:

1 Like
17

I don’t know all the details about how this works under the hood, but things should work well here: precompilation is re-done only when the sources have changed (seemingly based on a timestamp, not a hash of the contents).

See for example this little test:

Two projects/packages are created; MyApp devs MyLib:

shell$ julia -q -e 'using Pkg; pkg"generate MyLib"'
 Generating  project MyLib:
    MyLib/Project.toml
    MyLib/src/MyLib.jl

shell$ julia -q -e 'using Pkg; pkg"generate MyApp"'
 Generating  project MyApp:
    MyApp/Project.toml
    MyApp/src/MyApp.jl

shell$ julia -q --project="MyApp" -e 'using Pkg; pkg"dev MyLib"'
[ Info: Resolving package identifier `MyLib` as a directory at `/tmp/tst/MyLib`.
Path `MyLib` exists and looks like the correct package. Using existing path.
  Resolving package versions...
   Updating `/tmp/tst/MyApp/Project.toml`
  [f5901e7e] + MyLib v0.1.0 [`../MyLib`]
   Updating `/tmp/tst/MyApp/Manifest.toml`
  [f5901e7e] + MyLib v0.1.0 [`../MyLib`]

No precompiled file exists yet:

shell$ ls -lh ~/.julia/compiled/v1.4/MyLib/
ls: cannot access '/home/francois/.julia/compiled/v1.4/MyLib/': No such file or directory

The first time MyApp uses MyLib, it is precompiled:

shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
[ Info: Precompiling MyLib [f5901e7e-c96d-4cf6-9760-37e60dbc0491]
Hello World!

shell$ ls -l ~/.julia/compiled/v1.4/MyLib/                                                                                                               
total 20
-rw-r--r--   1 francois francois  2604 Apr 12 22:20 qx7Zn_pzCEQ.ji

Subsequent uses of MyLib incur no precompilation, since the sources have not changed:

shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
Hello World!

When MyLib sources change, however, precompilation has to be performed again:

shell$ touch MyLib/src/MyLib.jl

shell$ julia -q --project="MyApp"
julia> using MyLib; MyLib.greet(); exit()
[ Info: Precompiling MyLib [f5901e7e-c96d-4cf6-9760-37e60dbc0491]

That being said, if compilation times are long for your application, I would advise against relying only on precompilation: you can get higher benefits with a custom system image built using PackageCompiler. This could be especially useful when you put things into production.

4 Likes
18

Does it work when we are in a project? In my case it complains that the package cannot be found.

19

Yes. Could you start another issue? Feel free to ping me on it.

20

I’ve had similar struggles at leveraging modules/packages in a way that makes software development simple & straightforward.

Here are some key insights that might help (using Julia v1.6.3).

Modules: Namespaces for code organization

Julia modules mostly correspond to namespaces in C++ (and other languages).

  • They they provide mechanisms to avoid name collisions between unrelated code.
  • They allow developers to organize their code into separate logical, hierarchical units.
  • They are therefore an ideal logical boundary for delimiting what code should be allowed to access the innards of a given software component & mutate its state.
  • Julia modules therefore provide base capabilities needed to develop conceptual “software modules”.
  • But modules don’t deal with deployment issues!!!
  • (Sub)-module code is directly directly “compiled” in (using include() when stored in separate files).
  • import and using is only used to make (sub)-module code available in your namespace (it should already be “compiled”).

Packages: A way to deploy w/dependencies & manage compile time

Julia packages provide a layer around modules. They help with code sharing/deployment in multiple projects:

  • Due to practical requirements, packages are themselves wrapped within Julia modules.
  • Packages provide Project.toml files to specify requirements (what needs to be installed/accessible).
  • Packages are the minimum unit of “pre-compilation” Julia supports at this time (as far as I am aware).
  • If a package can make use of fewer dependencies, it should compile faster.
  • If multiple packages include the same base dependencies, “shared” code might might still need to be recompiled (as far as I know).
  • External packages are loaded using either import or using - and subsequently “compiled”.

Adding non-Git-controlled packages

Though your first instinct might be to add a local package using pkg> add:

julia> ]
(MyProjEnv) pkg> add /path/to/MyPackageLib/MyAwesomePkg

But this won’t work unless MyAwesomePkg is a Git repository (the Git root itself; it is insufficient to be a subdirectory of a Git repo).

  • It would appear that pkg> add assumes the provided path is a refrence repository that is not to be altered.
  • Julia’s package manager therefore clones this repo & generates a working snapshot of it under ~/.julia/packages.

However, unlike pkg> add, pkg> dev simply adds packages with a direct link to the supplied path (not a clone):

julia> ]
(MyProjEnv) pkg> dev /path/to/MyPackageLib/MyAwesomePkg

So, this call to pkg> dev will work even if MyAwesomePkg is not a Git repository.

Adding “single-file” packages

Unfortunately, Julia (1.6.3) won’t let you pkg> dev a “single-file” package:

pkg> dev /path/to/MyPackageLib/ASingleFilePkg
ERROR: Unable to parse `/path/to/MyPackageLib/ASingleFilePkg` as a package.

However, if you have something like:

/path/to/MyPackageLib
├── MyFolderBasedPackage
|   └── src
|       └── MyFolderBasedPackage.jl
└── ASingleFilePkg.jl

Then, on startup, your project can register this library with Base.LOAD_PATH to access package ASingleFilePkg:

# Somewhere in your project's initialization code:
push!(Base.LOAD_PATH, "/path/to/MyPackageLib")

# Further on in your project:
using ASingleFilePkg #Should work

Comments on LOAD_PATH

LOAD_PATH is what defines your environment stack, and therefore indirectly (1 level) adds packages to your project:

  • If a directory in LOAD_PATH does not have a Project.toml file, Julia recognize {single .jl files} and {subdirectories having proper “package structures”} as packages.
  • If a directory in LOAD_PATH does have a Project.toml file, Julia does not recognize single .jl files or subdirectories as packages. Only the packages specified in Project.toml are made available.

More info in Julia docs:

Might a monolithic, multi-module package be a better solution?

Aerodynamics/ # The Julia part
  .git/ #Single .git repository
  Project.toml # Dependencies for the whole of Aerodynamics.jl package
  src/
    Aerodynamics.jl
    AerodynamicsBase/ # base definitions (likely its own module)
      definitions.jl
      ...
    CommandTypes/ # Another module
      barrel_roll.jl #Seems kind of wreckless, but still...
      ...
    HugeSim/ # Directly depends on Base, but not on CommandTypes
      flightpaths.jl
      stabilization.jl
      ...
    SmallSim/
      ...

Code/files could be included as follows:

#src/Aerodynamics.jl
module Aerodynamics

module AerodynamicsBase
include("Base/definitions.jl")
#...

#Export symbols defined in one of the above files:
export ReallyImportantBaseType, ReallyImportantBaseFunction
end

module CommandTypes
include("CommandTypes/barrel_roll.jl")
#...
end

end #module Aerodynamics

Functions in the CommandTypes module could access types & functions using:

import ..AerodynamicsBase: SomeBaseFunction, SomeBaseType

And typical package usage would look something like the following:

import Aerodynamics                   #import: Don't pull in export-ed symbols
using Aerodynamics.AerodynamicsBase   #using:  DO pull in export-ed symbols

SomeBaseFunction() #Now readily available without fully-qualified-path (FQP)

Monolithic, multi-module package: Principal downside

All the code in this Aerodynamics package needs to be compiled to use any of its (sub)-modules.

  • Might be acceptable/{what you want} in an Aerodynamics package.
  • Not like the situation where users of Plots.jl don’t want to pull in/“compile” ALL plotting backends when they only want/need a single one.
1 Like