Could we make first-class support for packages that are "just files" and not repositories?

General Usage
,
20

I’ve had similar struggles at leveraging modules/packages in a way that makes software development simple & straightforward.

Here are some key insights that might help (using Julia v1.6.3).

Modules: Namespaces for code organization

Julia modules mostly correspond to namespaces in C++ (and other languages).

  • They they provide mechanisms to avoid name collisions between unrelated code.
  • They allow developers to organize their code into separate logical, hierarchical units.
  • They are therefore an ideal logical boundary for delimiting what code should be allowed to access the innards of a given software component & mutate its state.
  • Julia modules therefore provide base capabilities needed to develop conceptual “software modules”.
  • But modules don’t deal with deployment issues!!!
  • (Sub)-module code is directly directly “compiled” in (using include() when stored in separate files).
  • import and using is only used to make (sub)-module code available in your namespace (it should already be “compiled”).

Packages: A way to deploy w/dependencies & manage compile time

Julia packages provide a layer around modules. They help with code sharing/deployment in multiple projects:

  • Due to practical requirements, packages are themselves wrapped within Julia modules.
  • Packages provide Project.toml files to specify requirements (what needs to be installed/accessible).
  • Packages are the minimum unit of “pre-compilation” Julia supports at this time (as far as I am aware).
  • If a package can make use of fewer dependencies, it should compile faster.
  • If multiple packages include the same base dependencies, “shared” code might might still need to be recompiled (as far as I know).
  • External packages are loaded using either import or using - and subsequently “compiled”.

Adding non-Git-controlled packages

Though your first instinct might be to add a local package using pkg> add:

julia> ]
(MyProjEnv) pkg> add /path/to/MyPackageLib/MyAwesomePkg

But this won’t work unless MyAwesomePkg is a Git repository (the Git root itself; it is insufficient to be a subdirectory of a Git repo).

  • It would appear that pkg> add assumes the provided path is a refrence repository that is not to be altered.
  • Julia’s package manager therefore clones this repo & generates a working snapshot of it under ~/.julia/packages.

However, unlike pkg> add, pkg> dev simply adds packages with a direct link to the supplied path (not a clone):

julia> ]
(MyProjEnv) pkg> dev /path/to/MyPackageLib/MyAwesomePkg

So, this call to pkg> dev will work even if MyAwesomePkg is not a Git repository.

Adding “single-file” packages

Unfortunately, Julia (1.6.3) won’t let you pkg> dev a “single-file” package:

pkg> dev /path/to/MyPackageLib/ASingleFilePkg
ERROR: Unable to parse `/path/to/MyPackageLib/ASingleFilePkg` as a package.

However, if you have something like:

/path/to/MyPackageLib
├── MyFolderBasedPackage
|   └── src
|       └── MyFolderBasedPackage.jl
└── ASingleFilePkg.jl

Then, on startup, your project can register this library with Base.LOAD_PATH to access package ASingleFilePkg:

# Somewhere in your project's initialization code:
push!(Base.LOAD_PATH, "/path/to/MyPackageLib")

# Further on in your project:
using ASingleFilePkg #Should work

Comments on LOAD_PATH

LOAD_PATH is what defines your environment stack, and therefore indirectly (1 level) adds packages to your project:

  • If a directory in LOAD_PATH does not have a Project.toml file, Julia recognize {single .jl files} and {subdirectories having proper “package structures”} as packages.
  • If a directory in LOAD_PATH does have a Project.toml file, Julia does not recognize single .jl files or subdirectories as packages. Only the packages specified in Project.toml are made available.

More info in Julia docs:

Might a monolithic, multi-module package be a better solution?

Aerodynamics/ # The Julia part
  .git/ #Single .git repository
  Project.toml # Dependencies for the whole of Aerodynamics.jl package
  src/
    Aerodynamics.jl
    AerodynamicsBase/ # base definitions (likely its own module)
      definitions.jl
      ...
    CommandTypes/ # Another module
      barrel_roll.jl #Seems kind of wreckless, but still...
      ...
    HugeSim/ # Directly depends on Base, but not on CommandTypes
      flightpaths.jl
      stabilization.jl
      ...
    SmallSim/
      ...

Code/files could be included as follows:

#src/Aerodynamics.jl
module Aerodynamics

module AerodynamicsBase
include("Base/definitions.jl")
#...

#Export symbols defined in one of the above files:
export ReallyImportantBaseType, ReallyImportantBaseFunction
end

module CommandTypes
include("CommandTypes/barrel_roll.jl")
#...
end

end #module Aerodynamics

Functions in the CommandTypes module could access types & functions using:

import ..AerodynamicsBase: SomeBaseFunction, SomeBaseType

And typical package usage would look something like the following:

import Aerodynamics                   #import: Don't pull in export-ed symbols
using Aerodynamics.AerodynamicsBase   #using:  DO pull in export-ed symbols

SomeBaseFunction() #Now readily available without fully-qualified-path (FQP)

Monolithic, multi-module package: Principal downside

All the code in this Aerodynamics package needs to be compiled to use any of its (sub)-modules.

  • Might be acceptable/{what you want} in an Aerodynamics package.
  • Not like the situation where users of Plots.jl don’t want to pull in/“compile” ALL plotting backends when they only want/need a single one.
1 Like