Use package not in current environment for testing/development

General Usage
1

There is one part of the development workflow that I may not be doing quite right.

When I am developing a package, the environment that package is generally the one activated.

Frequently, I have to use, for testing, some other packages which are not part of the package environment, meaning they are not dependencies of the package in question.

If the package is installed in the “main” environment, that is fine, because I can just use them. However, that leads to the progressive cluttering of the “main” environment.

Ideally, I would like to use packages without having to add them to the current environment (I don’t want them to be added to Project.toml).

One option is to have a third environment only for the development of the package, including test packages and deving the project in question. However, this is inconvenient because the corresponding Project.toml file cannot be placed in the same directory of the package itself.

Am I missing some other alternative? How do you use packages for testing and development which are not part of the project?

3 Likes
2

If the package is in the main env, should be available regardless? package - Julia - If I activate a new environment, why can I still load modules from my general environment? - Stack Overflow

1 Like
3

“Dev packages”: for those I use environment stacking, i.e. I have dev packages like Revies, BenchmarkTools etc. in the default / global environment. Hence they can always be loaded (unless I mangle with the LOAD_PATH).

“Test packages”: Well, I put them under extra / targets test. If you want you can use TestEnv.jl

I often have a scripts or examples folder in my package repo in which I ] dev .. or push! the main package into the LOAD_PATH.

Having said all of the above, I agree that we could still need a bit more tooling / convenience functions for testing / developing. For example, we currently don’t have a nice way to add packages to the test environment (i.e. the test target section).

1 Like
4

This has to be done by hand? Meaning, by editing the Project.toml file and moving it there after adding it?

5

Yes, they are, that is one option, but the main environment become cluttered very rapidly.

6

I think this is https://github.com/JuliaLang/Pkg.jl/issues/1233, which has no solution yet…

1 Like
7

Yes

1 Like
8

Environments: My understanding

Julia appears to have been developed more as an “analysis” language than a language to develop end-user “applications”.

In other words, the intent is for users to launch Julia in a given “environment” that is appropriate for solving certain types of problems. Note that this environment is typically “interactive” (ex: REPL, Jupyter, Pluto, …).

Suggested Julia usage flow

I used to launch Julia from any old directory where my data and “scripts” were found.

Lately, I started ensuring these directories are actually “proper Julia environments”. The following is an example of how I might organize said environments:

/path/to/JuliaRunEnvironments
├─ GrammarOfGraphics
|  └─ Project.toml
├─ CircuitDesign
|  └─ Project.toml
└─ SatelliteImagery
   └─ Project.toml

What do I consider to be a “proper Julia environment”?

  • Quite simply, an environment is a folder that includes a “Project.toml” file appropriate for the analysis I am performing.
  • In other words: an environment should include any/all the packages you might need for a particular analysis task.

Applying this methodology, you can mitigate conflicts where packages required in two different solution spaces might otherwise force certain dependencies to be downgraded. This is not possible if you are working in the default (@v1.6) environment.

Regarding suggestions wrt the environment stack:

  • Personally, I rarely push multiple Project.toml-based environments onto my environment stack.
  • On startup of a new Julia session, I might make available a local library of custom packages using push!(Base.LOAD_PATH, "/path/to/MyPackageRepo") — but that’s about it.
  • I prefer to keep my default/base (@v1.6) environment relatively clean/minimal to avoid issues.

Sample set of run environments

For illustration purposes, a developer might want the following set of what I would call “run” environments:

/path/to/JuliaRunEnvironments
├─ GrammarOfGraphics
|  ├─ Project.toml
|  ├─ ArtProjects
|  |  └─ starrynight_on_julia.jl
|  |  └─ monalisa_smiling.jl
|  |  └─ monalisa_frowning.jl
|  └─ TestPatterns
|     └─ nautilus_drawings.jl
|     └─ fractal_trees.jl
├─ CircuitDesign
|  ├─ Project.toml
|  ├─ DeltaSigma
|  |  └─ 1_bit_dac.jl
|  |  └─ 8_bit_adc.jl
|  └─ Filters
|     └─ butterworth.jl
|     └─ chebyshev.jl
└─ SatelliteImagery
   ├─ Project.toml
   ├─ MarsRover
   |  └─ detect_iron_concentration.jl
   └─ Hubble
      └─ compensate_for_gravity_lensing.jl

Seems to me like an ideal strategy to house domain-specific scripts.

So when do I ]dev a package under development, exactly?

Typically only when I want to add/remove dependencies from said package.

Otherwise, I tend to create environments like:

/path/to/JuliaDevEnvironments
├─ MyPackageDev
|  ├─ Project.toml
|  └─ ...
└─ ...

which would, of course, include an appropriate Project.toml file.

These environments are very light weight (basically one file). You just need to take a few extra steps to create them.

Then, when I wish to develop said package, I simply:

$ cd /path/to/JuliaDevEnvironments/MyPackageDev
$ julia
julia> ]
(@v1.6) pkg> activate .
(MyPackageDev) pkg>
julia> #Ready to start!
2 Likes
9

This is what I do too, but I type it as

$ cd /path/to/JuliaDevEnvironments/MyPackageDev
$ julia --project
julia> #Ready to start!
10

One option is

cd ~/.julia/dev/MyPackage
mkdir development
echo development/ >> .git/info/exclude
cd development
julia
]activate .
]dev MyPackage

Now you have a development environment for your package which doesn’t clutter the repo. Here you can put whatever scripts you want, or just repl around.

11

I think this does not really work, it conflicts with the environment of the package, for example when running tests: https://github.com/JuliaLang/Pkg.jl/issues/1585

(I had this same error trying to use an environment in a subfolder of my project).

12

@MA_Laforge I am also having brain muddle over this. I am developing a modular real-time control system application that has several packages (e.g. for telemetry, automated model-building, database interfaces, also a dynamic simulator, FMI/FMU blah blah etc. even a GUI). I am making heavy use of Julia’s multi-processing capabilities. I am trying to keep dependency bloat to a minimum and I never quite understand why I have to type instantiate so often. I don’t know how it’s all going to work(?) when I come to use the Package Compiler system, because, obvs., it can’t suddenly do a JIT comple in the middle of something scary. If you or any one else can recommend a good (non-beginner’s, seriously big project) tutorial I would be so happy. :grin:

13

Sounds like you want Tutorial on precompilation (but there will always be a risk of “a JIT comple in the middle of something scary” - I don’t think there’s any guaranteeing against it)

1 Like