How to install package dependencies for unregistered package

I have created a few packages for numerical simulations, which are not (yet) registered anywhere. My current development workflow is to just clone the repository manually and to use a script to start the simulations (which includes the main module file and then starts the main method).

The Project.toml/Manifest.toml files of the package include all required packages. However, I am wondering how new users can easily install all dependencies for the package in their main Julia environment (i.e., as if I use add PackageName for regular, registered packages)?

If I do ] activate . + instantiate in my project, it will download & install all dependencies but does not make them available if I do not start julia with --project=.. :frowning:

1 Like

That is as per design: when you activate the project then the installation of packages happens into that project and not into the global environment. Just work in that project. Or is there a reason not to?

1 Like

Thank you for your answer. I am looking at this from a usability perspective: Our modules are to be used regularly by novice users (students…) with little programming experience, and we want to make the experience for them as smooth as possible. Therefore, I was hoping that there is a way to avoid calling --project=path/to/module each time they run the module scripts with julia - really just to make it as hassle-free as possible for the students (and their advisor :wink: ).

However, I understand the design choice (and support it). I just wanted to see if there is a way to do it (or, alternatively, to confirm that there is no sane way to achieve what I was hoping to do, such that I can stop looking).

1 Like

Think of it as a feature — the intention is to provide a reproducible environment.

Debugging issues from a mismatched environment is usually guaranteed to be two orders of magnitude more hassle than this.

I have an alias for alias juliap='julia --project=@.' and start julia in the project folder with juliap. Maybe that is an option for your students? I think jupyter notebooks have something similar built-in, but I might be wrong.

Nope, I agree. But just to be sure: There is no sane way to achieve what I try to do, except by installing all packages by hand?

1 Like

Providing the manifest and then pkg> instantiate?

Sorry, I was not clear enough: How to install all packages automatically in the default environment?

The project already ships with a Project.toml and Manifest.toml, but AFAIU, instantiate will only install the packages for the currently activated project (or am I missing something?).

What does the @. do for you that the . does not? Other than that, good suggestion, thank you!

The idea is to activate that project, then instantiate from the manifest.

What you are proposing (doing this in the default environment) is very brittle: what if Julia is used for some other purpose, packages are updated accidentally, etc.

The help says

--project[={<dir>|@.}]    Set <dir> as the home project/environment

I think it has something to do with the shell interpreting . specially, but I don’t know.

The sane solution is to register your packages. If they are not material for the General registry, create your own registry. See https://github.com/GunnarFarneback/LocalRegistry.jl for tools.

2 Likes

Thank you for the advice and the hint to your package. However, the overall Package/Registry setup is very Julia-specific and way more involved than what I can expect my targeted user demographic (students with little programming or console experience) to handle.

If my users are able to install Julia and git, clone a Julia package, and run its scripts, without needing personal assistance from my side, I am already happy :wink: For this very specific use case I was looking for a simple solution to have them install all required dependencies with (preferably) a single command.

Nevertheless, we will eventually register the packages (either with the General registry or a domain-specific one), and make them work as Julia intends it.

I don’t know what you are teaching, but if it involves programming, these may be skills that can be reasonable expected (especially if help is available on demand).

Alternatively, you can set up a Jupyter server with everything pre-installed, so students who are unwilling to deal with all of the above can use it as a fallback.

I already tried this but failed to set up a central Julia installation with multiple pre-installed modules. Everything seems to work, except the Julia kernel won’t start when requested from JupyterHub. Anyways, thanks for your input!

I’m not quite following you here. Creating the registry and registering the packages would fall entirely on you and not involve students at all.

When you have the packages registered you get the dependency management for free from the package manager. The setup scenario would be

  • Install Julia.
  • Start Julia.
  • Run the following (or equivalent commands in Pkg REPL mode):
using Pkg
pkg"registry add General https://url.to/your/registry.git"
pkg"add YourUserFacingPackage"
1 Like

But this precludes me from simply cloning and editing the files during the development process, correct? Since I cannot (should not?) edit the downloaded package directly in ~/.julia, as far as I understand it. Even if I use dev MyPackage it will end up in ~/.julia/dev or in the fixed path JULIA_PKG_DEVDIR (according to https://julialang.github.io/Pkg.jl/stable/managing-packages/#Developing-packages-1).

Is there a way around this such as, e.g., using dev path/to/my/files to track a locally cloned repository? Or will that also just re-clone the package at path/to/my/files to JULIA_PKG_DEVDIR?

You should not edit packages in ~/.julia/packages. Developed packages in ~/.julia/dev are fine to edit; that’s kind of the idea of develop.

Yes, if you develop a local path, whatever is in that path will be used, in place.

1 Like

Thank you very much for these clarifications!

Just to be sure: in-place means that no clone will be made to ~/.julia/dev but that the files in the local path will be used directly?

Yes.