Using a dev --local package in module

New to Julia
1

Let’s say I have a module MyModule in which I want to use a modified version of a registered package, for instance DifferentialEquations. In order not to “pollute” any other environment I can dev --local DifferentialEquations from within my module, work on it and add my favorite new functions.

Now, if I simply do using MyModule when I start up julia, it will still load the registered version of the package (and not my local dev version). I can get around this by first doing ]activate MyModule and then using MyModule, which then loads my local version of DifferentialEquations. However, this is a bit annoying to do with multiple packages and seems like I’m doing something wrong.

(I think this might have to do with the fact that if I do ]activate MyModule; status then I see the dev version of DifferentialEquations, while if I simply do ]status MyModule from the main environment then I see the registered version of DifferentialEquations)

What am I doing wrong?

2

I think that you are actually doing it right.

] activate is the way to proceed in order to switch environments, such that in one environment you have some packages and in another you have other packages (or other versions of the same ones).

Modules may have their own environments, but you can also have environments for broader projects. If you are using your local dev version of DifferentialEquations for multiple modules, you can make an umbrella project that you may use to work with all of them. E.g.:

cd("MyProject")
] activate .
] dev DifferentialEquations
] dev MyModule
] dev AnotherModule

And then, any time you want to work in that project:

cd("MyProject")
] activate .
using MyModule
using AnotherModule
1 Like
3

I guess what I’m asking is: why is it the case that using MyModule does not load packages from its own environment rather than from the active environment?

Is there a way to avoid having to activate a module’s environment before using it?

There should be one, right? Because if I have two different modules that use dev versions of different packages (or different dev versions of the same package) and I need to load these modules into a third module I’d have no way to activate them first.

4

Ok, I understand: you mean that you’d expect MyModule.DifferentialEquations be the dev version regardless of the environment where you are using MyModule.

Well, I agree with that. I would expect that too.

5

Environments stack “upwards”, but not “downwards”: the manifest of the used projects is ignored. See

https://docs.julialang.org/en/v1/manual/code-loading/

For this particular use case, a possible workflow could be

  1. making a non-local development branch of DifferentialEquations,
  2. deving or adding that in the MyProject environment.
2 Likes
6

I carefully read the docs you linked. As you say it must have to do with the behavior of environment stacks, which is still not totally clear to me.

I find this behavior very counterintuitive: in MyModule’s Project.toml file DifferentialEquations is correctly identified by the UUID of its dev --local version and the Manifest.toml file points to the correct directory, i.e. /MyModule/dev/DifferentialEquations.
I would therefore expect that using DifferentialEquations in MyModule would load exactly that version. Instead it loads the version of whatever environment I’m in.

Also, why does a non-local development branch fixes the issue?

I still think there has to be something wrong going on. I’ll make an example to try to convey why:
Let’s say I have two different modules MyModule1 and MyModule2. In both modules I need to implement a change to DifferentialEquations, but they need to be different changes which for whatever reason are incompatible with each other. What I could then do is dev --local DifferentialEquations in both modules’ environments which would allow me to add the private features that I need in the correct module. The only thing I need to take care of is to first load the correct environment before using the necessary module, so far so good.

However, let’s now say I need to use both MyModule1 and MyModule2 in a ParentModule (which can also be the main module). Well now I have a problem, because loading both is going to be impossible, isn’t it?

I know this sounds like a very edge case (and is not actually my use case). I’m just thinking out loud as to why this doesn’t make sense to me.

7

I think what you are missing is that a Julia image cannot have different versions of the same module loaded.

So if the active project A uses B which uses C, the resolver will figure out the versions given all the constraints in A and B, which will mean a specific version of C, and the manifest of B will be ignored.

So that A (in the example above) can find it.

Yes.

Your best option is creating a merge of these in yet another branch.

8

Yes, you’re obviously right.

I find this to be a bit of a limit of the package manager, but I’m definitely missing some reason why it was chosen to be this way: I absolutely love everything else of it.

Well I guess I’ll have to live with loading its environment before using the package. Unfortunately the non-local branch solution does not really work as I use the non-dev version in other packages (I’ll mark it as solved anyway, thanks for the replies :slight_smile:).