Why does Pluto not seem to find self-written modules I keep in a directory ~/lib/julia when that directory is included via a file .julia/config/startup.jl and when they are found by Jupyter notebooks and by the REPL?
When I force it by pushing it into LOAD_PATH in a begin...end block, some “official” modules that were installed with the package manager and used from within the self-written module are not found anymore.
Module inclusion somehow seems rather broken in Pluto.
might be a bit complicated here, but can you maybe try to provide an example? I for example do not fully understand (maybe grammar, maybe not my mother tongue) the sentence “When I force it by” what you are pushing where and what you do afterwards.
Note that by default every Pluto notebook aims to be reproducible and opens its own environment. I think that also resets the load path to be empty (so what you do in the startup is ignored basically).
You could start your notebook with a line
using Pkg; Pkg.activate(".");
to use your own (main/global) environment.
This is on purpose since Pluto notebooks are meant to be reproducible, even when you sent them around by mail. So that is not broken, that is both expected and on purposely chosen behaviour.
Pluto does not run your .julia/config/startup.jl at all by default. if you want it, you can add a cell that runs it manually. As @kellertuer said, this will make your notebook unreproducible, which is not suggested!
Please share more about your use case if you still have questions
Thanks for the explanation, that wasn’t clear to me and at least explains this behavior. However, the solution with Pkg.activate causes all the regular packages not to be found anymore, because the default is overridden, whereas I simply wanted to have my local path in addition to all the rest.
By “forcing it” I meant push!(LOAD_PATH, "/Users/myhome/lib/julia"). Now that local module is also found, but regularly installed modules called from within that local module are not:
module PolyBound
using Optim
PolyBound is my local module, Optim is a module I have installed the usual way with the package manager.
I do agree that the setup is a bit involved, but without being able to reproduce the scenario here locally, I am unable to provide further tipps or hints what to look for.
It seems to work in your scripts? Does that “forcing it” work in other circumstances so that you would expect it to work within Pluto as well?
I would suppose it PolyBound is a module (and not a full package) then Optim has to be installed in the environment you are in (without the activation, the Pluto one, with the activation your global one).
I am also unsure what
Now that local module is also found, but regularly installed modules called from within that local module are not:
means. What is the error you get? What do you expect to work? What are you actually doing?
So overall, sorry to not be able to help, but this seems a bit of a unused scenario to me at least, so I feel it is hard to follow your workflow and expected things.
As far as I understand the situation:
- you have some code living in files in
~/lib/julia. These files just containmodules - you usually
includethese files instartup.jl - this does not work in Pluto.jl
So to me the simple fix is to also include them in a cell in Pluto. Just make sure you tell Pluto to load all the libraries beforehand by using them. Pluto manages its own environments and starts from a clean slate but automatically adds/install all packages you are using. Alternatively you can activate your global environment in Pluto with import Pkg; Pkg.activate("@v1.xx) where xx is your Julia version.
The more sustainable fix would involve some proper managing of your environments. To me it sounds like you currently just put everything you install into the same default environment - the same that you include your libraries into on startup. That’s why it currently works: the dependencies of each of your modules just happen to be available in your default, global environment. While that works for now, it is a very brittle setup and will almost surely break in the future and continue to be a constant source of annoyance.
A more sustainable solution is to convert your local modules into proper local packages and start using environments for your work. If you are not yet familiar with the concepts, I recommend to give this a quick read:
The main benefit is that a package knows its dependencies, so you can never run into this issue
To circle this back to Pluto: once you have some nice enviroment, where you added all your own packages, you can activate that in Pluto via Pkg.
Alternatively, you can also use Pkg in Pluto to selectively load the packages you really need for the notebook.
Thank you for taking the time for this detailed advice. Yes, what you outlined summarizes essentially the situation, and includeing the startup.jl is more or less what I ended up doing; the push line I mentioned is from my startup.jl. It seems that using Optim in the main code allows Pluto to make use of it in the module as well. At least it looks like it works now.
So I guess I’ll eventually reorganize my environments the way you suggested for use with Pluto.
1 Like