I’m trying to understand how to best work with local julia modules.
Our project is in C++ but we use Julia for testing, partly because it is convenient to generate and verify the complex input/output data we are dealing with. The Julia code is complex enough to organize it in multiple modules. The Julia code is placed inside the project’s git repo, since it is quite specific to this project.
Currently, each Julia module is its own package, in its own folder inside the project repo. Then there is a top-level project environment that have all packages as dependencies.
The packages could however not be added to the project environment using Pkg.add(path), because Pkg.add() expects each package to be its own git repository. Instead each package was added using Pkg.develop(path). This works with code loading and in particular with Revise.
However this seems to work only if the project environment’s Manifest.toml file is checked in to the project repo along with the Project.toml file. The reason being that the path information is stored in Manifest.toml and not in Project.toml. However, I was under the impression that Manifest.toml files should not be checked in, but rather re-generated locally.
My question is whether this setup is reasonable. How are others working in similar scenarios? Is it “right” to use packages for local modules like this, or should packages only be seen as a tool to distribute modules on their own?
Best regards,
Niclas Wiberg
Seems like a perfectly fine setup, the top-level project that you describe:
is more of a standalone “application” and for such things it is good to include the manifest.
1 Like
I do a similar thing for those projects ( like papers ) that won’t be public while I’m developing them. I have a directory where I put modules and proto-packages that I dev .
This works great unless I forget to get rid of manifest.toml (with paths that only make sense on my dropbox account) before I put the modules out there when I publish. I learned this the hard way when someone tried to reproduce my results and failed thanks to a path in the manifest.
If you use relative paths in your Manifest, this shouldn’t be a problem.
1 Like
I could edit the manifest to do that, but it’s simpler to put it in .gitignore so nobody but me will ever see it. I did this automatically with packages to CI would work, but forgot with a module. That’s how I got burned.
But if you don’t publish your manifest, how can someone reproduce your results (the manifest stores the version of all packages)?
Sorry, Misspoke. I publish the manifest with papers for reproducability and try and avoid my local stuff when I do that. That’s generally easy because by that time I’ve moved the files anyhow.
For notebooks and packages/modules under development, where the target audience is students and reviewers of the publications, I find it better to leave it out rather than edit the manifest. When I got busted it was a reviewer who noticed the problems in the manifest. The reviewer deleted the file and all went well.