Julia has a detailed, comprehensive, punctilious package management system. As a user of Julia, I am grateful for this.
I’m finishing up a paper. Part of it is a new solver for some nonlinear PDEs. Although I’m a seasoned programmer, I’m a novice when it comes to Julia so I apologize in advance.
I’ve spent a couple of days reading tutorials on how to make packages with Julia, and I’ve tried and failed to do this with my software. This is perhaps not entirely bad, reading this, it seems like simply having a package in the registry requires an engineer to monitor things because upper version numbers need to be kept up to date. It also doesn’t help that I seem to be the only one using jupyter, everyone else is using vscode. I don’t think ] works to do Pkg commands.
So I have given up on having my package in the registry, but I am wondering if there’s a way I can make my package “easy to use” even if it’s not in the registry? Is it easier to make a package that is to be installed via Pkg.add(url=“https://github.com/…”)?
Also, it is tremendously confusing to me to then have to Pkg add my own package, then Pkg dev it, and now it’s in ~/.julia, so it’s not in my Dropbox anymore. Also, Julia seems to want to know my Github username and passwords, can that be simplified down? What is the minimal way that I can put my package on Github, have it simple for me to develop, but also have other people be able to use it?
Thanks!
S
Edit. Solution was found by several people, I’ve credited nsajko. My main problem was that LOAD_PATH and JULIA_LOAD_PATH can interact in a confusing manner with package loading, and apparently their use is discouraged. I had . in my JULIA_LOAD_PATH. Once I deleted it from the path, I was able to do using MyPackage. Details in the thread below.
Thank you so much for the links. I’ve already read those links and attempted the techniques therein, but unfortunately, I have failed. After activating my package and adding dependencies, I got baffling error messages when I did “using MyPackage”. I’ve reverted all of this because I needed to be able to work on my package, and it was no longer functional under this package infrastructure. Because I’ve reverted all these changes, I can’t tell you the literal error messages.
Roughly speaking, it told me that the dependencies had not been properly installed, and it suggested a Pkg command to fix it. The Pkg command it suggested did nothing.
Part of the issue may be that I found no clear explanation of “projects”. I think all the tutorials that explain how to make packages, gloss over which project should be activated at what time.
using SpectralBarrierMethod
[ Info: Precompiling SpectralBarrierMethod [aee288fa-9512-4637-b72d-5ce5ca970827]
ERROR: LoadError: ArgumentError: Package ForwardDiff [f6369f11-7733-5829-9624-2563aa707210] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.
...
Pkg.instantiate()
(nothing happens)
using SpectralBarrierMethod
(same error message)
Personally, I don’t use projects at all, they’re certainly not necessary, but I assume using projects can be convenient in some cases.
Start from installing ForwardDiff into your root environment (which is named after my Julia version, for me, at least). You can do it from the Pkg REPL, which can be entered from the basic REPL by entering ]. After that just enter add ForwardDiff.
I’ve never used environment before. I can’t say I like them, but that’s not my decision to make. If you are aware of a way of making a package that people can install, all that without having to mess with environments, then I’d love to hear it.
Actually, I advise against storing packages that you may modify locally in ~/.julia, when you value the local changes. You can always do dev path/to/package/directory to choose the directory explicitly. PkgTemplates also defaults to storing to ~/.julia for some reason.
Briefly, do ]generate Foo to create a blank package, then add using ForwardDiff to Foo.jl, then ]activate Foo and ]add ForwardDiff. Finally, try using Foo, you’ll get the above error message. No combination of ]instantiate and ]resolve are of any help to me.
I don’t use the generate functionality, I think it’s basically obsoleted by PkgTemplates, so here’s an example of how I set this up using PkgTemplates. I used ExamplePackage as the package name:
[nsajko@aceramd bin]$ ./julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0-beta3 (2023-10-03)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PkgTemplates
julia> generate()
Package name: ExamplePackage
Template keywords to customize:
[press: Enter=toggle, a=all, n=none, d=done, q=abort]
[X] user
[X] authors
[X] dir
[ ] host
[ ] julia
> [ ] plugins
Enter value for 'user' (required): example_user
Enter value for 'authors' (comma-delimited, default: SNIP): example_author
Enter value for 'dir' (default: ~/.julia/dev): /tmp/
[ Info: Running prehooks
[ Info: Running hooks
Activating project at `/tmp/ExamplePackage`
Updating registry at `~/.julia/registries/General.toml`
No Changes to `/tmp/ExamplePackage/Project.toml`
No Changes to `/tmp/ExamplePackage/Manifest.toml`
Precompiling project...
1 dependency successfully precompiled in 1 seconds
Activating project at `~/.julia/environments/v1.10`
[ Info: Running posthooks
[ Info: New package is at /tmp/ExamplePackage
Template:
authors: ["example_author"]
dir: "/tmp/"
host: "github.com"
julia: v"1.0.0"
user: "example_user"
[... cut ...]
(@v1.10) pkg> activate /tmp/ExamplePackage/
Activating project at `/tmp/ExamplePackage`
(ExamplePackage) pkg> add ForwardDiff
Resolving package versions...
Updating `/tmp/ExamplePackage/Project.toml`
[f6369f11] + ForwardDiff v0.10.36
Updating `/tmp/ExamplePackage/Manifest.toml`
[bbf7d656] + CommonSubexpressions v0.3.0
[... cut ...]
[nsajko@aceramd bin]$ ./julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.0-beta3 (2023-10-03)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.10) pkg> dev /tmp/ExamplePackage/
Resolving package versions...
Updating `~/.julia/environments/v1.10/Project.toml`
[63b52bce] + ExamplePackage v1.0.0-DEV `/tmp/ExamplePackage/`
Updating `~/.julia/environments/v1.10/Manifest.toml`
[63b52bce] + ExamplePackage v1.0.0-DEV `/tmp/ExamplePackage/`
julia> using ExamplePackage
Precompiling ExamplePackage
1 dependency successfully precompiled in 2 seconds. 17 already precompiled.
(@v1.10) pkg> st
Status `~/.julia/environments/v1.10/Project.toml`
[63b52bce] ExamplePackage v1.0.0-DEV `/tmp/ExamplePackage/`
[f6369f11] ForwardDiff v0.10.36
[14b8a8f1] PkgTemplates v0.7.45
(@v1.10) pkg> activate /tmp/ExamplePackage/
Activating project at `/tmp/ExamplePackage`
(ExamplePackage) pkg> st
Project ExamplePackage v1.0.0-DEV
Status `/tmp/ExamplePackage/Project.toml`
[f6369f11] ForwardDiff v0.10.36
I don’t have clear ideas as to what might be going wrong when you do it. Maybe try deleting your package’s manifest? I think you’d probably succeed if you tried again from scratch.
One thing that jumps out at me is ;cat Foo/src/Foo.jl, which suggests to me that your working directory is not Foo/, but is rather the parent of Foo/.
Did you look at the Foo/Project.toml to make sure ForwardDiff is added correctly? I would also check your working directory to make sure the Project.toml didn’t end up there for some reason.
And by the way, I’m sorry you’re struggling with this - it shouldn’t be this hard
I believe I have followed your instructions and have failed again. There is one important difference. I have to add using ForwardDiff to Bar/src/Bar.jl, in order to trigger the error. See below.
sebastienloisel@MacBook-Pro julia % julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.3 (2023-08-24)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PkgTemplates
julia> generate()
Package name: Bar
Template keywords to customize:
[press: Enter=toggle, a=all, n=none, d=done, q=abort]
[X] user
[X] authors
> [X] dir
[ ] host
[ ] julia
[ ] plugins
Enter value for 'user' (required): example_user
Enter value for 'authors' (comma-delimited, default: Sébastien Loisel <sloisel@gmail.com> and contributors): example_authors
Enter value for 'dir' (default: ~/.julia/dev): .
[ Info: Running prehooks
[ Info: Running hooks
Activating project at `~/Dropbox/2023/Spectral Barrier Method/julia/Bar`
Updating registry at `~/.julia/registries/General.toml`
No Changes to `~/Dropbox/2023/Spectral Barrier Method/julia/Bar/Project.toml`
No Changes to `~/Dropbox/2023/Spectral Barrier Method/julia/Bar/Manifest.toml`
Precompiling project...
1 dependency successfully precompiled in 1 seconds
Activating project at `~/.julia/environments/v1.9`
[ Info: Running posthooks
[ Info: New package is at /Users/sebastienloisel/Dropbox/2023/Spectral Barrier Method/julia/Bar
Template:
authors: ["example_authors"]
dir: "~/Dropbox/2023/Spectral Barrier Method/julia"
host: "github.com"
julia: v"1.0.0"
user: "example_user"
[...]
(@v1.9) pkg> activate ./Bar
Activating project at `~/Dropbox/2023/Spectral Barrier Method/julia/Bar`
(Bar) pkg> add ForwardDiff
Resolving package versions...
Updating `~/Dropbox/2023/Spectral Barrier Method/julia/Bar/Project.toml`
[f6369f11] + ForwardDiff v0.10.36
Updating `~/Dropbox/2023/Spectral Barrier Method/julia/Bar/Manifest.toml`
[...]
sebastienloisel@MacBook-Pro julia % cat Bar/src/Bar.jl
module Bar
# Write your package code here.
using ForwardDiff
end
sebastienloisel@MacBook-Pro julia % julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.3 (2023-08-24)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.9) pkg> dev ./Bar
Resolving package versions...
Updating `~/.julia/environments/v1.9/Project.toml`
[8c4bdf5c] + Bar v1.0.0-DEV `../../../Dropbox/2023/Spectral Barrier Method/julia/Bar`
Updating `~/.julia/environments/v1.9/Manifest.toml`
[8c4bdf5c] + Bar v1.0.0-DEV `../../../Dropbox/2023/Spectral Barrier Method/julia/Bar`
julia> using Bar
[ Info: Precompiling Bar [8c4bdf5c-6288-441c-a86f-7718170b64ef]
ERROR: LoadError: ArgumentError: Package ForwardDiff [f6369f11-7733-5829-9624-2563aa707210] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.
[...]
EDIT2: to clarify, the load path is mostly used for testing/debugging things, at least nowadays. I don’t have it set up (it has the default value), so I’m guessing you have a leftover setting for load path that makes it prefer packages in the current directory to the global packages, or something like that. This is how it looks for me:
There are a number of different ways. When you’re developing Bar, I typically want to be in that working directory. But you could also make a separate project with Bar as a dependency. Eg, you’re in ./my_project, and it has a subfolder ./my_project/Bar, you can do ] activate ., then add ./Bar, which will make ./my_project/Project.toml with Bar as a dependency.
