Packages for "my own" code

New to Julia
1

I am developing code that I expect will be used only by myself for one or two projects (solving a specific model in economics). This depends on a few modules with reusable code (likely reusable mainly by myself for other projects).
My question is: should I organize this code in packages?
The potential benefit is automatic tracking of dependencies.
But then, for different projects to track different versions of the reusable code, the packages would presumably have to be registered (?).
I am reluctant to pollute the registry with many small packages that are likely of interest only to myself.
If I don’t register the packages, how would my code track versions?

2

Not really, you can just pkg> add them directly from the URL. Just commit the Manifest.toml into the project.

2 Likes
3

I have 2 “reusable” modules that I’m using between a few applications. Each of the reusable modules has it’s own GIT repository (currently just on my file system). For the application that uses a module I went into the package manager and did an: add file:///home/pixel/julia/module1 which brought the module into my project.

When I make updates to module1, I commit them to git, then go back to my application, in the package manager do an update, that pulls in the newer version. (This is all under Atom so I then have to restart my julia console so it actually uses the new version, there are probably some commands to force this without the restart.)

1 Like
4

Sorry - not quite following your reply.
Are you suggesting to change the version number, commit to github, and then Pkg can pull the right (earlier) version when needed?
Sorry if there is some documentation that spells all of this out that I am missing.

5

Thanks - that’s pretty much what I am doing now (except via github rather than local storage).
What I am looking for, though, is the ability to come back to the “main” project and have it pull the old version of the “reusable” packages.

1 Like
6

AFAIK there is no way to version non-registered packages (in a way that Pkg is aware of the versions). You might be able to create your own registry though so that you’d register the packages there instead of Julia’s general one.

(Please, someone correct me if I am wrong)

7

In 12. API Reference · Pkg.jl Under Pkg.add it has:

Pkg.add(PackageSpec(url="https://github.com/JuliaLang/Example.jl", rev="master")) # From url to remote gitrepo

Which suggests if you have branches…maybe labels…you should be able to specify which version of the module you want.

PackageSpec has the following constructor:

PackageSpec(; name, url, path, rev, version, mode, level)

My uneducated guess is that version might reference a tag, if rev references a branch. The documentation doesn’t really describe how the values map to a git…

8

One option would be to create a personal package registry. @GunnarFarneback maintains a branch of Registrator.jl (https://github.com/GunnarFarneback/Registrator.jl) that provides commands for managing a registry Git-repo locally. You can easily host such a registry on GitHub without running a Registrator server.

3 Likes
9

Thank you for all the suggestions.
Let me try to summarize the options as I understand them:

  1. Create my own registry. Drawbacks: It is not clear to me from the documentation of GitHub - GunnarFarneback/Registrator.jl how this would interact with the main registry. It is also not clear how I would access the registry on a remote machine (just rsync it?).
  2. Create a local copy of each package each time it is reused. Then add the local path to each projects Project.toml. Seems workable but “inelegant.”
  3. It may be possible to specify non-registered package versions using Pkg, but it’s not clear whether / how that actually works.
  4. Register each package in the general registry, which would create a large number of entries that are not meant for general reuse (each requiring a unique name and manual review for each update).

Am I missing something or this a very common problem without a clear solution?

1 Like
10

I have found working with my own personal registry to work relatively seamlessly with the General registry. That is how I have dealt with this issue. Many thanks to @GunnarFarneback. To access the personal registry on a new machine you just pkg"registry add [url] it. That requires storing the registry somewhere (e.g. your github account). I think you could also just rsync it into the appropriate folder for registries, but I find it easier to have a place where the registry is synced from to make it easier to update later on.

1 Like
11

Thank you for that suggestion. Well, I now tried the personal registry, but without success.
When I issues register(MyPackage, registryPath) I end up with an error:

HEAD is now at 5e6de9f update
ERROR: TOML.ParserError(150, 151, "unexpected character: `9`")

With enough effort, I’m sure one could track this down, but it’s not a promising start.

12

Are you sure it isn’t just a problem with your project’s files? I vaguely remember a similar issue and I think I fixed it by revising the project file for the package I was registering based on the line number in the error (in your case 150… I think?).

13

I think 150 is a byte offset in Registry.toml (which is far shorter than 150 lines).
I rebuilt the registry from scratch and get another parse error which suggests that the UUID is not valid.
I don’t think I will try to debug this myself. Plan B for now is to see if I can make a private registry using Pkg (which the docs say is possible).
Thanks for the suggestion, though.

14

By way of status update:
I tried to make a private registry, but without success (“dirty registry” error; I probably have an inconsistency somewhere). This suggests to me that maintaining a registry by hand is too much work.
The only approach that I can see then is to use Pkg.develop after each edit of the unregistered package dependencies to force a recompile.
When it comes time to “freeze” the package versions of the dependencies, I suppose I could use Pkg.develop with the local option.
Still hoping for a better solution…

15

I have plenty of non-registered packages that are used by

] add https://github.com/PetrKryslUCSD/FinEtoolsDeforNonlinear.jl.git

No problem. That works smoothly.

1 Like
16

I have two problems with this approach:

  1. If A uses B, how to get A to track changes in B? The only way I could find is Pkg.develop(PackageSpec(B)). Is there a better way?
  2. At some point, A becomes a paper submitted for publications. Then I need to freeze the dependencies. How can I do this when B is not registered?

Thank you.

17

To develop the package instead of adding it works fine.

If you keep the package up-to-date with the Project.toml and Manifest.toml, freezing the dependencies is not a problem. You simply freeze these two files.

1 Like
18

When you say “freeze these two files” do you mean editing [compat] in Manifest.toml to fix the version of the dependency?
The question is how Pkg can resolve versions when the package is not registered? I get the following error:

(TestPkg2LH) pkg> test
   Testing TestPkg2LH
 Resolving package versions...
ERROR: empty intersection between TestPkgLH@0.2.0 and project compatibility 0.1.8-0.1
19

Perhaps this is the way:

  1. While editing the project, develop the unregistered dependencies. (I now realize that this changes the url in Manifest.toml to the local path.
  2. To freeze the dependencies, remove and add them (which repoints Manifest.toml to a specific version on github with its commit git-tree-sha1).

Is this the way? Thanks again.

1 Like
20

You got it.