Trying to understand developing a new package

New to Julia
1

As far as I have been able to figure out, the steps necessary to start developing a new package are:

  1. pkg> generate PkgNm
  2. julia> cd(“PkgNm”)
  3. julia> activate
  4. Replace template file in PkgNm/src/PkgNm.jl by actual file
  5. julia> cd(“…”)
  6. julia> dev PkgNm
  7. using Revise
  8. using PkgNm
    Does this look correct or am I missing something? This process looks overly-complicated to me; have I misunderstood, or just missed something? can it be simplified? I have had to go back and forth between reading documentation for both packages and for revise; is there somewhere that a step-by-step recipe for starting the development of a new package is described.
1 Like
2

Uh, I think 6. might need to be
pkg> dev PkgNm
not
julia> dev PkgNm
eventually I will figure it out; just have to keep plugging away.

3

Yech, and 3. should be
pkg> activate
not
julia> activate
maybe there is a simpler and less convoluted way to achieve this?

4

Checkout GitHub - invenia/PkgTemplates.jl: Create new Julia packages, the easy way I think it helps creating new packages and especially setting them up in Github.

2 Likes
5

Thanks Tero, I have looked at but it seems even more complicated that just using pkg> generate. Generally, I would think that until you got the new package developed to a reasonable level, you do not want to push it up to github?

1 Like
6

I would also recommend you look at PkgSkeleton.jl

However, neither that package nor PkgTemplates.jl actually changes the steps you list. They just get more ready for you. Those steps aren’t so bad though (IMO). You can skip the cd commands by using

]activate PkgNm

2 Likes
7

tbeacon, thank you. It still looks like a convoluted process. For example, in my module I have

using YAML, Printf, MAT

when I first tried using the new development module at step 8), I got

julia> using LpFlt
[ Info: Precompiling LpFlt [a4479d8e-72c7-4f04-95c3-75b31da30ff3]
ERROR: LoadError: ArgumentError: Package LpFlt does not have YAML in its dependencies:
- If you have LpFlt checked out for development and have
  added YAML as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with LpFlt
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:905
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] top-level scope at none:2
 [4] eval at ./boot.jl:331 [inlined]
 [5] eval(::Expr) at ./client.jl:449
 [6] top-level scope at ./none:3
in expression starting at /home/martin/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/LpFlt/src/LpFlt.jl:3
ERROR: Failed to precompile LpFlt [a4479d8e-72c7-4f04-95c3-75b31da30ff3] to /home/martin/.julia/compiled/v1.4/LpFlt/dpTiH_t1zGv.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922

I then did

(@v1.4) pkg> add YAML
  Resolving package versions...
   Updating `~/.julia/environments/v1.4/Project.toml`
  [ddb6d928] + YAML v0.4.0
   Updating `~/.julia/environments/v1.4/Manifest.toml`
  [ddb6d928] + YAML v0.4.0

(@v1.4) pkg> add MAT
  Resolving package versions...
   Updating `~/.julia/environments/v1.4/Project.toml`
 [no changes]
   Updating `~/.julia/environments/v1.4/Manifest.toml`
 [no changes]

Okay, next I did

(@v1.4) pkg> dev LpFlt
[ Info: Resolving package identifier `LpFlt` as a directory at `~/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/LpFlt`.
Path `LpFlt` exists and looks like the correct package. Using existing path.
  Resolving package versions...
   Updating `~/.julia/environments/v1.4/Project.toml`
 [no changes]
   Updating `~/.julia/environments/v1.4/Manifest.toml`
 [no changes]

(@v1.4) pkg> 

julia> using Revise

julia> using LpFlt
[ Info: Precompiling LpFlt [a4479d8e-72c7-4f04-95c3-75b31da30ff3]
┌ Warning: Package LpFlt does not have YAML in its dependencies:
│ - If you have LpFlt checked out for development and have
│   added YAML as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with LpFlt
└ Loading YAML into LpFlt from project dependency, future warnings for LpFlt are suppressed.
jlSimDir /home/martin/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/filtOut.mat
Just wrote xout to /home/martin/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/filtOut.mat

okay, so I now tried resolve:

(@v1.4) pkg> resolve
   Updating `~/.julia/environments/v1.4/Project.toml`
 [no changes]
   Updating `~/.julia/environments/v1.4/Manifest.toml`
 [no changes]

julia> using LpFlt

julia>

All looks good; I then quit julia and restarted it

> julia
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
   Updating `~/.julia/environments/v1.4/Project.toml`
 [no changes]
   Updating `~/.julia/environments/v1.4/Manifest.toml`
 [no changes]
WARNING: Method definition transition(Function, REPL.LineEdit.PrefixSearchState, Any) in module LineEdit at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:1727 overwritten in module HeaderREPLs at /home/martin/.julia/packages/HeaderREPLs/xrHVE/src/HeaderREPLs.jl:327.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition activate(REPL.LineEdit.TextInterface, REPL.LineEdit.ModeState, Any, REPL.Terminals.TextTerminal) in module LineEdit at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:2257 overwritten in module HeaderREPLs at /home/martin/.julia/packages/HeaderREPLs/xrHVE/src/HeaderREPLs.jl:382.
  ** incremental compilation may be fatally broken for this module **

WARNING: Method definition deactivate(REPL.LineEdit.TextInterface, REPL.LineEdit.ModeState, Any, REPL.Terminals.TextTerminal) in module LineEdit at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:2252 overwritten in module HeaderREPLs at /home/martin/.julia/packages/HeaderREPLs/xrHVE/src/HeaderREPLs.jl:396.
  ** incremental compilation may be fatally broken for this module **

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0 (2020-03-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

I have no idea if these warngings are important, it seems if I quit julia again and restart, they go away. Irrespective, when I now try and use my new development package, I get

julia> using LpFlt
[ Info: Precompiling LpFlt [a4479d8e-72c7-4f04-95c3-75b31da30ff3]
┌ Warning: Package LpFlt does not have YAML in its dependencies:
│ - If you have LpFlt checked out for development and have
│   added YAML as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with LpFlt
└ Loading YAML into LpFlt from project dependency, future warnings for LpFlt are suppressed.
jlSimDir /home/martin/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/filtOut.mat
Just wrote xout to /home/martin/Dropbox/Matlab/Complex/KM_ComplexFilterToolbox/juliaFiles/filtOut.mat

julia>

But I thought I had already resolved the dependancies? Once again, very convoluted and the documentation is not very clear on best practices in starting new packages: it’s scattered over documentation for Pkg, for Revise, for PkgTemplates, for PkgSkeleton, etc. And again, it seems to me that when first developing a new package, you really don’t want to push it up to GitHub until you at least get it working? Or is this not correct. It would really help me if there was a line-by-line recipe for first starting development of a new package that has dependcies, without pusing to GitHub, that works without errors. Also, couldn’t this be automated? For example, couldn’t the dev command check the new package file for a using line in the module and automatically add dependencies? I do seem to eventually get something working, but I almost never can get rid of the dependancy warnings.

1 Like
8

One suggestion: put the Revise.jl using into your startup file (see the Revise.jl docs) (assuming you are using Revise pretty much all the time, which seems pretty much necessary).

Here is the way I’m doing things (disclaimer: I’m perhaps an intermediate user, not an expert):

Once when generating the package:

  • generate the package structure using PkgSkeleton.jl (your step 1)
  • edit file details (your step 4)

Then I write a script that does the cd / activate sequence (and could do the using PkgNm as well) (since I will have to do this many times).

I skip the dev PkgNm part since I test and try things with the package directory activated (using the script above).

As for dependencies: Yes, you have to add each by hand. This is really unavoidable because Julia cannot know just from the using YAML statement in your code exactly what you want to add as a dependency (e.g., there may be several YAML packages, potentially even multiple registered ones).

I have written up what I have figured out (again: written by an intermediate user, not an expert, but that may sometimes be a good thing) here. Perhaps it’s useful.

1 Like
9

When you are adding these you have to be in the PkgNm environment and not in your main environment. This is why you get these resolvency issues down the line.

3 Likes
10

Fliks, thank you. In order to get in the PkgNm environment, do I first need to do a cd(“PkgNm”) and then a pkg>activate or can I just do a pkg>activate PkgNm in the directory above PkgNm where I first ran pkg>generate?

11

hendri54, thank you. Some additional questions is you have time?

  1. It seems to me you are saying that one doesn’t need to do both a dev and an activate? I thought both were necessary. It is not clear to me when to use one and when to use the other; when reading docs on Revise I thought it said one was supposed to use dev?
  2. If adding dependancies using an editor, I assume they will be added to the Project.toml file. How does one determine what to use for the UUID of the dependencies being added? I couldn’t find this anywhere; if you also know where this is explained, it would be helpful.
12

  1. activate some/path basically says: From now on, use the Project.toml in some/path to resolve what I mean when I say using SomePackage.

  2. If you issue dev path/to/MyPackage, you are effectively editing Project.toml in the currently active environment. It does not matter what directory you are in.

  3. You would use dev path/to/MyPackage when you are in another environment that has MyPackage as a dependency and you want this package to track whatever code resides in path/to/MyPackage (as opposed to, say, a registered version of MyPackage or a version of MyPackage that is identified by a git commit).

  4. While you work on MyPackage you can just activate its environment and start using it. You need to do so anyway when you want to add dependencies, for example.

  5. You don’t edit Project.toml or Manifest.toml directly. They are adjusted for you by Pkg when you add or dev or rm or update dependencies. It’s actually quite instructive to open the toml files in an editor, issue a command (add Foo) and watch those files update. I think this may help to understand what’s actually going on under the hood.

  6. The UUID is looked up in the Registry, if the package is registered, or in the package’s Project.toml if you either dev the package or add it from a git repo. Theoretically, the registry could contain multiple packages with the same name, but that does not actually happen AFAIK. Not an issue to worry about.

4 Likes
13

Not certain if the answer was satisfactory:

dev vs activate

  • Projects (=environments): activate, generate, instantiate, resolve, …
    • activate: switches to a different package/environment.
    • By default, Julia creates and uses a default “project/environment” that matches its version number (v1.4, at the moment).
    • When developing a product, you would create/use a “project” to control which libraries/versions to use. This way, you can guarantee future operation if ever newer library iterations introduce breaking changes.
  • Package (=libraries): add, remove, build, pin, free, develop, …
    • add: Adds a package/library to the active project/environment - so it can be made available in this project.
    • dev: Checks out git repository of package/library already "add"ed to the active project. This way, you can make your own modifications on the library (typically because you found a bug in an external library and intend to submit a fix).
1 Like
14

The UUID is stored in the Project.toml file of the package/library you want to add.

For example, the project UUID for YAML.jl is found here:
→ https://github.com/BioJulia/YAML.jl/blob/master/Project.toml

name = "YAML"
uuid = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6"
version = "0.4.0"

[deps]
...

But I agree with @hendri54. You typically won’t edit this file directly. It is usually easier to use Julia’s package add function to automatically add it to the active Project.toml file.:

(v1.4) pkg> add YAML

This line will add the YAML package to the active project (“v1.4” in this case). Julia’s package manager automatically looks in Julia’s “General” package registry to find the YAML.jl git repository, and extract the UUID from its Project.toml file.

Some documentation

→ Code Loading · The Julia Language
→ Pkg · The Julia Language
→ Constants · The Julia Language
→ Environment Variables · The Julia Language
→ Environment Variables · The Julia Language

15

I just went through all the same issues recently and the dependencies are one of the main pain points. But yeah, you want to work on the activated environment of your package, add the libraries you want, update the main module (using, import) save and resolve. I’ve been writing down a handy guide for my use. I do use PkgTemplates, and I have that package on my main environment and a saved template on a file on my documents, I open Julia and include the file, loading PkgTemplates and the general template, then I create the package.

I move to the new package folder (I save everything under ~/.julia/dev/), activate the new environment, dev it and start working. With Revise loaded, things are usually smooth. And yes, many times I forget to resolve after adding a dependency or things like that, but I now know how to solve it. I’d love the process to be a little simpler, but once you get the hang of it, it makes sense and having the different environments ends up being a nice feature. I developed some packages in R and that’s hard to do.

Anyway, I hear and understand your pain, it wasn’t super easy.

16

Indeed: You must run pkg> activate PkgNm from the directory above PkgNm (where you first ran pkg> generate). But you can also specify a the path to the project, if you wish. For example:

(v1.4) pkg> activate /home/me/juliaprojects/PkgNm

But please note: you should be activating a Project, and not a Package (which appears to be implied by the name PkgNm). A package is included using the “using” or “import” statement:

using MyPackage1
import MyPackage2

Irrespective of the directory you are in, if you run pkg> activate (without arguments), you will activate the default project set by the shell environment variable JULIA_PROJECT before you launched Julia. You can see this if you run “help activate” from the package manager:

(v1.1) pkg> help activate
  activate
  activate [--shared] path

  Activate the environment at the given path, or the home project environment
  if no path is specified. The active environment is the environment that is modified
  by executing package commands. When the option --shared is given, path will be
  assumed to be a directory name and searched for in the environments folders of
  the depots in the depot stack. In case no such environment exists in any of the
  depots, it will be placed in the first depot of the stack.
17

Using pkg> activate with Packages

@alejandromerchan is correct. There is one good reason to use pkg> activate with Packages. Once you activate a package, you can use pkg> add to add dependencies to your package without editing the Project.toml file manually.

This is especially useful when you wish to publish your packages externally (ex: to the open source community).

However, if your internal, project-specific packages/libraries are not to be published externally, you can just omit/delete the Project.toml & Manifest.toml files from the package subdirectories. If you do this, the project-specific packages inherit the (global) project-level Project.toml & Manifest.toml settings.

It is much easier to develop your internal packages/libraries without package-level Project.toml/Manifest.toml files. You don’t need as much up-front research work to get things up-and-running.

18

@MA_Laforge I did not know that… That’s actually good to know, since many of my packages are for private use. I do like to encapsulate my projects in environments and I do that pretyy much all the time, but this is good to know.

19

I would advise against this approach. It seems to negate the point of having packages to begin with.

When you create a package, you have a completely self-contained environment with all dependencies explicitly stated and their versions fixed by the Manifest. You can then use (using MyPackage) this package elsewhere without having to worry about changes to MyPackage breaking your code later on.

I come from Matlab where none of this was possible. So I would submit a paper for publication, wait 6 months for referee reports only to find that meanwhile I had changed my general purpose code (that was used in code for the submission). So I ended up making manual copies of my general purpose code each time I submitted a paper, just to freeze dependencies.

With Julia, I register my supporting code as packages (in my personal registry). Now when I come back to code months after writing it, it is guaranteed to run. If a version update breaks something, I can just roll back the dependency to an older version and I am back in business.

4 Likes
20

General Purpose Code in Packages/Libraries

Good point about general purpose code @hendri54. There are good reasons to explicitly control a package’s environment even if the code is not “externally” published to people outside your organization. I should have been more explicit.

It is true: Having package-level Project.toml/Manifest.toml files will help ensure the runnability/reproducibility of your “internal” packages if ever you want to use them in other projects/environments. This is especially true if you specify exact version numbers for its dependencies.

Caveat: Initial Development Process

The one caveat here when I do the initial development of my general-purpose packages/libraries. In this case, I find it easier to do the initial development without specifying the Project.toml/Manifest.toml sub-environments for these new packages.

That’s because I usually find the development of new packages to be very dynamic. I often find myself breaking out parts of the code into sub-packages, and then moving around code among these sub-packages to find a more practical hierarchy.

During this initial development, I typically rely on the active project’s environment (Project.toml/Manifest.toml) to control what is accessible by these new packages-under-development. Once things solidify a bit more, and I am ready to push the libraries out into a more centralized location, that’s when I would start including a package-specific environment (Project.toml/Manifest.toml).

Project-Specific Packages

Having said that, I don’t typically find it necessary to specify package-specific environments for packages/libraries that are project-specific (i.e. will not be used elsewhere). In that case, I find it less disruptive to rely on the implicit, project-level environment (Project.toml/Manifest.toml).

There are 2 exceptions here:

  1. Two project-specific packages require 2 different versions of the same library (ex: DSP.jl v1.0 & DSP.jl v1.2). But this is probably a rare occurrence.
  2. Two project-specific packages require different libraries that happen to have the same name (but different UUIDs). Also likely rare.

(There’s probably other exceptions, but I can’t think of them right now.)