Good guide to setting up a reproducible julia project?

Hello,

I was looking for a good guide on how to set up a new julia project/application, so that it is reproducible, and stays up to date with its dependencies.

My coworkers were setting up a new julia project, and I couldn’t find any good “getting started setting up a new project” advice online. Does anyone know of any existing guides like this?

Here is what I recommended to them:

  • commit your *.toml files (created when you Pkg.add() dependencies)
    • Project.toml
    • Manifest.toml (This provides reproducible builds)
  • Add this CompatHelper file to your .github/workflows directory: https://github.com/JuliaLang/Example.jl/blob/master/.github/workflows/CompatHelper.yml
    • this will open a PR to your repo whenever there is a breaking change released in your dependencies.
    • You should test these PRs and merge them only if everything still works
  • Ideally you should also set up automated CI tests, to make sure that your project continues working as you update the SDK and make other changes.
  • finally, you’ll want to run Pkg.update() every once in a while and commit the changes (after testing) to keep up to date with patch releases of your dependencies

Does a guide already exist that covers these kinds of steps? It’s especially important to me to recommend that they include CompatHelper (and ideally CI.yml as well).

Also, I should say:
I looked at these two pages in the Manual, but I don’t feel that they sufficiently covered the “extra stuff” I mentioned above:

Well, I wrote a guide for reproducible Julia projects; Working with Julia projects | Julia programming notes

You could add Alexander Plavin / CompatHelperLocal.jl · GitLab if that is what you want…

But if you also want CI… Why not setting up a full Julia package?

2 Likes

There is also Home · PkgTemplates.jl which automates some of these steps. I also liked this video: Developing Julia Packages - YouTube

3 Likes

I think I would be happy to recommend them to set up a full Julia Package. :+1: As long as there is a clear and straightforward guide to get started, I think that sounds like a great recommendation, yeah.

Do you feel that there is already a good guide on getting started setting up a Julia package that has all of the above?

No… I will try to write one…

2 Likes

Not necessarily a good guide, but simple it is:

https://m3g.github.io/JuliaNotes.jl/stable/new_package/#How-to-create-a-new-package

(It does NOT have all of the above)

1 Like

With using PkgTemplates; Template(interactive=true)("MyPkg") I think most of the above should be covered as it will ask for the available options which should be added.

2 Likes

I think PkgTemplates might already add CompatHelper.yml by default as well!

PkgTemplates is a great suggestion, but I was worried that for someone just getting started with Julia, that it might be still a bit mysterious how to use it.

But actually on reflection, I think you’re right that it’s the best thing to recommend for new people as well. (It’s what I use to create packages :+1:.)

One is that if they’ve already started their project, it’s less obvious how to turn it into a good project using PkgTemplates (usually i will create a new package and then copy all the files it generated into my git repository… but this is already starting to be pretty unwieldy).

1 Like

But also, regarding this question: do note that in this case they’re trying to set up a private project, to share with their consulting clients, and they do not want to create a public package.

I know that it’s easy to control that, but I just wanted to note that a guide for “how to create a public package” isn’t exactly what we’re after.

Thanks for all the great recommendations so far! My coworker said this guide was particularly helpful, @ufechner7!

Thank you!

Hi, I did some write-up here: Julia: Project Workflow

Currently trying this out with several projects.

2 Likes

Also, you can look here: Introduction · DrWatson

3 Likes

PkgTemplates is a great suggestion, but I was worried that for someone just getting started with Julia, that it might be still a bit mysterious how to use it.

I have a PR to the julia vscode extension which I’m hoping might help for new users if it’s merged. It exposes a Create New Project command that uses PkgTemplates under the hood but without the need for the user to install or understand how to use it.

5 Likes