How to use workspaces to test?

General Usage
1

I’m writing a program and I’d like to add tests. I search the web and find that there’s a new way:

The recommended way to add test-specific dependencies is to use workspaces. This is done by:

  1. Adding a [workspace] section to your package’s Project.toml:
[workspace]
projects = ["test"]

Then what? Having done that, how do I add a test-specific dependency to the workspace? Do I make a test subdirectory? Are workspaces available starting in 1.12 (which I’m running) or 1.13 (which doesn’t exist yet, according to juliaup)?

3 Likes
2

Workspaces are available in 1.12.

This is a good template to follow: GitHub - JuliaBesties/BestieTemplate.jl: Template for Julia Programming Language packages using the copier engine.

As you can see there is a test folder with its own Project.toml. Then there is a runtests.jl which discovers all the tests which you can just copy.

(the template doesn’t use workspaces yet, but I did create an issue for it: Usage of `[workspace]` section in `Project.toml` · Issue #559 · JuliaBesties/BestieTemplate.jl · GitHub, this doesn’t matter for the test folder itself)

3

This looks like a neat feature but restricted to 1.12+, will it be back ported to lts ? Otherwise, can I still use it if my package supports current lts or do I have to wait ?

4

New features are not backported to earlier versions, lts or not.

2 Likes
5

I know the old way, but this is a new project and I just found out there’s a new way. What’s the new way?

If in Project.toml I create a workspace called test, how do I configure a workspace?

Each project in a workspace can include their own dependencies, compatibility information, and even function as full packages.

How do I put these three things into a workspace? I don’t see an explanation or example.

6

That part hasn’t changed if I understand correctly. Your folder test is the workspace.

7

Do I still need to put a Project.toml in test?

8

I added a single spot-check test and got it to pass, using a workspace. Can you check if I’m doing it right?

9

Looks okay to me!

10

I think it is simply ignored in previous versions, so you can use it.

1 Like
11

Most of you already know this, but I cannot hold it in: workspaces are such a game changer for interactive work.

I now define various workspaces in addition to test and docs, eg in a monorepo where I have code for Bayesian estimation of structural econ model, one for profiling where I have tools like PProf.jl to Ctulhu.jl to speed up the code, estimation where I have CSV.jl and JLD2.jl to read data and save results, diagnostics where I have MCMCDiagnosticTools.jl, analysis where I make the plots and tables. No extra precompilation, single manifest, which is committed to a repo, always a consistent work environment.

Thanks @kristoffer.carlsson and everyone who made this feature happen!

18 Likes
12

Sounds very interesting. I’m wondering how exactly you structure such a project. Do you think you could you share the repo where you use it?

1 Like
13

I also have lots of questions about exactly how this works that aren’t answered by the documentation. Most importantly, does anyone know if there’s any path for backwards compatibility? At least to LTS?

14

No, this is a new feature and new features are not backported to earlier versions of Julia (only bug fixes). That is a requirement of semantic versioning.

15

Do you mean that the mere presence of a [workspace] section will cause an error on older Julia versions, or just that we won’t get the new functionality on older versions?

I ask because this new way looks a lot like one of the old ways of including tests — the one that also had a tests/Project.toml file. It seems like you could still make the old way work, even if the [workspace] section was just ignored. An analogy is the situation with extensions: we can use them (less elegantly) on old Julia versions with Requires.jl.

16

Do you mean that the mere presence of a [workspace] section will cause an error on older Julia versions, or just that we won’t get the new functionality on older versions?

Generally, older Julia version will just ignore sections they don’t understand.

Since this topic seems to cause a lot of confusion here’s a full example on how I use it and how it plays with julia versions from 1.10 to 1.12:

./Project.toml

name = "NetworkDynamics"
uuid = "22e9dc34-2a0d-11e9-0de0-8588d035468b"
version = "0.10.14"
authors = [...]

[workspace]
projects = ["test", "benchmark", "docs"]

[deps]
ArgCheck = "dce04be8-c92d-5529-be00-80e4d2c0e197"
Atomix = "a9b6321e-bd34-4604-b9c9-b65b8de01458"

[compat]
ArgCheck = "2.3.0"
Atomix = "1"

./test/Project.toml

name = "NetworkDynamics-Tests"

[deps]
Adapt = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"

[sources]
NetworkDynamics = {path = ".."}

[compat]
Adapt = "≥0.0.1"
Aqua = "0.8.9"

I consider 2 use cases for my sub environments:
A) automated testing (what happens in ] test), mainly for CI
B) manual activation for dev, i.e. to start julia --project=@. in the test folder and start sending parts of your test files to the repl interactively, thats how I do 90% of my Julia work. (similar to what TestEnv.jl achives)

A) just works. Since ever (pre 1.2 maybe?) Julia supports nested test environments. Pkg will see that you have a Project.toml in the test subfolder and use that to create a combined Manifest.

B) is a bit more nuanced:

  • Julia 1.10: That one is a bit annoying. It neither supports Mainfest-v1.10 nor the sources block. So you need to manually ] dev .. from test to add your main package to that env. Then it kinda works. However switching between main env, doc env and test env can be annoying since you need to update all envs independently which will lead to a lot of recompilation. (EDIT: apparently Mainfest-v.1.10 is a thing now!)
  • Julia 1.11: Just activate the environment. Ideally, rename your Manifest to Manifest-v1.11. It will respect the sources section, so it is easier to set up than 1.10, but still has a different manifest in each subfolder (thus cause more compilation because of slighly different versions)
  • Julia 1.12: If you start julia --project=@. in the test dir, it will activate the test env as part of the workspace (indicated as NetworkDynamics/test) and resolve everything in the main Mainfest in root. Problem: I think this detection is blocked if there is a Manifest in test, hence the Manifest-v1.11 for 1.11. (i am not 100% sure on that).

So if you often change between v1.10 and v1.12 in your development it is annoying. Switching between 1.10 and 1.11 is fine, because you’ll have Manifest.toml and Manifest-v1.11.toml in your test folder (and other worspaces). I mainly switch between 1.11 and 1.12, so I use the v1.11 Manifests in root and all my workspaces, while v1.12 resolves everything in one Manifest-v1.12 in root.

Usecase B) is equivalent for docs and benchmark environments. Those sub-envs look ballistically identical to the test one shown above.

Bonus pet peeve: CompatHelper requires you to have compats for all packages. Lets say my main package depends on adapt so does my test env. I don’t want duplicate entries, because they will never be resolved independent of each other. Thats why I put the open Adapt = "≥0.0.1" compat there, just to please CompatHelper.

6 Likes
17

That’s all very helpful information! Thank you very much!

1 Like
18

Julia v1.10.0 didn’t support, v1.10.10 does however. It has been backported at some point.

3 Likes
19

So if I want to run functions in test from the REPL, instead of running ]test, I have to start Julia in the test directory? I can’t start it in the main project directory and then activate test?

I don’t flip between versions 1.10 and 1.12. I’ve been working on a project which will require moving files to be atomic, which is a new feature in 1.12.

20

No that hasn’t changed. You can just do:

julia --project=.
] test