# About the use of the test suite when developing a package

**URL:** <https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862>\
**Category:** General Usage\
**Tags:** question, package\
**Created:** [March 20, 2024, 9:45am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862 "2024-03-20T09:45:50Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 20, 2024, 9:45am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/1 "2024-03-20T09:45:51Z")

</div>

I have added a test suite to the package that is in development, where I am `using LinearAlgebra` in the `MyPkg/test/runtests.jl` file without `add`-ing it in the sub-environment `test` before the use. I can run the `runtests.jl` script successfully in the terminal if I `cd` to its directory and type `julia runtests.jl`, but I can’t if I am in the `MyPkg` environment and type `] test`. The latter tells me the following error:

```julia
ERROR: LoadError: ArgumentError: Package LinearAlgebra not found in current path.
- Run `import Pkg; Pkg.add("LinearAlgebra")` to install the LinearAlgebra package.

```

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 10:12am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/2 "2024-03-20T10:12:50Z")

</div>

I think this is because `LinearAlgebra` is part of the standard library.

In your specific case, add `LinearAlgebra` to your `test` environment:

```julia
] activate test
add LinearAlgebra

```

~~Since you have a `Project.toml` for the test environment, I did not find a better workflow than doing the following: after loading `Revise.jl`, start Julia in the `MyPkg` base directory:~~

```julia
] activate .
using MyPkg
] activate test
include("test/runtests.jl")

```

EDIT: as @goerz points out, there is a [better way](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/11) of doing this.

(Almost) Any change you make to `MyPkg` will be tracked by `Revise.jl` so you can develop your tests and the package at the same time. Remember to comment out `using MyPkg` in `runtests.jl`; you should re-enable it before running the test suite with

```julia
] test

```

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 20, 2024, 10:32am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/3 "2024-03-20T10:32:56Z")

</div>

Thanks!

> [@mzaffalon](#):
>
> I think this is because `LinearAlgebra` is part of the standard library.

I am very confused by the inconsistency that I can `using` any standard library without `add`-ing it in any situation except when using the test suite.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 20, 2024, 10:34am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/4 "2024-03-20T10:34:41Z")

</div>

> [@mzaffalon](#):
>
> (Almost) Any change

What do you mean by saying “almost”? 🤔

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 10:44am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/5 "2024-03-20T10:44:43Z")

</div>

… and all packages in the default environment.

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 10:45am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/6 "2024-03-20T10:45:53Z")

</div>

See the [limitations](https://timholy.github.io/Revise.jl/stable/limitations/).

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 20, 2024, 11:18am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/7 "2024-03-20T11:18:11Z")

</div>

> [@mzaffalon](#):
>
> … and all packages in the default environment.

Oh, yes, indeed. But that is understandable for a package’s test suit that is to be executed on different platforms. I can’t understand why it treats the standard library differently.

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 11:21am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/8 "2024-03-20T11:21:44Z")

</div>

As you correctly write in your initial post, you must have all packages listed in the `Project.toml` file. You should not test your package using `import("test/runtests.jl")` or `julia runtests.jl`.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [March 20, 2024, 5:38pm UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/9 "2024-03-20T17:38:27Z")

</div>

This is because `] test` creates a sandbox with a special value for [JULIA\_LOAD\_PATH](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_LOAD_PATH)

You can get similar results to `] test` using:

```sh
cd test
julia --project --check-bounds=yes -e 'using Pkg; pkg"dev .."; pkg"instantiate";'
JULIA_LOAD_PATH="@" julia --project --startup-file=no --depwarn=yes --warn-overwrite=yes --warn-scope=yes --check-bounds=yes runtests.jl 

```

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 20, 2024, 6:16pm UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/10 "2024-03-20T18:16:15Z")

</div>

> [@mzaffalon](#):
>
> You should not test your package using `include("test/runtests.jl")` or `julia runtests.jl`.

What? Why not? You absolutely should run your tests in the REPL while you’re developing. You do have to make sure that that REPL is in the correct environment, though (especially, that your current code is dev-installed in that environment) The [TestEnv package](https://github.com/JuliaTesting/TestEnv.jl) is designed to help with that, unless you prefer to set something up manually (my projects always have a `make devrepl` set up so that I can run tests and build the documentation)

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 20, 2024, 6:22pm UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/11 "2024-03-20T18:22:37Z")

</div>

> [@mzaffalon](#):
>
> I did not find a better workflow than doing the following: after loading `Revise.jl`, start Julia in the `MyPkg` base directory:
> 
> ```julia
> ] activate .
> using MyPkg
> ] activate test
> include("test/runtests.jl")
> 
> ```

That seems sketchy. I’m not sure what behavior Julia guarantees if you load a package in one environment and then switch to another environment. What you might want to do instead (in the root folder of the project, and assuming a `test/Project.toml` file):

```julia
] activate test
] dev .
include("test/runtests.jl")

```

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 8:04pm UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/12 "2024-03-20T20:04:57Z")

</div>

As far as I understand, `include("test/runtests.jl")` does not work directly in the `MyPkg` environment nor in `MyPkg/test`, unless one uses the workflow explained [here](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/2).

~~In another post, you say that the workflow is sketchy. Can you explain how you do it?~~  
EDIT: see [this post](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/11).

---

<div class="post-metadata">

**Author:** ![mzaffalon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mzaffalon/32/214168_2.png) [@mzaffalon](https://discourse.julialang.org/u/mzaffalon)\
**Post date:** [March 20, 2024, 8:08pm UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/13 "2024-03-20T20:08:21Z")

</div>

Oops, I see it:

```julia
] activate test
] dev .
include("test/runtests.jl")

```

Thank you for the clarification!

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 21, 2024, 3:32am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/14 "2024-03-21T03:32:49Z")

</div>

Now I am following the following workflow and haven’t found anything unreasonable (is there any?):

1. Open the root folder of `MyPkg` with VScode.
2. In the REPL type `] activate .` to activate the `MyPkg` environment.
3. Write code in `/src/MyPkg.jl` to develop the package and add dependencies in the REPL by `] add SomePkg`.
4. Write code in `/test/runtests.jl` to implement package test. Without doubt the code should begin with `using MyPkg` (but you don’t have to `] add` or `] dev` this package in the REPL). Only when you’re to add test-specific dependencies, `] activate test` in the REPL and `add SomeTestSpecPkg` although the package is part of the standard library of Julia (That’s where I feel rather stange).
5. If you want to run the test, now there are two ways:
  - Come back to the `MyPkg` environment by `] activate .` and type `] test`.
  - Come back to the `MyPkg` environment by `] activate .`, select the code block you’re going to run in `/test/runtests.jl` and tap `Ctrl` + `Enter`.

I’d like to know whether the above workflow is logical and recommendable, and does it coincide with your opinion?

Many thanks!

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 21, 2024, 3:43am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/15 "2024-03-21T03:43:45Z")

</div>

Yes, that all sounds good, except for the very last step (if I’m understanding you correctly, and also if VSCode doesn’t have some magic behavior that I don’t know about):

If your active environment is `MyPkg` and your run testing code, that assumes that all your test dependencies are available in the `MyPkg` environment, which, generally, they aren’t. You have to run test code in the `MyPkg/test` environment. So:

```julia
] activate test
] dev . # make sure you have to correct code for your main project
] include("test/runtests.jl") # or select code and press Ctrl + Enter

```

Or, using the excellent [TestEnv.jl](https://github.com/JuliaTesting/TestEnv.jl):

```julia
] activate .
using TestEnv # hopefully installed in your default environment, just like Revise etc.
TestEnv.activate()
# run testing code

```

Note that

```julia
] activate .
] test.

```

works because `Pkg.test()` runs in a subprocess and knows how to activate the test environment itself.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 21, 2024, 4:09am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/16 "2024-03-21T04:09:39Z")

</div>

> [@goerz](#):
>
> If your active environment is `MyPkg` and your run testing code, that assumes that all your test dependencies are available in the `MyPkg` environment, which, generally, they aren’t. You have to run test code in the `MyPkg/test` environment.

Yes, I agree with it.

But If I `] dev .` in the `test` environment, I found it report error when I run the test with `] test` in the `MyPkg` environment:

```julia
ERROR: can not merge projects

```

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 21, 2024, 4:11am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/17 "2024-03-21T04:11:49Z")

</div>

Yes, that’s a problem (bug, in my opinion) with `Pkg`. You’ll have to delete the `test/Manifest.toml` file for `] test` to work again.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [March 21, 2024, 4:41am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/18 "2024-03-21T04:41:16Z")

</div>

> [@goerz](#):
>
> ```julia
> ] activate test
> ] dev . # make sure you have to correct code for your main project
> 
> ```

I found another approach to adding the `MyPkg` package in the `test` environment that also works:

```julia
] activate test
] dev ../MyPkg

```

I can understand them separately but I can’t understand them together.

`] dev .` works because current root directory is `MyPkg` (`.` stands for the current work directory). It works just like one can activate the `MyPkg` environment by `] activate .` even in the `test` environment.

`] dev ../MyPkg` works in the `test` environment because it treats the `test` folder as the current work directory (I think) and `../MyPkg` directs to its parent folder.

Why does the inconsistency coexist?

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 21, 2024, 6:11am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/19 "2024-03-21T06:11:44Z")

</div>

> [@WuSiren](#):
>
> `] dev ../MyPkg` works in the `test` environment because it treats the `test` folder as the current work directory (I think) and `../MyPkg` directs to its parent folder.

No, much simpler: If you are in a folder `X` then `../X` is the same as `.`

Try it in your shell! 🙂

The current working directory in both cases is `MyPkg`, not the `test` folder.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [March 21, 2024, 6:43am UTC](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862/20 "2024-03-21T06:43:44Z")

</div>

In case it hasn‘t been mentioned, there‘s TestEnv.jl.

[Next page](https://discourse.julialang.org/t/about-the-use-of-the-test-suite-when-developing-a-package/111862.md?page=2)