About the use of the test suite when developing a package

I have added a test suite to the package that is in development, where I am using LinearAlgebra in the MyPkg/test/runtests.jl file without add-ing it in the sub-environment test before the use. I can run the runtests.jl script successfully in the terminal if I cd to its directory and type julia runtests.jl, but I can’t if I am in the MyPkg environment and type ] test. The latter tells me the following error:

ERROR: LoadError: ArgumentError: Package LinearAlgebra not found in current path.
- Run `import Pkg; Pkg.add("LinearAlgebra")` to install the LinearAlgebra package.

I think this is because LinearAlgebra is part of the standard library.

In your specific case, add LinearAlgebra to your test environment:

] activate test
add LinearAlgebra

Since you have a Project.toml for the test environment, I did not find a better workflow than doing the following: after loading Revise.jl, start Julia in the MyPkg base directory:

] activate .
using MyPkg
] activate test
include("test/runtests.jl")

EDIT: as @goerz points out, there is a better way of doing this.

(Almost) Any change you make to MyPkg will be tracked by Revise.jl so you can develop your tests and the package at the same time. Remember to comment out using MyPkg in runtests.jl; you should re-enable it before running the test suite with

] test
2 Likes

Thanks!

I am very confused by the inconsistency that I can using any standard library without add-ing it in any situation except when using the test suite.

What do you mean by saying “almost”? :thinking:

… and all packages in the default environment.

1 Like

See the limitations.

1 Like

Oh, yes, indeed. But that is understandable for a package’s test suit that is to be executed on different platforms. I can’t understand why it treats the standard library differently.

As you correctly write in your initial post, you must have all packages listed in the Project.toml file. You should not test your package using import("test/runtests.jl") or julia runtests.jl.

1 Like

This is because ] test creates a sandbox with a special value for JULIA_LOAD_PATH

You can get similar results to ] test using:

cd test
julia --project --check-bounds=yes -e 'using Pkg; pkg"dev .."; pkg"instantiate";'
JULIA_LOAD_PATH="@" julia --project --startup-file=no --depwarn=yes --warn-overwrite=yes --warn-scope=yes --check-bounds=yes runtests.jl 
3 Likes

What? Why not? You absolutely should run your tests in the REPL while you’re developing. You do have to make sure that that REPL is in the correct environment, though (especially, that your current code is dev-installed in that environment) The TestEnv package is designed to help with that, unless you prefer to set something up manually (my projects always have a make devrepl set up so that I can run tests and build the documentation)

2 Likes

That seems sketchy. I’m not sure what behavior Julia guarantees if you load a package in one environment and then switch to another environment. What you might want to do instead (in the root folder of the project, and assuming a test/Project.toml file):

] activate test
] dev .
include("test/runtests.jl")
2 Likes

As far as I understand, include("test/runtests.jl") does not work directly in the MyPkg environment nor in MyPkg/test, unless one uses the workflow explained here.

In another post, you say that the workflow is sketchy. Can you explain how you do it?
EDIT: see this post.

Oops, I see it:

] activate test
] dev .
include("test/runtests.jl")

Thank you for the clarification!

1 Like

Now I am following the following workflow and haven’t found anything unreasonable (is there any?):

  1. Open the root folder of MyPkg with VScode.
  2. In the REPL type ] activate . to activate the MyPkg environment.
  3. Write code in /src/MyPkg.jl to develop the package and add dependencies in the REPL by ] add SomePkg.
  4. Write code in /test/runtests.jl to implement package test. Without doubt the code should begin with using MyPkg (but you don’t have to ] add or ] dev this package in the REPL). Only when you’re to add test-specific dependencies, ] activate test in the REPL and add SomeTestSpecPkg although the package is part of the standard library of Julia (That’s where I feel rather stange).
  5. If you want to run the test, now there are two ways:
    • Come back to the MyPkg environment by ] activate . and type ] test.
    • Come back to the MyPkg environment by ] activate ., select the code block you’re going to run in /test/runtests.jl and tap Ctrl + Enter.

I’d like to know whether the above workflow is logical and recommendable, and does it coincide with your opinion?

Many thanks!

Yes, that all sounds good, except for the very last step (if I’m understanding you correctly, and also if VSCode doesn’t have some magic behavior that I don’t know about):

If your active environment is MyPkg and your run testing code, that assumes that all your test dependencies are available in the MyPkg environment, which, generally, they aren’t. You have to run test code in the MyPkg/test environment. So:

] activate test
] dev . # make sure you have to correct code for your main project
] include("test/runtests.jl") # or select code and press Ctrl + Enter

Or, using the excellent TestEnv.jl:

] activate .
using TestEnv  # hopefully installed in your default environment, just like Revise etc.
TestEnv.activate()
# run testing code

Note that

] activate .
] test.

works because Pkg.test() runs in a subprocess and knows how to activate the test environment itself.

1 Like

Yes, I agree with it.

But If I ] dev . in the test environment, I found it report error when I run the test with ] test in the MyPkg environment:

ERROR: can not merge projects

Yes, that’s a problem (bug, in my opinion) with Pkg. You’ll have to delete the test/Manifest.toml file for ] test to work again.

1 Like

I found another approach to adding the MyPkg package in the test environment that also works:

] activate test
] dev ../MyPkg

I can understand them separately but I can’t understand them together.

] dev . works because current root directory is MyPkg (. stands for the current work directory). It works just like one can activate the MyPkg environment by ] activate . even in the test environment.

] dev ../MyPkg works in the test environment because it treats the test folder as the current work directory (I think) and ../MyPkg directs to its parent folder.

Why does the inconsistency coexist?

No, much simpler: If you are in a folder X then ../X is the same as .

Try it in your shell! :slight_smile:

The current working directory in both cases is MyPkg, not the test folder.

1 Like

In case it hasn‘t been mentioned, there‘s TestEnv.jl.

1 Like