Unable to reproduce unit test outcomes from package manager's "clean environment"

Tooling Package Management
,
1

Hi everyone,
I ran into a strange behaviour recently related to Julia environments. Let’s say I develop a package MyPackage with the common structure:

├── Manifest.toml
├── Project.toml     
├── src  
└── test
    ├── runtests.jl
    └── some_tests_included_in_runtests.jl

with the Project.toml having [deps], [extras] & [targets]. When I activate this package by typing Pkg.activate(".") in that folder and I run runtests.jl all unit tests in some_tests_included_in_runtests.jl work just fine as expected. However, when I test the package by using the package manager via ] test MyPackage from any environment I’ll get a different result and the tests error. This must be some version-clashing. So I read a bit and found out that ] test MyPackage sets up a “clean” environment, activating this, instantiating the package and runs the tests. So I am trying to reproduce these outcomes somehow and I am not able to do it.

In order to also mimic a “clean” environment I tried a few things:

  1. I removed [extras] & [targets] from the Project.toml and put it in another Project.toml IN my /test-folder (together with a Manifest.toml):
├── Manifest.toml
├── Project.toml     
├── src  
└── test
    ├── runtests.jl
    ├── Manifest.toml
    ├── Project.toml  
    └── some_tests_included_in_runtests.jl

Then I activated the /test-folder, instantiated and also dev path/to/MyPackage. Then I ran the runtests.jl, but the tests pass and I can not reproduce the failing tests.

  1. To make sure it is really using only the deps I want I also included empty!(LOAD_PATH) followed by push!(LOAD_PATH,"Project.toml") at the beginning of the runtests.jl (this will use the ./test/Project.toml). Still, the tests do pass.

What am I missing here? I’d like to know an easy way of setting up a clean environment and populate it with MyPackage-dependencies, in order to write robust tests and being able to exactly reproduce what ] test MyPackage is doing. No need to mention that when the different ways of running the tests already fail on my own machine, that they indeed fail on a GitHub-runner on different OS’ as well. This never happened to me before while developing other packages and the tests results were always identical on my local machine no matter which way I executed runtests.jl.

Any help is appreciated,
Best

Hauke

2

Your problem might come from interaction with your main environment (@v1.7). Maybe you have some packages in there, which are needed for the tests, but which aren’t available in the pkg test environment? I’d try an ] up in the main environment. Also, check your ~/.julia/config/startup.jl for using/import statements which might load some packages at incompatible versions. Maybe try

~ julia --startup-file=no
julia> ] activate .
julia> include("test/runtestes.jl")

and see if you can reproduce the test failures.

3

Finally, I found the source of the error and this is not very promising…
Instead of

costs = zeros(T, NN, D)
@inbounds for (i,v) in enumerate(ns)
   costs[i,:] = (Vector(prediction[i]) .- Vector(Y[v+Tw])).^2
end

this here

costs = abs2.(Matrix(prediction) - Matrix(Y[ns.+Tw]))

works perfectly on all machines and runners and test-environments. So in the end that was not an issue with the test-environment. Confusing…