we have encountered a weird error.
We have some package and when I run the test in repl, the test fails. But when we run the tests using Pkg.test(), the test fails. We have been expecting that it is due to different environment, therefore when Pkg.test() shows, which environment it creates, we have copied the environment to a safe place and run the test again. They do not fail. I suspect though that the environment is still different, since when we run tests using Pkg.test(), we can see a warning
WARNING: importing deprecated binding ChainRulesCore.Zero into ChainRules.
WARNING: ChainRulesCore.Zero is deprecated, use ZeroTangent instead.
likely near /Users/tomas.pevny/Work/Julia/Pkg/ExplainMill.jl/test/structuralmasks.jl:33
WARNING: importing deprecated binding ChainRulesCore.Composite into ChainRules.
WARNING: ChainRulesCore.Composite is deprecated, use Tangent instead.
from ChainRulesCore, which we normally do not see. Yet and surprisingly, the version of ChainRulesCore in a Manistfest captured from the test environment (copied from /private/var/folders/t0/tf6g1xyn04qcrc6fn_0ljhfr0000gn/T/jl_TEbdwp/ ) matches of what we have in the root library of package and which does not produce such warning.
Does anyone has a clue, what might be going on? Why the test environment I save does not match the environment used during Test?
Not an immediate answer but something that may help you debug, the TestEnv package will activate the test environment of your package and allow you to work inside it. Something like
TestEnv.activate("ProjectName")
What is the actual error that the tests fail with? The deprecation warning shouldn’t cause a test failure (I think).
It’s possible that during testing in your other environment, Pkg resolves new versions and installs them. Can you explicitly go into that test environment, activate it and check the versions? Are they still the versions that you expected? Can you run runtests.jl from your test environment manually (the the test environment activated and seperately with the project environment activated)? Do you still observe the difference?
This is what I did, when I have said that I have copied the Project and Manifest to different location, such that it will not be deleted when test finishes. And that did not work in the sense that my runtest has succeeded, where Pkg.test() is failing. This is rather tricky.
It’s hard to help without code and exact steps to reproduce the issue - can you maybe share the Manifes.toml and Project.toml files in question, together with precise steps on what you did and what you observed?
We have found the culprit. The difference with that during test, Julia is always checking bounds. We had a problem of accessing something outside bounds and we have not realize it. What a devious error.