# Test fails when run through Pkg.test() but not when run interactively

**URL:** <https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352>\
**Category:** General Usage\
**Created:** [August 30, 2021, 1:28pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352 "2021-08-30T13:28:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 30, 2021, 1:28pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/1 "2021-08-30T13:28:06Z")

</div>

Hello,

we have encountered a weird error.  
We have some package and when I run the test in repl, the test fails. But when we run the tests using `Pkg.test()`, the test fails. We have been expecting that it is due to different environment, therefore when `Pkg.test()` shows, which environment it creates, we have copied the environment to a safe place and run the test again. They do not fail. I suspect though that the environment is still different, since when we run tests using `Pkg.test()`, we can see a warning

```julia
WARNING: importing deprecated binding ChainRulesCore.Zero into ChainRules.
WARNING: ChainRulesCore.Zero is deprecated, use ZeroTangent instead.
  likely near /Users/tomas.pevny/Work/Julia/Pkg/ExplainMill.jl/test/structuralmasks.jl:33
WARNING: importing deprecated binding ChainRulesCore.Composite into ChainRules.
WARNING: ChainRulesCore.Composite is deprecated, use Tangent instead.

```

from ChainRulesCore, which we normally do not see. Yet and surprisingly, the version of ChainRulesCore in a Manistfest captured from the test environment (copied from `/private/var/folders/t0/tf6g1xyn04qcrc6fn_0ljhfr0000gn/T/jl_TEbdwp/` ) matches of what we have in the root library of package and which does not produce such warning.

Does anyone has a clue, what might be going on? Why the test environment I save does not match the environment used during Test?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![mmiller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mmiller/32/20805_2.png) [@mmiller](https://discourse.julialang.org/u/mmiller)\
**Post date:** [August 30, 2021, 1:34pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/2 "2021-08-30T13:34:42Z")

</div>

Not an immediate answer but something that may help you debug, the [`TestEnv`](https://www.google.com/search?client=safari&rls=en&q=testenv.jl&ie=UTF-8&oe=UTF-8) package will activate the test environment of your package and allow you to work inside it. Something like

```nohighlight
TestEnv.activate("ProjectName")

```

What is the actual error that the tests fail with? The deprecation warning shouldn’t cause a test failure (I think).

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 30, 2021, 1:49pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/3 "2021-08-30T13:49:58Z")

</div>

> [@Tomas\_Pevny](#):
>
> Does anyone has a clue, what might be going on? Why the test environment I save does not match the environment used during Test?

It’s possible that during testing in your other environment, `Pkg` resolves new versions and installs them. Can you explicitly go into that test environment, activate it and check the versions? Are they still the versions that you expected? Can you run `runtests.jl` from your test environment manually (the the test environment activated and seperately with the project environment activated)? Do you still observe the difference?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 30, 2021, 3:02pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/4 "2021-08-30T15:02:36Z")

</div>

This is what I did, when I have said that I have copied the Project and Manifest to different location, such that it will not be deleted when test finishes. And that did not work in the sense that my runtest has succeeded, where Pkg.test() is failing. This is rather tricky.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [August 30, 2021, 3:04pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/5 "2021-08-30T15:04:38Z")

</div>

It’s hard to help without code and exact steps to reproduce the issue - can you maybe share the Manifes.toml and Project.toml files in question, together with precise steps on what you did and what you observed?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 30, 2021, 3:41pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/6 "2021-08-30T15:41:06Z")

</div>

We have found the culprit. The difference with that during test, Julia is always checking bounds. We had a problem of accessing something outside bounds and we have not realize it. What a devious error.

Thanks for help. We have again learned a lot.

Tomas

---

<div class="post-metadata">

**Author:** ![mmiller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mmiller/32/20805_2.png) [@mmiller](https://discourse.julialang.org/u/mmiller)\
**Post date:** [August 30, 2021, 4:07pm UTC](https://discourse.julialang.org/t/test-fails-when-run-through-pkg-test-but-not-when-run-interactively/67352/7 "2021-08-30T16:07:38Z")

</div>

Glad you found it, `@inbounds` can definitely produce some confusing errors until you realise it’s the culprit!