# Inconsistent Test results between VScode TestItems.jl and Test.jl

**URL:** <https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344>\
**Category:** VS Code\
**Created:** [November 16, 2022, 12:25pm UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344 "2022-11-16T12:25:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maucejo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maucejo/32/39090_2.png) [@Maucejo](https://discourse.julialang.org/u/Maucejo)\
**Post date:** [November 16, 2022, 12:25pm UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344/1 "2022-11-16T12:25:21Z")

</div>

Hi all,

When testing a package I develop, I had an inconsistency in the result of the unit test of one of my functions between the testing framework TestItems.jl used within the VSCode UI and Test.jl.

Here is a code sample for reproducibility purpose

```julia
using TestItems
using TestItemRunner

@run_package_tests # For using ] test

@testitem "Test" begin
    Δt = 1e-6
    tf = 7e-2
    t = 0.:Δt:tf

    Ft = zeros(length(t))
    tb = 8e-3
    duration = 5e-2

    tm = (2tb + duration)/2.
    pos_beg = argmin((t .- tb).^2.)
    pos_m = argmin((t .- tm).^2.)
    pos_end = argmin((t .- tb .- duration).^2.)
    amp = 2/duration

    Ft[pos_beg:pos_m] = amp*(t[pos_beg:pos_m] .- tb)
    Ft[pos_m+1:pos_end] = 1. .- amp*(t[pos_m+1:pos_end] .- tm)

    @test sum(diff(Ft)) == 0.
end

```

When using TestItems.jl within the VSCode UI, the test is successful (I have also checked this assertion in the REPL). However, when using `] test`, the test fails and the following message is displayed:

```julia
Test: Test Failed at \test\runtests.jl
  Expression: sum(diff(Ft)) == 0.0
   Evaluated: 2.220446049250313e-16 == 0.0

```

On my machine, 2.220446049250313e-16 is the machine epsilon. Of course, because floating-point arithmetics, some differences can occur between what we expect and what is computed. However, in the present case, it seems that it is not the case, since when I execute `sum(diff(Ft))` in the REPL the result is 0. as expected.

Is it a bug or is it the expected behaviour ?

Thanks for your comments

---

<div class="post-metadata">

**Author:** ![kfrb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kfrb/32/29557_2.png) [@kfrb](https://discourse.julialang.org/u/kfrb)\
**Post date:** [November 16, 2022, 1:58pm UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344/2 "2022-11-16T13:58:52Z")

</div>

I cannot say much regarding `TestItems.jl`, but I always use the [`isapprox`](https://docs.julialang.org/en/v1/base/math/#Base.isapprox) function when comparing floats.  
You can also use the binary operator `≈` (`\approx<TAB>`) which is equivalent to [`isapprox`](https://docs.julialang.org/en/v1/base/math/#Base.isapprox) with default arguments.

```julia-repl
julia> 2.220446049250313e-16 ≈ 0.0
false

julia> isapprox(2.220446049250313e-16, 0.0; atol=eps())
true

```

Nevertheless, for your example, I can reproduce the behavior.  
Maybe it has more to do with the computation itself, as the same error occurs when using the standard approach via `@testset`.

---

<div class="post-metadata">

**Author:** ![Maucejo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maucejo/32/39090_2.png) [@Maucejo](https://discourse.julialang.org/u/Maucejo)\
**Post date:** [November 18, 2022, 9:33am UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344/3 "2022-11-18T09:33:25Z")

</div>

Thanks @kfrb for your answer.

As you said the same error occurs when using the standard approach via `@testset`.  
When the code is executed in the REPL, the test is passed, while when it is executed using `] test`, it fails.

I agree with you regarding `isapprox`. However, it raises a problem of reproducibility. I don’t know how to trace the computations done when using `] test`. I will try to figure out what happens here.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [November 18, 2022, 9:44am UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344/4 "2022-11-18T09:44:28Z")

</div>

`] test` runs with bounds checking on which can inhibit some optimizations like SIMD (which can change the result slightly). And indeed, running the code in a julia with `--check-bounds=yes` gives

```julia
julia> sum(diff(Ft)) 
2.220446049250313e-16

```

---

<div class="post-metadata">

**Author:** ![Maucejo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maucejo/32/39090_2.png) [@Maucejo](https://discourse.julialang.org/u/Maucejo)\
**Post date:** [November 18, 2022, 4:10pm UTC](https://discourse.julialang.org/t/inconsistent-test-results-between-vscode-testitems-jl-and-test-jl/90344/5 "2022-11-18T16:10:24Z")

</div>

Thank you very much @kristoffer.carlsson for this explanation that clarifies everything.