# Optim returns slightly different result in unit testing environment

**URL:** <https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460>\
**Category:** General Usage\
**Tags:** optim\
**Created:** [September 3, 2018, 1:32am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460 "2018-09-03T01:32:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 3, 2018, 1:32am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/1 "2018-09-03T01:32:00Z")

</div>

I’ve been scratching my head in the past few hours with no answer. I hope someone has a clue.

I added some unit test cases for the [BoxCoxTrans.jl](https://github.com/tk3369/BoxCoxTrans.jl) package (unregistered) but the test fails only in the unit testing environment due to numerical precision problem. If I run the exact same code in REPL then the results matches exactly the test code.

If anyone wants to replicate the problem, this branch exhibits the problem with `atol=1e-9`.

```nohighlight
(v0.7) pkg> add https://github.com/tk3369/BoxCoxTrans.jl#tk/precision-issue

```

---

<div class="post-metadata">

**Author:** ![crbinz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/crbinz/32/1919_2.png) [@crbinz](https://discourse.julialang.org/u/crbinz)\
**Post date:** [September 3, 2018, 1:49am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/2 "2018-09-03T01:49:50Z")

</div>

Are you running into this (from [NEWS.md](https://github.com/JuliaLang/julia/blob/master/HISTORY.md))?

> - `isapprox(x,y)` now tests `norm(x-y) <= max(atol, rtol*max(norm(x), norm(y)))` rather than `norm(x-y) <= atol + ...` , and `rtol` defaults to zero if an `atol > 0` is specified ([#22742](https://github.com/JuliaLang/julia/issues/22742)).

This bit me a few days ago.

**EDIT** : maybe not, since this only surprised me when updating from julia 0.6 to 0.7, and it looks like your package used 0.7 from the get-go.

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 3, 2018, 1:55am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/3 "2018-09-03T01:55:13Z")

</div>

No… A little more information below.

The log shows that Base.Test calculated lambda of -0.9917203620435803.

```julia
Test Failed at /Users/tomkwong/.julia/dev/BoxCoxTrans/test/runtests.jl:32
  Expression: ≈(λ, -0.99172, atol=precision)
   Evaluated: -0.9917203620435803 ≈ -0.99172 (atol=1.0e-9) 

```

If I run it from REPL, I get:

```julia
julia> lambda(𝐱)
-0.9917203225477127

```

---

<div class="post-metadata">

**Author:** ![crbinz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/crbinz/32/1919_2.png) [@crbinz](https://discourse.julialang.org/u/crbinz)\
**Post date:** [September 3, 2018, 2:00am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/4 "2018-09-03T02:00:12Z")

</div>

I can confirm what you’re seeing both in the test failure and the REPL. But… maybe (probably) I’m missing something, but wouldn’t the test also fail if the REPL result was used?

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 3, 2018, 2:02am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/5 "2018-09-03T02:02:51Z")

</div>

Yes, it fails when I use REPL results in the test script.

For now, I’m working around the problem by using `atol=1e-4` (see [here](https://github.com/tk3369/BoxCoxTrans.jl/blob/681a1e81c947c1a674d6226f41d0fe53211bd4f3/test/runtests.jl#L29)), which is low enough for it to pass. But it bothers me because the calculation shouldn’t be any different depending on whether I run it from the REPL or not…

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 3, 2018, 2:53am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/6 "2018-09-03T02:53:13Z")

</div>

Bounds checking being on or off might cause e.g. SIMD to be used or not used which can slightly change the answer.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 3, 2018, 6:15am UTC](https://discourse.julialang.org/t/optim-returns-slightly-different-result-in-unit-testing-environment/14460/7 "2018-09-03T06:15:36Z")

</div>

I agree with @kristoffer.carlsson, you should not expect exactly identical results in a different environment. You should pick tolerance based on what you asked your algorithm to achieve.

Specifically, consider an explicit tolerance in [the call to `optimize`](https://github.com/tk3369/BoxCoxTrans.jl/blob/681a1e81c947c1a674d6226f41d0fe53211bd4f3/src/BoxCoxTrans.jl#L30), possibly exposed to the user (with a default), and just test that the result is within that range.