I am trying to use unit test to test the equality of two arrays, wheres the following test:
@test [2.36248e-314, 2.36248e-314] ≈ [0.0, 0.0]
would fail. The two entries in the vector, if compared separately with the same method, would pass the equality test. Is there an extended unit test for equality for arrays in julia?
2 Likes
What do you mean here?
julia> 2.36248e-314 ≈ 0.0
false
1 Like
There is (although as @kristoffer.carlsson pointed out, the two arguments are not \approx individually). You can see which method is being called for any function call in Julia with @which:
julia> @which [2.36248e-314, 2.36248e-314] ≈ [0.0, 0.0]
isapprox(x::AbstractArray, y::AbstractArray) in Base.LinAlg at linalg/generic.jl:1297
julia> @which 2e-314 ≈ 0.0
isapprox(x::Number, y::Number) in Base at floatfuncs.jl:205
and you can open the relevant line in your editor with @edit:
julia> @edit [2.36248e-314, 2.36248e-314] ≈ [0.0, 0.0]
2 Likes
To just test elementwise you could use dot syntax
julia> [1.0, 1e9] ≈ [0.0, 1e9]
true
julia> all([1.0, 1e9] .≈ [0.0, 1e9])
false
7 Likes
If you want to test that an array is approximately zero, you need to either pass an absolute tolerance (atol keyword) to isapprox (you can also do @test x ≈ y atol=sometolerance in 0.7) or just test norm(x) ≤ sometolerance.
Without an atol keyword, ≈ only tests that the relative error is small, and in particular that about half of the significant digits match. 2.36248e-314 ≈ 0.0 is false because none of the significant digits match.
The default atol is zero because there is no sensible way to pick a nonzero default absolute tolerance without additional information. The problem is that the absolute tolerance depends on the overall scale (it is “dimensionful”): the result of x ≈ y shouldn’t change if you change the units/scaling, e.g. if you multiply both x and y by 1e18.
(I feel like this is a FAQ …)
7 Likes
I am actually surprised by this. I tested with 0.9999999999 and 1, which returns true, so I thought there is some default tolerance for equality testing.
1 Like
As Steven described, there exists a default relative tolerance.
Copying the docstring here:
isapprox(x, y; rtol::Real=sqrt(eps), atol::Real=0, nans::Bool=false, norm::Function)
Inexact equality comparison: true if norm(x-y) <= atol + rtol*max(norm(x), norm(y)). The default atol is zero and the default rtol depends on the types of x and y. The
keyword argument nans determines whether or not NaN values are considered equal (defaults to false).
3 Likes
Why didn’t isapprox(x::AbstractArray, y::AbstractArray) disappear as part of the . broadcasting deprecations?
1 Like
Because it does something different from the dotted version.
Similar reason why *(::Matrix, ::Matrix) didn’t get removed.
1 Like
I came across this thread after struggling to make sense out of tests for array elements. It might be worth explicating in the docs. It appears that @test all(isapprox.(x, y, atol=0.05)) is the correct syntax (I couldn’t find anything simpler). One problem is @test isapprox(x, y, atol=0.05) passes with 2 elements per array and this could lead a person to construct the wrong tests in the more general case.
#Both pass
N = 2
x = fill(1.03, N)
y = fill(1.0, N)
@test all(isapprox.(x, y, atol=0.05))
@test isapprox(x, y, atol=0.05)
using Test
N = 5
x = fill(1.03, N)
y = fill(1.0, N)
# only the first passes
@test all(isapprox.(x, y, atol=0.05))
@test isapprox(x, y, atol=0.05)
As noted by Kristoffer Carlsson above,
isapprox(a, b)
compares under some norm determined by the arguments. It is different from
all(isapprox.(a, b))
It is really up to the user to select the best comparison — either one could be useful in some context.
It is important to note that a custom norm can be specified for isapprox, which allows things like
@test a ≈ b norm = my_custom_norm
in tests. I frequently find this useful.
4 Likes
In particular, isapprox(a, b, atol=1e-2) is checking whether norm(a - b) ≤ 1e-2, where norm is the default Euclidean norm. It sounds like what you want (or think you want) is the “infinity norm”. If you define norminf(x) = maximum(abs, x) (or use norm(x, Inf) from the LinearAlgebra package), then you could employ the norm=norminf keyword as @Tamas_Papp suggested.
However, more generally I would tend to recommend not using atol in floating-point approximate-equality tests — a more common choice should be rtol (relative tolerance). For example,
@test x ≈ y rtol=0.05
(equivalently, isapprox(x, y, rtol=0.05)) tests whether x and y are within 5% of one another in the sense that norm(x-y) ≤ rtol*max(norm(x), norm(y)). The reason to use a relative tolerance is that it is scale-invariant (“dimensionless”): multiplying both x and y by any factor (changing the “units”) will not require you to change the test. (Note in particular that this test will always pass when x = 1.03y as in your example, for any finite x and y.)
Moreover, all of the rules of floating-point arithmetic are designed to preserve relative accuracy, not absolute accuracy. For example x + y in floating point is guaranteed to give the exact result up to a relative tolerance of the machine precision ε (eps(Float64) = 2.220446049250313e-16 in double precision), not including overflow. (If you perform multiple addition operations, however, these errors can accumulate.) So it is much easier to select a reasonable rtol than atol.
The isapprox function has a default rtol of √ε, which means that it checks whether about half of the significant digits match. This is a good default for most floating-point unit tests — coarse enough that it won’t give false negatives due to accumulated roundoff errors, but fine enough to catch most bugs that give the wrong answer (as opposed to bugs that simply exacerbate roundoff). Because of that, most floating-point unit tests can simply do @test x ≈ y.
4 Likes
I’ve found the
norminf(x) = maximum(abs, x)
@test x ≈ y rtol = 0.01 norm = norminf
approach to work well, and fail with a clearer error message than the
@test all(isapprox.(x, y, rtol = 0.01))
approach. Could it be made so that
@test x .≈ y rtol = 0.01
is equivalent to the norminf approach? This seems fairly intuitive to me. Currently the above line fails with error
Expression evaluated to non-Boolean
Expression: .≈(x, y, atol = 0.01)
Value: Bool[1, 1]
Note that these tests are not mathematically equivalent.
Consider x = [1,0] and y = [1,1e-6], in which case your first test passes while your second test fails. The first test is checking \Vert x - y \Vert_\infty \le 0.01 \max \{ \Vert x \Vert_\infty, \Vert y \Vert_\infty \} (which is true), while the second test is checking |1 - 1| \le 0.01 \max \{ |1|, |1| \} (which is true) and |0 - 10^{-6}| \le 0.01 \max \{ |0|, |10^{-6}| \} (which is false) separately.
This could be done (because @test is a macro and can rewrite any valid syntax arbitrarily) but I don’t think it should be done, because .≈ suggests an elementwise isapprox.(...) call, which is not the same as isapprox with an infinity norm as noted above.
5 Likes
Ok, makes sense