How to test that bounds checks are happening

General Usage
1

When running CI, --check-bounds=yes is passed to julia as a startup option. How can I test to make sure I’m not misusing @inbounds?

In this example, my bounds checking is broken, but the test still passes:

x@X ~ % julia --banner=no --check-bounds=yes
julia> function bad(v, i)
           i < firstindex(v) && i > lastindex(v) && threw(BloundsError()) # bad
           @inbounds v[i]
       end
bad (generic function with 1 method)

julia> using Test

julia> @test_throws BoundsError bad([1,2,3], 4)
Test Passed
      Thrown: BoundsError

In a more real-world example, I want to test that a sorting function throws a bounds error when passed out of bounds indices rather than segfaulting. @test_throws sort!(...) will pass even when it shouldn’t.

2

Why should the test not pass? You are testing if the expression throws a BoundsError, it does, so the test pass.

3

How can I write a test that will fail in this case? How can I test to make sure that the BoundsError did not come from within an @inbounds block?

1 Like
4

I would extend Base.checkbounds(Bool, ::A, ::I) with a new method, and test that directly.

5

Making a new AbstractArray type NeverInboundsArray which would override getindex to throw BoundsError (or another TestError) error? (i.e. always check bounds explicitly and then pass through to underlying array)

1 Like
6

throw and test for ArgumentError instead?

7

Not super elegant but you could do throw(MyBoundsError()) in your own bounds check and define

MyBoundsError() = BoundsError()

Then in the tests you redefine it to something else which you can distinguish from BoundsError.

8

What is A? In neither example am I defining a subtype of AbstractArray.

If NeverInboundsArray throws a BoundsError, then tests would pass (bad). If NeverInboundsArray throws a different error, then the tests above would fail (good), but this would also fail (bad)

function good(v, i)
    @boundscheck checkbounds(v, i)
    @inbounds v[i]
end
@test_throws BoundsError good(NeverInboundsArray([1,2,3]), 4)

And this would pass regardless (bad)

function bad(v, i)
    i < firstindex(v) && i > lastindex(v) && threw(BloundsError()) # bad
    @inbounds good(v, i)
end
function good(v, i)
    @boundscheck checkbounds(v, i) # Throws BoundsError here when --check-bounds=yes
    @inbounds v[i]
end
@test_throws BoundsError bad(NeverInBoundsArray([1,2,3]), 4)

This is the best solution so far, but not great.

A BoundsError is most appropriate in this case, and because the error is user-facing I don’t want to compromise on its type.

If possible, I’d like to keep the source code elegant and do any necessary shenanigans only in testing. I’m worried it might be just as error-prone to do this as to leave the functionality untested.

To clarify, I’m looking to test that “This would throw a bounds error whether or not we are running Julia with --check-bounds=yes”.

To adapt @Dan’s answer,

function Base.getindex(a::NeverInBoundsArray, inds...)
    if !checkbounds(Bool, a.x, inds...)
        error = if @inbounds_is_active_in_current_scope
            SegfaultError()
        else
            BoundsError(a, inds)
        end
        throw(error)
    end
    @inbounds getindex(a.x, inds...)
end

would work, but I don’t know how to get the @inbounds_is_active_in_current_scope macro to work when Julia is run with --check-bounds=yes.

Alternatively, if Julia automatically converted BoundsErrors thrown in @checkbounds checks that are in the scope of @inbounds into SegfaultErrors or ElidedBoundsErrors, then the code in the OP would be correct.

9

What is A? In neither example am I defining a subtype of AbstractArray.

A is your type here. Although the docs for checkbounds mention “array”, I don’t think this function needs to be array-specific. Indeed, it’s also used for strings in Base.

10

I think the @boundscheck macro would help. It marks a block as a boundcheck block and doesn’t compile when @inbounds is active. Something like:

if !checkbounds(Bool, a.x, inds...)
        error = SegfaultError()
        @boundscheck error = BoundsError(a, inds)
        throw(error)
    end

note: couldn’t get this to work normally on REPL.
A little demo of @boundscheck in REPL.

julia> test(v) = begin
       @inline 
       error = BoundsError()
       @boundscheck error = SegmentationFault()
       throw(error)
       end

test (generic function with 1 method)

julia> test2(v) = @inbounds test(v)
test2 (generic function with 1 method)

julia> test2(rand(100))    # runs with @inbounds
ERROR: BoundsError

julia> test(rand(100))     # runs without @inbounds
ERROR: SegmentationFault()

Does this help solve the problem? Not if --bounds-check=yes since then the outputs are the same.

11

Do you actually just want --check-bounds=auto?

If you’re using the julia-runtest action in GitHub you can set check_bounds: auto.

1 Like
12

Yes the --check-bounds=yes as a default for CI makes sense, but I guess there is no way to overwrite this for single annotations? In order to use CI to test that @boundscheck is compiled away with @inbounds I’ve set up some tests like

f(A, i) = @inbounds A[i]
@test f(A, j) == A[k]

because for my custom array the out of bounds index j points to the element at k which isn’t correct but convenient for testing. So I’d need a @enforce_inbounds that’s always @inbounds regardless the CI settings.