# More informative \`@inferred\`, especially for unit testing?

**URL:** <https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481>\
**Category:** Internals & Design\
**Created:** [August 21, 2017, 2:48pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481 "2017-08-21T14:48:20Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 21, 2017, 2:48pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/1 "2017-08-21T14:48:20Z")

</div>

`@inferred` doesn’t seem to be sufficiently informative (especially in unit test sets), see e.g. [https://github.com/JuliaLang/julia/issues/23371](https://github.com/JuliaLang/julia/issues/23371) : contrary to the error message `@inferred` doesn’t show up on line 177 of test/statistics.jl, but at different places in the subsequent 68 lines.

I wrote a modified version of `@inferred` that shows the expression for which the inference test failed (see infer.jl in [https://gist.github.com/afniedermayer/9d3675e0a67ba140256120740127e6a8](https://gist.github.com/afniedermayer/9d3675e0a67ba140256120740127e6a8) ). Here’s what it does

```julia
julia> include("infer.jl")
julia> f(a,b,c) = b > 1 ? 1 : 1.0
julia> @inferred f(1,2,3)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64} in Expression f(1, 2, 3)
Stacktrace:
 [1] error(::String) at .\error.jl:21

```

I can make a PR, but I’d like to have some feedback first.

(There’s an additional issue that unit testing is aborted if there is an `@inferred` that fails, but I’m not sure what to do about that.)

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 22, 2017, 4:44pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/2 "2017-08-22T16:44:44Z")

</div>

There’s the additional issue that `@inferred` isn’t a unit test, so that

```julia
f(a,b,c) = b > 1 ? 1 : 1.0
@testset "foo" begin
  @inferred f(1,2,3)
  @test 2==3
  @inferred f(1,2,4)
  @test 4==5
end

```

will show only **one** failed unit test rather than four. Once [https://github.com/JuliaLang/julia/issues/21594](https://github.com/JuliaLang/julia/issues/21594) gets fixed, it will show **two** rather than four failed unit tests, which is a step forward, but doesn’t go all the way.

Would it make sense to have an `@isinferred` macro that returns a boolean? Here’s a simple (but not very elegant) implementation:

```julia
macro isinferred(ex)
 quote try
          @inferred $ex
          true
       catch err
         println(err)
         false
       end
 end
end

```

Usage:

```julia
@testset "foo" begin
  @test @isinferred f(1,2,3)
  @test 2==3
  @test @isinferred f(1,2,4)
  @test 4==5
end

```

This shows four failed tests and provides more informative messages about test failures.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [August 23, 2017, 9:26pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/3 "2017-08-23T21:26:41Z")

</div>

This seems like an excellent feature to me – definitely a worthwhile improvement.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [August 24, 2017, 4:31am UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/4 "2017-08-24T04:31:57Z")

</div>

Not sure if this helps you, but we do this quite frequently:

```julia
julia> @testset "foo" begin
         @test @inferred(f(1,2,3)) == 1
         @test 2==3
         @test @inferred(f(1,2,4)) == 1
         @test 4==5
       end
foo: Error During Test
  Test threw an exception of type ErrorException
  Expression: @inferred(f(1, 2, 3)) == 1
  return type Int64 does not match inferred return type Union{Float64, Int64}
  Stacktrace:
   [1] macro expansion at ./REPL[7]:2 [inlined]
   [2] macro expansion at ./test.jl:860 [inlined]
   [3] anonymous at ./<missing>:?
foo: Test Failed
  Expression: 2 == 3
   Evaluated: 2 == 3
Stacktrace:
 [1] macro expansion at ./REPL[7]:3 [inlined]
 [2] macro expansion at ./test.jl:860 [inlined]
 [3] anonymous at ./<missing>:?
foo: Error During Test
  Test threw an exception of type ErrorException
  Expression: @inferred(f(1, 2, 4)) == 1
  return type Int64 does not match inferred return type Union{Float64, Int64}
  Stacktrace:
   [1] macro expansion at ./REPL[7]:4 [inlined]
   [2] macro expansion at ./test.jl:860 [inlined]
   [3] anonymous at ./<missing>:?
foo: Test Failed
  Expression: 4 == 5
   Evaluated: 4 == 5
Stacktrace:
 [1] macro expansion at ./REPL[7]:5 [inlined]
 [2] macro expansion at ./test.jl:860 [inlined]
 [3] anonymous at ./<missing>:?
Test Summary: | Fail Error Total
foo | 2 2 4
ERROR: Some tests did not pass: 0 passed, 2 failed, 2 errored, 0 broken.

```

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 9:21am UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/5 "2017-08-24T09:21:24Z")

</div>

Thanks, @anon94023334. What’s good about `@test @inferred(f(1,2,3)) == 1` is that it shows up as a unit test and doesn’t abort the testset when inference fails. But failure of inference shows up as “Error During Test” rather than “Test Failed”. You also have to check for the return value (which may be a good or a bad thing; bad if the return value is too complicated or random).

In julia/test out of the 124 times `@inferred` is used, it is only in 35 cases that it is used in combination with `@test`:

```julia
C:\Users\andras\git\julia\test>grep "@test.*@inferred " *.jl */*.jl | wc
     35 295 2812

C:\Users\andras\git\julia\test>grep "@inferred " *.jl */*.jl | wc
    124 714 7413

```

Some of the `@inferred`s without `@test` are using random numbers, they probably couldn’t be rewritten with the `@test (@inferred f(1,2,3))==1` trick. Others probably could (should?) be rewritten that way.

I’ll open a PR adding `@isinferred` once I have enough time.

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 9:58am UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/6 "2017-08-24T09:58:17Z")

</div>

Before I start implementing `@isinferred`: is there a reason why `@inferred` compares the actual return value with the inferred type ( [https://github.com/JuliaLang/julia/blob/b056235d76d37f2dac309055b619b8f076c8ecb0/base/test.jl#L1175](https://github.com/JuliaLang/julia/blob/b056235d76d37f2dac309055b619b8f076c8ecb0/base/test.jl#L1175) ) rather than just checking whether the inferred type is abstract: `!isleaftype(Base.return_type(...))[1]`? Would this miss some special cases?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [August 24, 2017, 12:06pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/7 "2017-08-24T12:06:35Z")

</div>

The macro was originally written by @simonster and that part most recently tweaked by him as well, so perhaps he can shed some light on the issue.

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 12:25pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/8 "2017-08-24T12:25:18Z")

</div>

I went ahead and started implementing `@isinferred` with `isleaf`. While trying to make sure that `@isinferred` passes analogous tests to `@inferred`, I ran into a bunch of problems with special cases (I guess there’s a reason so much tweaking had to go into `@inferred`.)

In the end, I decided to simply call `@inferred` from `@isinferred` and catch exceptions.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 24, 2017, 12:48pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/9 "2017-08-24T12:48:33Z")

</div>

Couldn’t you rename `@inferred` to `@isinferred`, change the exception throwing to returning `false` and add a new `@inferred` that does something like:

```julia
macro inferred (ex)
    if !@isinferred($(esc(ex))
        throw(...)
    end
end

```

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 12:56pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/10 "2017-08-24T12:56:40Z")

</div>

`@inferred` also gives an error message with a description of the returned type and the inferred type. This couldn’t be recovered from a boolean.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 24, 2017, 1:08pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/11 "2017-08-24T13:08:27Z")

</div>

> [@Andras\_Niedermayer](#):
>
> started implementing @isinferred with isleaf

Please package it up and possibly register that, I would start using this immediately.

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 1:30pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/12 "2017-08-24T13:30:11Z")

</div>

@Tamas_Papp, I’m happy to hear that 🙂 I just made a WIP pull request (I’m waiting for tests to pass on the server), but the code is already there and works, feel free to use it: [https://github.com/JuliaLang/julia/pull/23426](https://github.com/JuliaLang/julia/pull/23426)  
I’m not sure that the code would warrant a separate package, because it’s relatively short. I spent most of the time documenting and making sure that unit tests pass.

---

<div class="post-metadata">

**Author:** ![Andras\_Niedermayer](https://avatars.discourse-cdn.com/v4/letter/a/7ba0ec/32.png) [@Andras\_Niedermayer](https://discourse.julialang.org/u/Andras_Niedermayer)\
**Post date:** [August 24, 2017, 3:53pm UTC](https://discourse.julialang.org/t/more-informative-inferred-especially-for-unit-testing/5481/13 "2017-08-24T15:53:07Z")

</div>

I also created a package, since I created the additional macros `@isinferred_noneval` and `@return_types` that I didn’t want to put into the PR, but which might be useful for someone: [https://github.com/afniedermayer/InferenceUtilities.jl](https://github.com/afniedermayer/InferenceUtilities.jl)  
`@isinferred` is also in the package.

The package is quite minimalistic and at an early stage, but it works.