Help resolving Dict access type inference

jackf · January 9, 2019, 9:28pm

I have some code that starts with a loosely type Dict (e.g. something returned from a web service). I want to write a function that will extract a certain key from it, convert that key into a concretely-typed array (Array{Int64, 1}), and return it. It is known (by me) that the particular key actually does have this type, and is a valid key. Whilst I don’t mind if the internals of this function are a bit dynamic / could raise an exception, I would like the return type of this function to be correctly inferred as Array{Int64, 1}. I can’t find a way to do that: the return type is inferred as Any:

julia> foo(x) = Array{Int64}(x["asd"])
foo (generic function with 1 method)

julia> a = Dict{String, Any}("asd" => Any[1,2,3])
Dict{String,Any} with 1 entry:
  "asd" => Any[1, 2, 3]

julia> typeof(a["asd"])
Array{Any,1}

julia> @code_warntype foo(a)
Body::Any
1 1 ─ %1  = invoke Base.ht_keyindex(_2::Dict{String,Any}, "asd"::String)::Int64                                                                         │╻  getindex
  │   %2  = (Base.slt_int)(%1, 0)::Bool                                                                                                                 ││╻  <
  └──       goto #3 if not %2                                                                                                                           ││
  2 ─ %4  = %new(Base.KeyError, "asd")::KeyError                                                                                                        ││╻  Type
  │         (Base.throw)(%4)                                                                                                                            ││
  └──       $(Expr(:unreachable))                                                                                                                       ││
  3 ─ %7  = (Base.getfield)(x, :vals)::Array{Any,1}                                                                                                     ││╻  getproperty
  │   %8  = (Base.arrayref)(false, %7, %1)::Any                                                                                                         ││╻  getindex
  └──       goto #5                                                                                                                                     ││
  4 ─       $(Expr(:unreachable))                                                                                                                       ││
  5 ┄ %11 = (Array{Int64,N} where N)(%8)::Any                                                                                                           │
  └──       return %11                                                                                                                                  │

Could someone please help me work out what branches / thing / corner case I’m missing here that will help the compiler infer the desired return type?

EDIT: perhaps an even simpler example:

julia> a = Dict{String, Any}("asd" => Int64)
Dict{String,Any} with 1 entry:
  "asd" => Int64

julia> foo(x) = Int64(x["asd"])
foo (generic function with 1 method)

julia> @code_warntype foo(a)
Body::Any
1 1 ─ %1  = invoke Base.ht_keyindex(_2::Dict{String,Any}, "asd"::String)::Int64                                                                         │╻  getindex
  │   %2  = (Base.slt_int)(%1, 0)::Bool                                                                                                                 ││╻  <
  └──       goto #3 if not %2                                                                                                                           ││
  2 ─ %4  = %new(Base.KeyError, "asd")::KeyError                                                                                                        ││╻  Type
  │         (Base.throw)(%4)                                                                                                                            ││
  └──       $(Expr(:unreachable))                                                                                                                       ││
  3 ─ %7  = (Base.getfield)(x, :vals)::Array{Any,1}                                                                                                     ││╻  getproperty
  │   %8  = (Base.arrayref)(false, %7, %1)::Any                                                                                                         ││╻  getindex
  └──       goto #5                                                                                                                                     ││
  4 ─       $(Expr(:unreachable))                                                                                                                       ││
  5 ┄ %11 = (Main.Int64)(%8)::Any                                                                                                                       │
  └──       return %11                                                                                                                                  │

ffevotte · January 9, 2019, 9:50pm

This might be what you want:

foo(x) = x["asd"] :: Array{Int64,1}

julia> foo(a)
3-element Array{Int64,1}:
 1
 2
 3

julia> @code_warntype foo(a)
Body::Array{Int64,1}
1 1 ─ %1  = invoke Base.ht_keyindex(_2::Dict{String,Any}, "asd"::String)::Int64   │╻  getindex
  │   %2  = (Base.slt_int)(%1, 0)::Bool                                           ││╻  <
  └──       goto #3 if not %2                                                     ││ 
  2 ─ %4  = %new(Base.KeyError, "asd")::KeyError                                  ││╻  Type
  │         (Base.throw)(%4)                                                      ││ 
  └──       $(Expr(:unreachable))                                                 ││ 
  3 ─ %7  = (Base.getfield)(x, :vals)::Array{Any,1}                               ││╻  getproperty
  │   %8  = (Base.arrayref)(false, %7, %1)::Any                                   ││╻  getindex
  └──       goto #5                                                               ││ 
  4 ─       $(Expr(:unreachable))                                                 ││ 
  5 ┄       (Core.typeassert)(%8, Array{Int64,1})                                 │  
  │   %12 = π (%8, Array{Int64,1})                                                │  
  └──       return %12                                                            │

IIUC, a::T is equivalent to typeassert(a, T), which checks (at run-time) that the type of a is T, as anticipated. This allows the compiler to assume that the expression is of the correct type.

jackf · January 9, 2019, 9:52pm

Thanks, that indeed looks correct! What is getting called when I do Array{Float64}(x)? Because if I call Array{Float64}(["string", "asdasd"]), it will error at runtime, so there must be some check happening.

jackf · January 9, 2019, 9:57pm

Actually, with the example I posted your solution doesn’t work:

julia> a = Dict{String, Any}("asd" => Any[1,2,3])
Dict{String,Any} with 1 entry:
  "asd" => Any[1, 2, 3]

julia> foo(x) = x["asd"] :: Array{Int64,1}

julia> foo(a)
ERROR: TypeError: in foo, in typeassert, expected Array{Int64,1}, got Array{Any,1}
Stacktrace:
 [1] foo(::Dict{String,Any}) at ./none:1
 [2] top-level scope at none:0

ffevotte · January 9, 2019, 9:58pm

I’m no specialist and might not understand all the details of this mechanism. But as I understand it, the specificity of typeassert is that the compiler is aware of it, so that it can extract some knowledge from a typeassert expression, and then base its optimization on that knowledge.

Array{Float64}(x) probably also does some runtime checks, but maybe the compiler does not know about it, so that it can not tell that anything produced by this function will necessarily be of type Array{Float64}.

ffevotte · January 9, 2019, 10:02pm

What about this? Array{Float64}() is used to actually convert the array, then ::Array{Float64} instructs the compiler about the type of the result:

foo(x) = Array{Int64,1}(x["asd"]) :: Array{Int64,1}

julia> a = Dict{String, Any}("asd" => Any[1,2,3])
Dict{String,Any} with 1 entry:
  "asd" => Any[1, 2, 3]

julia> typeof(a["asd"])
Array{Any,1}

julia> foo(a)
3-element Array{Int64,1}:
 1
 2
 3

julia> @code_warntype foo(a)
Body::Array{Int64,1}
1 1 ─ %1  = invoke Base.ht_keyindex(_2::Dict{String,Any}, "asd"::String)::Int64 
  │   %2  = (Base.slt_int)(%1, 0)::Bool                           
  └──       goto #3 if not %2                                           
  2 ─ %4  = %new(Base.KeyError, "asd")::KeyError      
  │         (Base.throw)(%4)  
  └──       $(Expr(:unreachable))
  3 ─ %7  = (Base.getfield)(x, :vals)::Array{Any,1}
  │   %8  = (Base.arrayref)(false, %7, %1)::Any    
  └──       goto #5                                                 
  4 ─       $(Expr(:unreachable))                            
  5 ┄ %11 = (Array{Int64,1})(%8)::Any                  
  │         (Core.typeassert)(%11, Array{Int64,1})   
  │   %13 = π (%11, Array{Int64,1})                      
  └──       return %13

jackf · January 9, 2019, 10:05pm

Thanks, that looks like it will work - but is annoyingly redundant. Will use it for now but if anyone can think of a less verbose way, I’d like to hear it!

yuyichao · January 9, 2019, 10:34pm

The correct return type is Array{Any, 1} as you saw in typeof(a["asd"]). If the correct return type is actually what you want and if you are sure from other sources that it is for sure of that type then a type assertion is exactly what you want.

yuyichao · January 9, 2019, 10:38pm

Now if what you want isn’t what you said in the title, but rather to do a convertion, then

what you need. For return value, this is also equivalent to having foo(x)::Array{Int64,1}, which does the convertion and the assertion. For local variable, you can use a typed local variable, i.e. y::Array{Int64,1} = x["asd"], which also does the convertion and assertion.

The reason that a assertion is required is that the convert function has no requirement for returning the correct type, you can define a convertion rule that doesn’t do that. (You shouldn’t but if you do, type inference will be fine with it, other people probably won’t). OTOH, there’s no way for the user to modify the behavior of type assertion so the compiler can assume it’s behavior easily.

sairus7 · January 10, 2019, 4:41am

I suppose, the goal is to have actual return value types for Dict{String, Any}.
To do that, you don’t have to convert to Any at the dictionary initialization or fill:

a = Dict{String, Any}("asd" => [1,2,3])
a["qwe"] = [1.0,2.0,3.0]

Then you have the right concrete types:

typeof(a["asd"])
>> Array{Int64,1}

typeof(a["qwe"])
>> Array{Float64,1}