Why is this type unstable?

yvikhlya
Thu, 21 Sep 2023 21:01:51 +0000
General Usage

using Glob, NCDatasets

const colpath="path_to_data"

ncload(ds, ::Type{Array{T,N}}) where {T,N} = ds["T"][:,:,:,:]::Array{T,N}

function foo()
    files=glob("*.??????.nc4",colpath)

    NCDataset(files[1]) do ds
        ncv=ds["T"]
        nctype=eltype(ncv)
        ncdims=size(ncv)
        @show nctype ncdims
        ncload(ds,Array{nctype,length(ncdims)})
    end
end

@code_warntype foo()
MethodInstance for foo()
  from foo() @ Main ~/tmp/julia/foo.jl:7
Arguments
  #self#::Core.Const(foo)
Locals
  #10::var"#10#11"
  files::Vector{String}
Body::Array
1 ─      (files = Main.glob("*.??????.nc4", Main.colpath))
│        (#10 = %new(Main.:(var"#10#11")))
│   %3 = #10::Core.Const(var"#10#11"())
│   %4 = Base.getindex(files, 1)::String
│   %5 = Main.NCDataset(%3, %4)::Array
└──      return %5

It knows that return value is Array, but does not know its dimensionality and element type, although by the time ncload is called, both nctype and ncdims are already defined. If I call ncload as

ncload(ds,Array{Union{Missing,Float32},4})

then @code_warntype shows that everything is type stable

@code_warntype foo()
MethodInstance for foo()
  from foo() @ Main ~/tmp/julia/foo.jl:7
Arguments
  #self#::Core.Const(foo)
Locals
  #12::var"#12#13"
  files::Vector{String}
Body::Array{Union{Missing, Float32}, 4}
1 ─      (files = Main.glob("*.??????.nc4", Main.colpath))
│        (#12 = %new(Main.:(var"#12#13")))
│   %3 = #12::Core.Const(var"#12#13"())
│   %4 = Base.getindex(files, 1)::String
│   %5 = Main.NCDataset(%3, %4)::Array{Union{Missing, Float32}, 4}
└──      return %5

I want to be able to get the type and size of data from a dataset and pass them to ncload function and get a type stable result, which is efficiently allocated as a single chunk in memory. What I am doing wrong?

12 posts - 5 participants

Read full topic