Can `load` work faster?

General Usage Performance
1

I have to load many files in a loop (to be more specific, I’m working with OBJ files to be used with MeshIO). I guess that including a line like load(filename) in the loop, with filename being a string with the file path, slows down the code because it is not type-stable (a String-type argument is not sufficient to tell load what type of output will be produced).

Is there a type-stable, smarter way to load those files, knowing that I’m always reading the same type of file? (The output of load in my case is always a Mesh{3, Float32, Triangle}.)

2

I suspect it disk access and memory allocation that is slowing you down. But have you checked @code_warntype to see if type stability is the issue?

You could do this:

for filename in files
    data::Mesh{3, Float32, Triangle} = load(filename)
    # Data operations on type 1
end

That will allow the compiler to know what data is, and generate the appropriate code in the rest of the loop. If load ever returns something else you will get a exception that the return from load() could not be converted to a Mesh{3,Float32, Triangle}.

3

I suspect @pixel27 is correct.
How large are the mesh files?
One simple way to see if disk access is the bottleneck would be to time a read, then time a second read. The second should read from memory cache.
And yes - I know this is not really good enough.

A second method might be to create a ramdisk (tmpdisk). Copy the file to that location then time a read.

4

@code_warntype confirms that it is not type-stable:

julia> @code_warntype load("example.obj")
Variables
  #self#::Core.Compiler.Const(FileIO.load, false)
  s::String
  args::Tuple{}

Body::Any
1 ─ %1 = Core.NamedTuple()::Core.Compiler.Const(NamedTuple(), false)
│   %2 = Base.pairs(%1)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│   %3 = Core.tuple(%2, #self#, s)::Core.Compiler.PartialStruct(Tuple{Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},typeof(load),String}, Any[Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false), Core.Compiler.Const(FileIO.load, false), String])
│   %4 = Core._apply(FileIO.:(var"#load#13"), %3, args)::Any
└──      return %4

Also tried to declarethe type of the returned variable, as you suggest, although the full parametric type is more convoluted, but I get the exception anyway:

ERROR: MethodError: Cannot `convert` an object of type GeometryBasics.Mesh{3,Float32,GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}},FaceView{GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}}}} to an object of type GeometryBasics.Mesh{3,Float32,GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}},FaceView{GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}}}}

Seems that convert is just undefined for such type.

After that, I have also been looking into the specific implementation of load for mesh files in MeshIO, and it seems the type-instability issue is also there:

function loadmesh1(s)
    f = File{DataFormat{:OBJ}}(s)
    MeshIO.load(f)
end

julia> @code_warntype loadmesh1("example.obj")
Variables
  #self#::Core.Compiler.Const(loadmesh1, false)
  s::String
  f::File{DataFormat{:OBJ}}

Body::GeometryBasics.Mesh
1 ─ %1 = Core.apply_type(Main.DataFormat, :OBJ)::Core.Compiler.Const(DataFormat{:OBJ}, false)
│   %2 = Core.apply_type(Main.File, %1)::Core.Compiler.Const(File{DataFormat{:OBJ}}, false)
│        (f = (%2)(s))
│   %4 = MeshIO.load::Core.Compiler.Const(MeshIO.load, false)
│   %5 = (%4)(f)::GeometryBasics.Mesh
└──      return %5

(GeometryBasics.Mesh is not Any, but yet an abstract type, labelled in red.)

Anyway, in this particular case you may be right: my OBJ files are ~10MB, not huge, but still big so that probably disk access is the bottleneck.