# Can \`load\` work faster?

**URL:** <https://discourse.julialang.org/t/can-load-work-faster/38291>\
**Category:** Performance\
**Created:** [April 27, 2020, 10:51am UTC](https://discourse.julialang.org/t/can-load-work-faster/38291 "2020-04-27T10:51:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![heliosdrm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heliosdrm/32/3851_2.png) [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)\
**Post date:** [April 27, 2020, 10:51am UTC](https://discourse.julialang.org/t/can-load-work-faster/38291/1 "2020-04-27T10:51:56Z")

</div>

I have to load many files in a loop (to be more specific, I’m working with `OBJ` files to be used with [MeshIO](https://github.com/JuliaIO/MeshIO.jl)). I guess that including a line like `load(filename)` in the loop, with `filename` being a string with the file path, slows down the code because it is not type-stable (a `String`-type argument is not sufficient to tell `load` what type of output will be produced).

Is there a type-stable, smarter way to load those files, knowing that I’m always reading the same type of file? (The output of `load` in my case is always a `Mesh{3, Float32, Triangle}`.)

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 27, 2020, 11:35am UTC](https://discourse.julialang.org/t/can-load-work-faster/38291/2 "2020-04-27T11:35:40Z")

</div>

I suspect it disk access and memory allocation that is slowing you down. But have you checked `@code_warntype` to see if type stability is the issue?

You could do this:

```julia
for filename in files
    data::Mesh{3, Float32, Triangle} = load(filename)
    # Data operations on type 1
end

```

That will allow the compiler to know what data is, and generate the appropriate code in the rest of the loop. If load ever returns something else you will get a exception that the return from load() could not be converted to a Mesh{3,Float32, Triangle}.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 27, 2020, 12:18pm UTC](https://discourse.julialang.org/t/can-load-work-faster/38291/3 "2020-04-27T12:18:56Z")

</div>

I suspect @pixel27 is correct.  
How large are the mesh files?  
One simple way to see if disk access is the bottleneck would be to time a read, then time a second read. The second should read from memory cache.  
And yes - I know this is not really good enough.

A second method might be to create a ramdisk (tmpdisk). Copy the file to that location then time a read.

---

<div class="post-metadata">

**Author:** ![heliosdrm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heliosdrm/32/3851_2.png) [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)\
**Post date:** [April 27, 2020, 1:00pm UTC](https://discourse.julialang.org/t/can-load-work-faster/38291/4 "2020-04-27T13:00:39Z")

</div>

`@code_warntype` confirms that it is not type-stable:

```julia
julia> @code_warntype load("example.obj")
Variables
  #self#::Core.Compiler.Const(FileIO.load, false)
  s::String
  args::Tuple{}

Body::Any
1 ─ %1 = Core.NamedTuple()::Core.Compiler.Const(NamedTuple(), false)
│ %2 = Base.pairs(%1)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│ %3 = Core.tuple(%2, #self#, s)::Core.Compiler.PartialStruct(Tuple{Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},typeof(load),String}, Any[Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false), Core.Compiler.Const(FileIO.load, false), String])
│ %4 = Core._apply(FileIO.:(var"#load#13"), %3, args)::Any
└── return %4 

```

Also tried to declarethe type of the returned variable, as you suggest, although the full parametric type is more convoluted, but I get the exception anyway:

```julia
ERROR: MethodError: Cannot `convert` an object of type GeometryBasics.Mesh{3,Float32,GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}},FaceView{GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}}}} to an object of type GeometryBasics.Mesh{3,Float32,GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}},FaceView{GeometryBasics.Ngon{3,Float32,3,GeometryBasics.PointMeta{3,Float32,GeometryBasics.Point{3,Float32},(:uv, :normals),Tuple{GeometryBasics.Vec{2,Float32},GeometryBasics.Vec{3,Float32}}}}}}

```

Seems that `convert` is just undefined for such type.

After that, I have also been looking into the specific implementation of `load` for mesh files in `MeshIO`, and it seems the type-instability issue is also there:

```julia
function loadmesh1(s)
    f = File{DataFormat{:OBJ}}(s)
    MeshIO.load(f)
end

```

```julia
julia> @code_warntype loadmesh1("example.obj")
Variables
  #self#::Core.Compiler.Const(loadmesh1, false)
  s::String
  f::File{DataFormat{:OBJ}}

Body::GeometryBasics.Mesh
1 ─ %1 = Core.apply_type(Main.DataFormat, :OBJ)::Core.Compiler.Const(DataFormat{:OBJ}, false)
│ %2 = Core.apply_type(Main.File, %1)::Core.Compiler.Const(File{DataFormat{:OBJ}}, false)
│ (f = (%2)(s))
│ %4 = MeshIO.load::Core.Compiler.Const(MeshIO.load, false)
│ %5 = (%4)(f)::GeometryBasics.Mesh
└── return %5

```

(`GeometryBasics.Mesh` is not `Any`, but yet an abstract type, labelled in red.)

Anyway, in this particular case you may be right: my OBJ files are ~10MB, not huge, but still big so that probably disk access is the bottleneck.