# BSON error when loading Flux model

**URL:** <https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085>\
**Category:** Machine Learning\
**Tags:** flux, bson\
**Created:** [March 18, 2022, 1:27pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085 "2022-03-18T13:27:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![luciano-drozda](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luciano-drozda/32/33789_2.png) [@luciano-drozda](https://discourse.julialang.org/u/luciano-drozda)\
**Post date:** [March 18, 2022, 1:27pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/1 "2022-03-18T13:27:18Z")

</div>

The command

```julia
m = BSON.load("model.bson", @ __MODULE__ )[:m]

```

is throwing

```julia
ERROR: MethodError: Cannot `convert` an object of type CuContext to an object of type CuPtr{Nothing}
Closest candidates are:
  convert(::Type{CuPtr{T}}, ::CUDA.Mem.UnifiedBuffer) where T at /scratch/drozda/.julia/packages/CUDA/9T5Sq/lib/cudadrv/memory.jl:234
  convert(::Type{CuPtr{T}}, ::CUDA.Mem.HostBuffer) where T at /scratch/drozda/.julia/packages/CUDA/9T5Sq/lib/cudadrv/memory.jl:129
  convert(::Type{CuPtr{T}}, ::CUDA.Mem.DeviceBuffer) where T at /scratch/drozda/.julia/packages/CUDA/9T5Sq/lib/cudadrv/memory.jl:59
  ...

```

Does anyone know what could it be ?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 18, 2022, 2:37pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/2 "2022-03-18T14:37:15Z")

</div>

Does the machine this is running on have a CUDA-supported GPU? Otherwise the note in [Saving & Loading · Flux](https://fluxml.ai/Flux.jl/stable/saving/#Saving-and-Loading-Models) may apply.

---

<div class="post-metadata">

**Author:** ![luciano-drozda](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luciano-drozda/32/33789_2.png) [@luciano-drozda](https://discourse.julialang.org/u/luciano-drozda)\
**Post date:** [March 18, 2022, 2:46pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/3 "2022-03-18T14:46:01Z")

</div>

Thanks for the reply @ToucheSir.  
I do offload the model back to the CPU before saving

```julia
m = model |> cpu
@save "model.bson" m opt

```

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 18, 2022, 3:10pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/4 "2022-03-18T15:10:45Z")

</div>

Ah, but `opt` is not offloaded. Can you get away with not saving the optimizer state as well?

---

<div class="post-metadata">

**Author:** ![luciano-drozda](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luciano-drozda/32/33789_2.png) [@luciano-drozda](https://discourse.julialang.org/u/luciano-drozda)\
**Post date:** [March 18, 2022, 7:14pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/5 "2022-03-18T19:14:56Z")

</div>

From what I’ve seen in Flux model zoo, you don’t need to offload the optimizer to any device. Instantiating `opt = ADAM()`, for instance, will work both on CPU and GPU.

What’s strange to me is that the reported issue appears just for models saved some time ago.

If I train a new model, save it (with the optimizer) and load it for inference, the issue isn’t raised.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 18, 2022, 8:14pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/6 "2022-03-18T20:14:50Z")

</div>

Yes, but if you offload the model itself and load it back in, the references in the optimizer will no longer point to the same thing. So no error is raised, but all the optimizer state has been silently invalidated. This is a big footgun and something we haven’t been able to address until very recently, since it’s a fundamental issue with the design of the current optimizer interface.

> [@luciano-drozda](#):
>
> What’s strange to me is that the reported issue appears just for models saved some time ago.
> 
> If I train a new model, save it (with the optimizer) and load it for inference, the issue isn’t raised.

That’s interesting, I assume you didn’t offload those older models before saving? If so, I wonder if internal changes in CUDA.jl might be causing the errors then.

---

<div class="post-metadata">

**Author:** ![luciano-drozda](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luciano-drozda/32/33789_2.png) [@luciano-drozda](https://discourse.julialang.org/u/luciano-drozda)\
**Post date:** [March 19, 2022, 8:45am UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/7 "2022-03-19T08:45:54Z")

</div>

I do offload these older models back to the CPU before saving.

Could changing modules over time lead to such an issue ?  
I’m using `@ __MODULE__ ` option when loading,  
but if the current version of a module differs from the older one,  
issues could appear, I suppose.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 19, 2022, 4:53pm UTC](https://discourse.julialang.org/t/bson-error-when-loading-flux-model/78085/8 "2022-03-19T16:53:42Z")

</div>

Did you save anything other than the model? Optimizer state could be a problem as mentioned above.

If you’re willing/able to provide one of these troublesome BSON files, I could take a look. Another idea would be reading them back in with [https://github.com/ancapdev/LightBSON.jl](https://github.com/ancapdev/LightBSON.jl) and seeing trying to recover the data manually. Even printing out a tree of all the type tags in the file could help to identify what is holding onto CUDA stuff.