Naive use of BSON not working as expected

Specific Domains Data
1

Sorry, I am a beginner to BSON and am a little stuck.

I have a struct Foo defined entirely in some package A. Package B imports A, defines an instance f=A.Foo() and serializes that instance using BSON.

Now if, in the REPL, I do import A; using BSON and attempt to
restore f then there is no problem. But if put identical code into
a new package C (having B as a dependency) then I get UndefVarError: A not defined. Why is that?

2

MWE available?

1 Like
3

package A

Here is the contents of A/src/A.jl:

module A

struct Foo
    x::Int
end

end # module

package C

Here is the contents of C/src/C.jl

module C

import A
using BSON

file = "/Users/anthony/test.bson"
f = BSON.load(file)[:f]
@show f

end # module

serializing and deserialising in the REPL

julia> import A
       using BSON

       f = A.Foo(7)

       file = "/Users/anthony/test.bson"
       bson(file, Dict(:f=>f))        

humus:toml anthony$ julia             # restarting the REPL

julia> import A
       using BSON
       file = "/Users/anthony/test.bson"
       f = BSON.load(file)[:f]
A.Foo(7)                              # working as expected

instead importing C to deserialise:

julia> using C
[ Info: Precompiling C [371b88f8-98b2-11e9-0613-bf108cd8ab40]
ERROR: LoadError: UndefVarError: A not defined
Stacktrace:
 [1] (::getfield(BSON, Symbol("##31#32")))(::Module, ::String) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:20
 [2] mapfoldl_impl(::typeof(identity), ::getfield(BSON, Symbol("##31#32")), ::NamedTuple{(:init,),Tuple{Module}}, ::Array{Any,1}) at ./reduce.jl:45
 [3] #mapfoldl#187(::Base.Iterators.Pairs{Symbol,Module,Tuple{Symbol},NamedTuple{(:init,),Tuple{Module}}}, ::Function, ::Function, ::Function, ::Array{Any,1}) at ./reduce.jl:72
 [4] #mapfoldl at ./none:0 [inlined]
 [5] _mapreduce_dim at ./reducedim.jl:306 [inlined]
 [6] #mapreduce#548 at ./reducedim.jl:304 [inlined]
 [7] #mapreduce at ./none:0 [inlined]
 [8] #reduce#549 at ./reducedim.jl:348 [inlined]
 [9] (::getfield(Base, Symbol("#kw##reduce")))(::NamedTuple{(:init,),Tuple{Module}}, ::typeof(reduce), ::Function, ::Array{Any,1}) at ./none:0
 [10] resolve(::Array{Any,1}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:20
 [11] (::getfield(BSON, Symbol("##35#36")))(::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:48
 [12] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:79
 [13] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:89
 [14] (::getfield(BSON, Symbol("##45#46")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:133
 [15] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:88
 [16] (::getfield(BSON, Symbol("##20#22")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:82
 [17] applychildren!(::getfield(BSON, Symbol("##20#22")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/BSON.jl:21
 [18] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:82
 [19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:89
 [20] raise_recursive at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:99 [inlined]
 [21] load(::String) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:104
 [22] top-level scope at none:0
 [23] include at ./boot.jl:326 [inlined]
 [24] include_relative(::Module, ::String) at ./loading.jl:1038
 [25] include(::Module, ::String) at ./sysimg.jl:29
 [26] top-level scope at none:2
 [27] eval at ./boot.jl:328 [inlined]
 [28] eval(::Expr) at ./client.jl:404
 [29] top-level scope at ./none:3
in expression starting at /Users/anthony/Dropbox/Julia7/MLJ/MLJ/sandbox/toml/C/src/C.jl:7
ERROR: Failed to precompile C [371b88f8-98b2-11e9-0613-bf108cd8ab40] to /Users/anthony/.julia/compiled/v1.1/C/6eWus.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
4

This works for me:

julia>
Julia has exited. Press Enter to start a new session.

Starting Julia...
[ Info: Recompiling stale cache file C:\Users\PK\.julia\compiled\v1.3\Atom\w9XOh.ji for Atom [c52e3926-4ff0-5f6e-af25-54175e0327b1]
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.0-DEV.359 (2019-06-04)
 _/ |\__'_|_|_|\__'_|  |  Commit b54bc8d7e8 (22 days old master)
|__/                   |

julia> include("A.jl")
Main.A

julia> import Main.A

julia> 

julia> using BSON

julia> 

julia> f = A.Foo(7)
Main.A.Foo(7)

julia> 

julia> file = "./test.bson"
"./test.bson"

julia> 

julia> bson(file, Dict(:f=>f))

julia> include("C.jl")
f = Main.A.Foo(7)
Main.C

julia>

No changes other than referring to Main.A.

5

Yes, thank you for looking at this!

Unfortunately, this doesn’t resolve my issue. Yes, I can include C.jl, but it I import the C package I get the error (for reasons I don’t understand). Since I need deserialization from within a package, my problem remains.

6

So what exactly do you do to be able to do import A? You have a package project file?
How does one reproduce your problem?

1 Like
7

Project file for package A (no dependencies):

name = "A"
uuid = "224fedf6-98b2-11e9-382a-43c3b7668a47"
version = "0.1.0"

Project file for package C:

name = "C"
uuid = "371b88f8-98b2-11e9-0613-bf108cd8ab40"
version = "0.1.0"

[deps]
A = "224fedf6-98b2-11e9-382a-43c3b7668a47"
BSON = "fbb218c0-5317-5bc6-957e-2ee96dd4b1f0"
8

So you did ]add A and ]add C?
And then import A ?
EDIT: On second thought, perhaps rather dev?

9

Okay, I think I understand the problem. When you do using C the module A is not known.
It needs to be added to the dependencies of C.

10

A is a dependency of C. See project file above.

And yes, generate the packages with Pkg.generate(“A”) and Pkg.generate(“C”), populate the source files with the code above, activate the project for C and dev BSON and A to make them dependencies of C.

11

Yes, but how do you set up the computation when you start Julia? If you start Julia and immediately do using C, I think A will not be loaded at that point.

12

This also does not work:

julia> using A
julia> using C
[ Info: Precompiling C [371b88f8-98b2-11e9-0613-bf108cd8ab40]
ERROR: LoadError: UndefVarError: A not defined
Stacktrace:
 [1] (::getfield(BSON, Symbol("##31#32")))(::Module, ::String) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:20
 [2] mapfoldl_impl(::typeof(identity), ::getfield(BSON, Symbol("##31#32")), ::NamedTuple{(:init,),Tuple{Module}}, ::Array{Any,1}) at ./reduce.jl:45
 [3] #mapfoldl#187(::Base.Iterators.Pairs{Symbol,Module,Tuple{Symbol},NamedTuple{(:init,),Tuple{Module}}}, ::Function, ::Function, ::Function, ::Array{Any,1}) at ./reduce.jl:72
 [4] #mapfoldl at ./none:0 [inlined]
 [5] _mapreduce_dim at ./reducedim.jl:306 [inlined]
 [6] #mapreduce#548 at ./reducedim.jl:304 [inlined]
 [7] #mapreduce at ./none:0 [inlined]
 [8] #reduce#549 at ./reducedim.jl:348 [inlined]
 [9] (::getfield(Base, Symbol("#kw##reduce")))(::NamedTuple{(:init,),Tuple{Module}}, ::typeof(reduce), ::Function, ::Array{Any,1}) at ./none:0
 [10] resolve(::Array{Any,1}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:20
 [11] (::getfield(BSON, Symbol("##35#36")))(::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:48
 [12] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:79
 [13] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:89
 [14] (::getfield(BSON, Symbol("##45#46")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/extensions.jl:133
 [15] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:88
 [16] (::getfield(BSON, Symbol("##20#22")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:82
 [17] applychildren!(::getfield(BSON, Symbol("##20#22")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/BSON.jl:21
 [18] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:82
 [19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:89
 [20] raise_recursive at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:99 [inlined]
 [21] load(::String) at /Users/anthony/.julia/packages/BSON/XPZLD/src/read.jl:104
 [22] top-level scope at none:0
 [23] include at ./boot.jl:326 [inlined]
 [24] include_relative(::Module, ::String) at ./loading.jl:1038
 [25] include(::Module, ::String) at ./sysimg.jl:29
 [26] top-level scope at none:2
 [27] eval at ./boot.jl:328 [inlined]
 [28] eval(::Expr) at ./client.jl:404
 [29] top-level scope at ./none:3
in expression starting at /Users/anthony/Dropbox/Julia7/MLJ/MLJ/sandbox/toml/C/src/C.jl:7
ERROR: Failed to precompile C [371b88f8-98b2-11e9-0613-bf108cd8ab40] to /Users/anthony/.julia/compiled/v1.1/C/6eWus.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
13

As @ayush-1506 has pointed out to me, the explanation for the behaviour is in BSON/src/extenstions.jl, line 20:

resolve(fs) = reduce((m, f) -> getfield(m, Symbol(f)), fs; init = Main

I think this means that the expressions parsed by the deserialiser are always resolved in the Main namespace, rather than in the namespace of the module calling BSON.load.

This seems like an unnecessary restriction.

14

I believe the reason for this is that the only other easy option is loading within the module that BSON.load executes in, which is always going to be BSON (because that’s where it’s defined). Not saying it can’t resolve elsewhere; someone would just need to thread the appropriate argument down the call stack.

15

I think you are right. It bombs there, in BSON. The Main restriction is weird, but at least that would be readily circumvented in your code.

16

I believe you may have encountered the same problem as described in

https://github.com/MikeInnes/BSON.jl/issues/25