Problems reading data with JLD (edited title)

New to Julia
1

I have a working code that consists of lots of separate .jl files.

I have a file “Code.jl” that includes them all in the form

using Package1, Package2

include("Function1.jl")
include("Function2.jl")

I type include("Code.jl") in the REPL, and everything is available to me, and works fine.

Now I’m trying to make a module, and hopefully later a package, by changing “Code.jl” to

module Code
export Function1
using Package1, Package2

include("Function1.jl")
include("Function2.jl")
end

After restarting julia, I type include("Code.jl") and `using .Code’ in the REPL. Now when I call Function1, it starts to run, but I get a splurge of errors that seem to be related to the packages that I am trying to use and it crashes.

I’ve tried to write a simpler example than my actual code that reproduces this error, but I can’t. I’m happy to post the actual code and error, but I wanted to check whether I was doing something fundamentally misguided before doing that.

Thanks,

JB

2

Did you add those two packages to the Project.toml of Code (via Pkg.add(…) after activating the „Code“ environment)

If you would use Pkg.generate(„Code“) or maybe even PkgSkeleton.jl that might help you to create a proper environment.

1 Like
3

Sorry, but I don’t understand. I’m starting by trying to make a module, not a package. I have not done the things that you mention.

4

Post the actual error. Without it, it is hard to say what is going on (for me at least).

1 Like
5

You may have used using in the repl. Then running include("Code.jl") would “know” about packages that are referenced in the code. Now if you make it into a module, the code inside does not have access to the same usings as before. You have to add them manually.

6

I will, but am on my mobile atm.

Not sure what you mean by ‘may have’. I’ve done the things I said after typing exit() and starting a new session so there are no stray 'using’s about.

But how do I ‘add them manually’? I want everything self-contained in the module. None of the individual files have any ‘using’ statements in them

7

File "f.jl":

f(x, y) = mean([x, y])

Then do

julia> include("f.jl");

julia> f(1, 3)                                                       
ERROR: UndefVarError: mean not defined                               
Stacktrace:                                                          
 [1] f(::Int64, ::Int64) at .\f.jl:1                                                          
 [2] top-level scope at REPL[2]:1                                    
                                                                     
julia> using Statistics                                              
                                                                     
julia> f(1, 3)
2.0

One must pull in Statistics so that the function f can do its job.

So now that works, let’s try creating a module as

module M

include("f.jl")

end

Now we include "M.jl":

julia> include("M.jl");       

julia> Main.M.f(1, 3)                                                            
ERROR: UndefVarError: mean not defined                                           
Stacktrace:                                                                      
 [1] f(::Int64, ::Int64) at .\f.jl:1 
 [2] top-level scope at REPL[7]:1                                                
                                                                                 
julia>

Notice that again mean is not available. We have to say explicitly using Statistics in "M.jl".

module M

using Statistics
include("f.jl")

end

Now it works

julia> include("M.jl");
WARNING: replacing module M.                                                     

julia> Main.M.f(1, 3)
2.0
8

If you look at what I posted, that’s exactly what I’ve done. I will post the code and error when I’m back on my laptop.

9

Sure. You haven’t posted the error, so I am guessing one package must have been missed when creating the module.

10

OK. Here’s the module code (not all the individual files ofc)


export Solution,
    Params, Target, Path, Beamparams, initialise, etch, outplot, makegcode

using Dierckx,
    CSV,
    Roots,
    QuadGK,
    Polynomials,
    SparseArrays,
    OSQP,
    Printf,
    Statistics,
    AbstractPlotting.MakieLayout,
    AbstractPlotting,
    GLMakie,
    Colors,
    ColorSchemes,
    FileIO,
    JLD,
    Interpolations

include("Params.jl")
include("Beamparams.jl")
include("Target.jl")
include("Path.jl")
include("Solution.jl")
include("initialise.jl")
include("makeinterpolant.jl")
include("calibration.jl")
include("makerange.jl")
include("findtracks.jl")
include("rasterpaths.jl")
include("joinpaths.jl")
include("Dmap.jl")
include("invDmap.jl")
include("makeinfluencematrix.jl")
include("makesmoothingmatrix.jl")
include("refinetarget.jl")
include("inversesolve.jl")
include("outplot.jl")
include("GSsearch.jl")
include("optimize_Zrange.jl")
include("compresspath.jl")
include("makemaskpath.jl")
include("find_dtheta.jl")
include("etch.jl")
include("findgradient.jl")
include("improve_path.jl")
include("optimise_paths.jl")
include("optimiseoverlap.jl")
include("makegcode.jl")

end

Here’s the complete REPL history when I try to use it

julia> include("Etching.jl");

julia> using .Etching

julia> sol, sol0 = etch("coin",25.0,2; Zrange = 0.5);
target = coin, size = 25.0.
┌ Warning: type Interpolations.FilledExtrapolation{Core.Float64,3,Interpolations.GriddedInterpolation{Core.Float64,3,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Base.StepRangeLen{Core.Float64,Base.TwicePrecision{Core.Float64},Base.TwicePrecision{Core.Float64}},Core.Array{Core.Float64,1},Core.Array{Core.Float64,1}}},Interpolations.Gridded{Interpolations.Linear},Core.Float64} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.GriddedInterpolation{Core.Float64,3,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Base.StepRangeLen{Core.Float64,Base.TwicePrecision{Core.Float64},Base.TwicePrecision{Core.Float64}},Core.Array{Core.Float64,1},Core.Array{Core.Float64,1}}} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.Gridded{Interpolations.Linear} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.Linear not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
ERROR: MethodError: Cannot `convert` an object of type JLD.var"##Interpolations.FilledExtrapolation{Core.Float64,3,Interpolations.GriddedInterpolation{Core.Float64,3,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Base.StepRangeLen{Core.Float64,Base.TwicePrecision{Core.Float64},Base.TwicePrecision{Core.Float64}},Core.Array{Core.Float64,1},Core.Array{Core.Float64,1}}},Interpolations.Gridded{Interpolations.Linear},Core.Float64}#325" to an object of type Interpolations.FilledExtrapolation{Float64,3,Interpolations.GriddedInterpolation{Float64,3,Float64,Interpolations.Gridded{Interpolations.Linear},Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},Array{Float64,1}}},Interpolations.Gridded{Interpolations.Linear},Float64}
Closest candidates are:
  convert(::Type{T}, ::T) where T<:AbstractArray at abstractarray.jl:14
  convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray at C:\Users\pmzjb1\AppData\Local\JuliaPro-1.4.2-1\Julia-1.4.2\share\julia\stdlib\v1.4\LinearAlgebra\src\factorization.jl:55
  convert(::Type{S}, ::T) where {S, T<:(Union{CategoricalArrays.CategoricalString{R}, CategoricalArrays.CategoricalValue{T,R} where T} where R)} at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\CategoricalArrays\dmrjI\src\value.jl:103
  ...
Stacktrace:
 [1] macro expansion at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:422 [inlined]
 [2] jlconvert(::Type{Beamparams}, ::JLD.JldFile, ::Ptr{UInt8}) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:551
 [3] read_scalar_default(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type{T} where T) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:400
 [4] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type{T} where T) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:396
 [5] read(::JLD.JldDataset) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:367
 [6] read(::JLD.JldFile, ::String) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:343
 [7] #49 at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:1258 [inlined]
 [8] map at .\tuple.jl:158 [inlined]
 [9] #48 at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:1258 [inlined]
 [10] jldopen(::JLD.var"#48#50"{Tuple{String,String}}, ::String, ::Vararg{String,N} where N; kws::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:243
 [11] jldopen(::Function, ::String, ::String) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:241
 [12] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::Tuple{String,String}) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:1257
 [13] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String, ::String) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\JLD.jl:1255
 [14] load(::String, ::String, ::Vararg{String,N} where N; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\ZknoK\src\loadsave.jl:118
 [15] load at C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\FileIO\ZknoK\src\loadsave.jl:118 [inlined]
 [16] etch(::String, ::Float64, ::Int64; Zmin::Float64, Zmax::Float64, Zrange::Float64, maskfiles::String, gcodeflag::Bool, dirname::String, plotflag::Bool, saveflag::Bool, verbose::Bool, straightflag::Bool, overlap::Float64, gamma::Float64, dsmin::Float64, dsmax::Float64, dthetamax::Float64, order::Int64, vmin::Float64, vmax::Float64, pixellength::Float64, linsys_solver::String, eps_abs::Float64, eps_rel::Float64) at C:\Users\pmzjb1\Google Drive\LaserJet Julia Project\etch.jl:76
 [17] top-level scope at none:0

It looks like something to do with the packages JLD and Interpolations to me, but I don’t really know what.

Just to emphasise, when I don’t try to make this a module, and just have the ‘using’ statement and the list of 'include’s, it works fine, so I don’t see how I can be missing something in the ‘using’ list.

11

It looks like your reading from HDF5 (JLD)? Are you sure all types have been defined before you read?

12

Yes, because everything works until I make it a module.

13

Well, my thinking is the only difference seems to be that the module has its own environment.
Seems something is not defined in that environment because the way I read the error it seems there was a failure constructing some data structure.

14

Since you are not loading your module as a package, you need to add the directory to your module to LOAD_PATH.

push!(LOAD_PATH,  "path_to_my_module")
using my_module
15

OK. How about this: you wrote the JLD file from the command line, but you are
reading it when you have a module. Those types may be different then. (BeamParams, in particular.)

1 Like
16

The jld file isn’t written from the command line. There’s an initialise function that writes it, so I don’t think it’s that. I have some ideas, but I’m probably going to have to wait until tomorrow to work on it as it’s Father’s Day here in the UK and I’m expecting to be treated like a king by my family… although that’s not guaranteed!

17

Happy father’s day!

If Julia knows about Interpolations in one setting and not another, could the problem here be a confusion between a package and a module? A package, as you probably know, has its own environment which you have to explicitly set up, including adding all the dependencies yourself.

There’s a very good video on the mechanics of setting up a package and adding all the dependencies.

Graham

1 Like
18

I’m not sure. The only place a ‘using Interpolations’ statement occurs is in that module.

I’ll definitely watch the video. I’m hoping to be able to turn it into a package without having to come back to this forum.

19

I told my long-suffering family that what I most want to do for Father’s Day is get this bloody Module sorted out.

Here’s the minimal code to reproduce the problem.

module Testmodule

using JLD, Interpolations

export Function1, Function2

function Function1()
    f = LinearInterpolation(vec(collect(1.0:5.0)), rand(Float64,5))
    save("test.jld","f",f)
end

function Function2(x::Float64)
    f = load("test.jld","f")
    println(f(x))
end
end

And now in the REPL, Function1 saves the interpolant and Function2 tries to load and use it.

julia> include("Testmodule.jl");

julia> using .Testmodule

julia> Function1()

julia> Function2(0.5)
┌ Warning: type Interpolations.Extrapolation{Core.Float64,1,Interpolations.GriddedInterpolation{Core.Float64,1,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Core.Array{Core.Float64,1}}},Interpolations.Gridded{Interpolations.Linear},Interpolations.Throw{Core.Nothing}} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.Throw{Core.Nothing} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.GriddedInterpolation{Core.Float64,1,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Core.Array{Core.Float64,1}}} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.Gridded{Interpolations.Linear} not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
┌ Warning: type Interpolations.Linear not present in workspace; reconstructing
└ @ JLD C:\Users\pmzjb1\.juliapro\JuliaPro_v1.4.2-1\packages\JLD\jeGJb\src\jld_types.jl:697
ERROR: MethodError: objects of type JLD.var"##Interpolations.Extrapolation{Core.Float64,1,Interpolations.GriddedInterpolation{Core.Float64,1,Core.Float64,Interpolations.Gridded{Interpolations.Linear},Core.Tuple{Core.Array{Core.Float64,1}}},Interpolations.Gridded{Interpolations.Linear},Interpolations.Throw{Core.Nothing}}#253" are not callable
Stacktrace:
 [1] Function2(::Float64) at C:\Users\pmzjb1\Google Drive\LaserJet Julia Project\Testmodule.jl:14
 [2] top-level scope at none:0

Is this a problem with JLD and/or Interpolations that I should be posting elsewhere? I have no idea what’s going on.

20

This is probably unsatisfying, but have you tried using Interpolations at the REPL before calling Function2?
Otherwise Interpolations won’t “be defined in the workspace”.