How to read range of JLD file?

Specific Domains Data
,
1 
julia> using JLD

julia> D=load("D.jld","D",(:,1))
ERROR: too few arguments
 in println at strings/io.jl:25
2

No any idea ?
…

3

Maybe show use how you created the JLD? Otherwise it is a bit hard to guess what you are trying to do.

4

Like below:

           _

_ _ ()_ | A fresh approach to technical computing
() | () () | Documentation: http://docs.julialang.org
_ _ | | __ _ | Type “?help” for help.
| | | | | | |/ ` | |
| | || | | | (| | | Version 0.4.5 (2016-03-18 00:58 UTC)
/ |_‘|||_’_| | Official http://julialang.org/ release
|__/ | x86_64-w64-mingw32

julia> using JLD

julia> D=rand(10,10)
10x10 Array{Float64,2}:
0.0870527 0.436815 0.215768 0.374257 0.477098 0.920663 0.0679468 0.893296 0.656644
… 0.687406 0.00995926 0.96358 0.846045 0.188301 0.722089 0.109776 0.86346 0.725332 0.36931

julia> save(“D.jld”,“D”,D)

julia> D=load(“D.jld”,“D”,(:,1))
ERROR: too few arguments
in println at strings/io.jl:25

julia>

5

Why do you have (:, 1) in there? Does it work without it?

6

Are you trying to do this? load("D.jld","D")[:,1]? ( a common error for people that come from Matlab)

7

load(“D.jld”,“D”)
works
and
load(“D.jld”,“D”)[:,1]
works, but a ned to save memory
Paul

8

load(“D.jld”,“D”)
works

9

Then you might try

save("D.jld", Dict("D$i" => D[:,i] for i in 1:10))
load("D.jld", "D1")
10

Thanks, Idea nice but somethink missing
julia> save(“D.jld”, Dict(“D$i” => D[:,i] for i in 1:10))
ERROR: syntax: missing comma or ) in argument list
Paul

11

It works on my system. If there is an error, I suspect you should be able to find it yourself.

12

One explanation might be if you’re on julia 0.4. You shouldn’t unless you have a very good reason to. The old syntax in julia 0.4 would have been:

save("D.jld", Dict(["D$i" => D[:,i] for i in 1:10]))
13

For completeness, it may be worth mentioning that if memory is an issue, there is a lower level interface that allows memory mapped arrays, which only needs a couple more of lines of code.

Example with a larger array:-

julia> function bigsave()
          save("/tmp/test.jld","bigarray",rand(10000,10000))
       end
bigsave (generic function with 1 method)

julia> function bigload1() # simple interface
          bigarray=load("/tmp/test.jld","bigarray")
          println(sum(bigarray))
       end
bigload1 (generic function with 1 method)

julia> function bigload2() # lower level interface
          f=jldopen("/tmp/test.jld","r",mmaparrays=true)
          bigarray=read(f["bigarray"])
          println(sum(bigarray))
          close(f)
       end
bigload2 (generic function with 1 method)

julia> bigsave()

julia> bigload1()
5.000035653882745e7

julia> @time bigload1()
5.000035653882745e7
  0.190904 seconds (242 allocations: 762.950 MB, 1.12% gc time)

julia> bigload2()
5.000035653882745e7

julia> @time bigload2()
5.000035653882745e7
  0.160253 seconds (189 allocations: 9.828 KB)
3 Likes
14

That is very cool, and actually as fast:

 function bigsave2()
       D = rand(1000,1000)
       save("D.jld", Dict("D$i" => D[:,i] for i in 1:1000))
       end

function bigload3() # lower level interface
                 f=jldopen("/tmp/test.jld","r",mmaparrays=true)
                 bigarray=read(f["bigarray"])[:,10]
                 println(sum(bigarray))
                 close(f)
              end

function bigload4() # lower level interface
                 f=jldopen("D.jld","r",mmaparrays=true)
                 bigarray=read(f["D10"])
                 println(sum(bigarray))
                 close(f)
              end

bigload3(); bigload4();

@time bigload3()
  0.001467 seconds (188 allocations: 89.938 KB)

@time bigload4()
  0.001425 seconds (125 allocations: 16.734 KB)

very nice trick to know.

1 Like
15

lokk nice but something wrong on last step

Dir tmp exist.

D:\pawel\Julia_5.0_otwarty>C:\Users\PC\AppData\Local\Julia-0.5.0\bin\julia.exe
-L start.jl
Pracujesz w aktalogu: D:\pawel\Julia_5.0_otwarty

16

or this , the same but without functions

julia> save(“tmp/test.jld”,“bigarray”,rand(10000,10000))
julia> f=jldopen(“tmp/test.jld”,“r”,mmaparrays=true)
Julia data file version 0.1.1: tmp/test.jld

julia> bigarray=read(f[“bigarray”])
ERROR: could not create file mapping: Odmowa dostępu.
in #mmap#1(::Bool, ::Bool, ::Function, ::IOStream,
::Type{Array{Float64,2}}, ::Tuple{Int64,Int64}, ::UInt64) at .\mmap.
jl:133
in mmap(::IOStream, ::Type{Array{Float64,2}}, ::Tuple{Int64,Int64},
::UInt64) at .\mmap.jl:102
in readmmap(::HDF5.HDF5Dataset, ::Type{Array{Float64,N}}) at
C:\Users\PC.julia\v0.5\HDF5\src\plain.jl:1504
in read_vals(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type{Float64},
::Int32, ::Int32, ::Tuple{Int64,Int64}) at C:\User
s\PC.julia\v0.5\JLD\src\JLD.jl:438
in read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int32, ::Int32,
::Tuple{Int64,Int64}) at C:\Users\PC.julia\v0.5
\JLD\src\JLD.jl:429
in read(::JLD.JldDataset) at C:\Users\PC.julia\v0.5\JLD\src\JLD.jl:392

julia>

Paul,

W dniu 2017-01-16 14:47, Simon Bolland pisze:

17

It would appear that the HDF5 format used by JLD doesn’t support the memory mapping functionality on windows, so I think you’ll have to make do with just the Dict solution.

HDF5 Issue #89

1 Like
18

Big THX Simon,
I never used Dict, please show me place with some info about it, (using
Dict for save big matrix…)
Paul

W dniu 2017-01-17 23:17, Simon Bolland pisze:

19

Look at my suggestion above.

function bigload5()
                        bigarray=load("D.jld","D10")
                        println(sum(bigarray))
                     end

bigload5(); @time bigload5()
0.001044 seconds (264 allocations: 22.906 KB)
20

Thx, is very fine !
Paul