Save BigFloat Array to a file

General Usage
1

Hello together,
I am trying to save a big BigFloat Array to a file (The format does not matter to me).
HDF5 hasn’t any function that would accept BigFloat Datatype, JLD is incredibly slow on saving and loading the array and JLD2 standard functions for some reason fail to load the *.jld2 again throwing the following Error

@JLD2.load (dataPath) data
ERROR: ArgumentError: cannot parse "" as BigFloat
Stacktrace:
 [1] _parse_failure(::Type, ::String, ::Int64, ::Int64) at ./parse.jl:360 (repeats 2 times)
 [2] tryparse_internal at ./parse.jl:356 [inlined]
 [3] parse at ./parse.jl:366 [inlined]
 [4] rconvert at /home/benejo/.julia/packages/JLD2/KjBIK/src/data.jl:809 [inlined]
 [5] jlconvert at /home/benejo/.julia/packages/JLD2/KjBIK/src/data.jl:576 [inlined]
 [6] macro expansion at /home/benejo/.julia/packages/JLD2/KjBIK/src/dataio.jl:70 [inlined]
 [7] macro expansion at ./simdloop.jl:73 [inlined]
 [8] read_array!(::Array{BigFloat,2}, ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{BigFloat,JLD2.CustomSerialization{String,JLD2.Vlen{String}}}) at /home/benejo/.julia/packages/JLD2/KjBIK/src/dataio.jl:68
 [9] read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{BigFloat,JLD2.CustomSerialization{String,JLD2.Vlen{String}}}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/benejo/.julia/packages/JLD2/KjBIK/src/datasets.jl:323
 [10] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{BigFloat,JLD2.CustomSerialization{String,JLD2.Vlen{String}}}, ::Array{JLD2.ReadAttribute,1}) at /home/benejo/.julia/packages/JLD2/KjBIK/src/datasets.jl:173
 [11] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/benejo/.julia/packages/JLD2/KjBIK/src/datasets.jl:149
 [12] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/benejo/.julia/packages/JLD2/KjBIK/src/datasets.jl:92
 [13] getindex(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String) at /home/benejo/.julia/packages/JLD2/KjBIK/src/groups.jl:108
 [14] read(::JLD2.JLDFile{JLD2.MmapIO}, ::String) at /home/benejo/.julia/packages/JLD2/KjBIK/src/JLD2.jl:326
 [15] #293 at /home/benejo/.julia/packages/JLD2/KjBIK/src/loadsave.jl:77 [inlined]
 [16] #jldopen#31(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##293#294")), ::String) at /home/benejo/.julia/packages/JLD2/KjBIK/src/loadsave.jl:4
 [17] jldopen(::Function, ::String) at /home/benejo/.julia/packages/JLD2/KjBIK/src/loadsave.jl:2
 [18] top-level scope at /home/benejo/.julia/packages/JLD2/KjBIK/src/loadsave.jl:76

Has anybody experience with saving BigFloat to a file and could help me here?
Greetings,
GoodNews

2

maybe a csv?

using DelimitedFiles
#creating a BigFloat array, 1D, but the method should work with more dimensions of BigFloats
x1=BigFloat("1.22222222222222222222222222222222222222222222222221")
x2=BigFloat("1.22222222222222222222222222222222222222222222222222")
A = Array{BigFloat,1}(undef,2)
A = [x1,x2]
writedlm("temp.csv", A,BigFloat) #writing a CSV
readdlm("temp.csv", BigFloat)  #Loading a CSV

Greetings,
longemen3000

Edit: it only works with 1D arrays :confused: , but is something

3

serialize works:

julia> X = big.(rand(3,3))
3×3 Array{BigFloat,2}:
 0.555244  0.285951  0.531511
 0.469538  0.609854  0.577057
 0.977457  0.248332  0.0436916

julia> using Serialization

julia> serialize("X", X)

julia> deserialize("X")
3×3 Array{BigFloat,2}:
 0.555244  0.285951  0.531511
 0.469538  0.609854  0.577057
 0.977457  0.248332  0.0436916
5 Likes
5

If I try this,

open("delim_file.csv", "w") do io
    serialize(io, sm)
    writedlm(io, sm)
end

open("delim_file.csv", "r") do io
    deserialize(io)
    readdlm(io, Double64)
end

I get the error from before:

at row 1, column 1 : UndefVarError(:isnothing)
error(::String) at error.jl:33
dlm_fill(::DataType, ::Array{Array{Int64,1},1}, ::Tuple{Int64,Int64}, ::Bool, ::String, ::Bool, ::Char) at DelimitedFiles.jl:514
readdlm_string(::String, ::Char, ::Type, ::Char, ::Bool, ::Dict{Symbol,Union{Char, Integer, Tuple{Integer,Integer}}}) at DelimitedFiles.jl:470
#readdlm_auto#8 at DelimitedFiles.jl:230 [inlined]
readdlm_auto at DelimitedFiles.jl:230 [inlined]
#readdlm#6 at DelimitedFiles.jl:225 [inlined]
readdlm at DelimitedFiles.jl:225 [inlined]
#readdlm#1 at DelimitedFiles.jl:57 [inlined]
readdlm at DelimitedFiles.jl:57 [inlined]
(::getfield(Main, Symbol("##103#104")))(::IOStream) at StreamUtility.jl:289
#open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##103#104")), ::String, ::Vararg{String,N} where N) at iostream.jl:369
open(::Function, ::String, ::String) at iostream.jl:367
top-level scope at none:0

Do I apply (de-)serialize here properly?

6

No, you don’t need writedlm/readdlm if you’re using serialize/deserialize. My example above was writing/reading to a file named X.

1 Like
7

True, that’s perfect. Thanks!