Hi,
I just come to face the problem reported here with only one “small” table of 62 Mo (4001×4933). The very strange thing is that I did this test on 3 computers :
1- Linux Manjaro with 8 Go RAM
2- Linux Manjaro with 64 Go RAM
3- A supercomputer node with 192 Go RAM (CentOS if I remember).
In all 3 computers I have the last Julia (1.4.1) with the packages up to date. And the very strange result is that I cannot read the small table on all computers except the small one (see results below - I cannot share the file it contains data, it is a dataframe of floats (mainly 0.0) with the first row, the first column containing strings and the last column of Integers )
I did not try the solution reported here because my goal is to use the software on a supercompter without root access.
Disabling mmap does not solve anything.
I gave a try to CSVFiles, unfortunately the memory it takes is much more important than CSV (more than 2X) and my small computer start to swap and I have to kill Julia
I think it is a serious problem because my table is not so large and in a computer without root access there is currently no solution. The problem is difficult to observe because it is not directly related to the size of the table, and can be observed in some computers and not others, even if the package versions are the same…
Here are the results of my tests. I hope it will help.
On the computer Linux Manjaro with 64 Go RAM :
(@v1.4) pkg> status
Status `~/.julia/environments/v1.4/Project.toml`
[336ed68f] CSV v0.6.2
[861a8166] Combinatorics v1.0.1
[a93c6f00] DataFrames v0.21.0
julia> x = CSV.read(file; delim ="\t", header=true, use_mmap=false)
ERROR: TaskFailedException:
SystemError: memory mapping failed: Ne peut allouer de la mémoire
Stacktrace:
[1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
[2] #systemerror#50 at ./error.jl:167 [inlined]
[3] systemerror at ./error.jl:167 [inlined]
[4] mmap(::Mmap.Anonymous, ::Type{Array{UInt64,1}}, ::Tuple{Int64}, ::Int64; grow::Bool, shared::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:209
[5] #mmap#14 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:251 [inlined]
[6] mmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:251 [inlined]
[7] parsetape(::Val{false}, ::Int64, ::Dict{Int8,Int8}, ::Array{Array{UInt64,1},1}, ::Array{Array{UInt64,1},1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Int64, ::Array{Int64,1}, ::Float64, ::Array{Dict{String,UInt64},1}, ::Array{UInt64,1}, ::Int64, ::Array{Int8,1}, ::Array{Int64,1}, ::Bool, ::Parsers.Options{false,false,true,false,Missing,UInt8,Nothing}, ::Nothing) at /home/fred/.julia/packages/CSV/vyG0T/src/file.jl:474
[8] macro expansion at /home/fred/.julia/packages/CSV/vyG0T/src/file.jl:313 [inlined]
[9] (::CSV.var"#39#42"{Array{Int8,1},Array{UInt8,1},Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Nothing,Float64,Int64,Dict{Int8,Int8},Int64,Bool,Int64,Array{Int64,1},Int64,Array{Int64,1},Array{Array{Array{UInt64,1},1},1},Array{Array{Array{UInt64,1},1},1},Array{Array{Dict{String,UInt64},1},1},Array{Array{UInt64,1},1},Array{Array{Int8,1},1},Array{Array{Int64,1},1},Int64})() at ./threadingconstructs.jl:126
...and 20 more exception(s).
x = CSV.read(file; delim ="\t", header=true)
ERROR: TaskFailedException:
SystemError: memory mapping failed: Ne peut allouer de la mémoire
Stacktrace:
[1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
[2] #systemerror#50 at ./error.jl:167 [inlined]
[3] systemerror at ./error.jl:167 [inlined]
[4] mmap(::Mmap.Anonymous, ::Type{Array{UInt64,1}}, ::Tuple{Int64}, ::Int64; grow::Bool, shared::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:209
[5] #mmap#14 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:251 [inlined]
[6] mmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:251 [inlined]
[7] parsetape(::Val{false}, ::Int64, ::Dict{Int8,Int8}, ::Array{Array{UInt64,1},1}, ::Array{Array{UInt64,1},1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Int64, ::Array{Int64,1}, ::Float64, ::Array{Dict{String,UInt64},1}, ::Array{UInt64,1}, ::Int64, ::Array{Int8,1}, ::Array{Int64,1}, ::Bool, ::Parsers.Options{false,false,true,false,Missing,UInt8,Nothing}, ::Nothing) at /home/fred/.julia/packages/CSV/vyG0T/src/file.jl:469
[8] macro expansion at /home/fred/.julia/packages/CSV/vyG0T/src/file.jl:313 [inlined]
[9] (::CSV.var"#39#42"{Array{Int8,1},Array{UInt8,1},Parsers.Options{false,false,true,false,Missing,UInt8,Nothing},Nothing,Float64,Int64,Dict{Int8,Int8},Int64,Bool,Int64,Array{Int64,1},Int64,Array{Int64,1},Array{Array{Array{UInt64,1},1},1},Array{Array{Array{UInt64,1},1},1},Array{Array{Dict{String,UInt64},1},1},Array{Array{UInt64,1},1},Array{Array{Int8,1},1},Array{Array{Int64,1},1},Int64})() at ./threadingconstructs.jl:126
...and 22 more exception(s).
julia> x = CSV.read(file; delim ="\t")
ERROR: LLVM ERROR: out of memory
signal (6): Abandon
in expression starting at REPL[3]:0
LLVM ERROR: out of memory
signal (6): Abandon
in expression starting at REPL[3]:0
gsignal at /usr/lib/libc.so.6 (unknown line)
abort at /usr/lib/libc.so.6 (unknown line)
LLVM ERROR: out of memory
On the computer Linux Manjaro with only 8 Go RAM :
(@v1.4) pkg> status
Status `~/.julia/environments/v1.4/Project.toml`
[336ed68f] CSV v0.6.2
[5d742f6a] CSVFiles v0.16.0
[861a8166] Combinatorics v1.0.1
[a93c6f00] DataFrames v0.21.0
[31c24e10] Distributions v0.22.6
[09f84164] HypothesisTests v0.10.0
[b1bec4e5] LIBSVM v0.4.0
[2fda8390] LsqFit v0.10.0
[5fb14364] OhMyREPL v0.5.5
[91a5bcdd] Plots v1.2.2
[1a8c2f83] Query v0.12.2
[ce6b1742] RDatasets v0.6.1
[f2b01f46] Roots v1.0.1
[b8865327] UnicodePlots v1.2.0
julia> x = CSV.read(file; delim ="\t")
4001×4933 DataFrame. Omitted printing of 4922 columns