ReadOnlyMemoryError in CSV.read

e_johnson · June 13, 2019, 7:09am

Hello,

I am trying to read in space separated files, but I am getting a ReadOnlyMemoryError when calling CSV.read.

My minimum working example runs fine on my laptop, but it fails on my desktop. I am using Julia 1.1.1 and Ubuntu 16.04.6 on both.

It successfully reads in data with 3 columns and 5000 rows (the pos_mwe.txt file) but fails when reading a 5000x5000 space delimited file (the VwF_mwe.txt file). As far as I can tell the data files are not the problem, which seems to be corroborated by the fact that they run fine on my laptop.

What could be the problem here, and why will it work on my laptop but not desktop? I’m new to Julia but this seems very strange. Thanks!

Here is the example:

using DelimitedFiles
using CSV
using DataFrames

function mwe_memory_error()

pos = Array{Float64}(undef, 5000, 3)
pos = CSV.read(“pos_mwe.txt”, header=false, delim=’ ')
println(“past position file read”)

F = Array{Float64}(undef, 5000, 5000)
F = CSV.read(“VwF_mwe.txt”, header=false, delim=’ ')
println(“past view factor file read”)

end;

And here is the error output:

julia> mwe_memory_error()
past position file read
ERROR: ReadOnlyMemoryError()
Stacktrace:
[1] setindex! at ./array.jl:767 [inlined]
[2] parsevalue!(::Type{Float64}, ::Int8, ::Array{UInt64,1}, ::Int64, ::Array{UInt8,1}, ::Int64, ::Int64, ::Parsers.Options{false,true,false,Missing,UInt8,Nothing}, ::Int64, ::Int64, ::Array{Int8,1}) at /home/odak6/.julia/packages/CSV/2IO2Z/src/CSV.jl:466
[3] parsetape(::Val{false}, ::Int64, ::Dict{Int8,Int8}, ::Array{UInt64,1}, ::Array{UInt8,1}, ::Int64, ::Int64, ::Int64, ::Nothing, ::Array{Int64,1}, ::Float64, ::Array{Dict{String,UInt64},1}, ::Array{UInt64,1}, ::Int64, ::Array{Int8,1}, ::Bool, ::Parsers.Options{false,true,false,Missing,UInt8,Nothing}) at /home/odak6/.julia/packages/CSV/2IO2Z/src/CSV.jl:271
[4] #File#20(::Bool, ::Bool, ::Int64, ::Nothing, ::Int64, ::Int64, ::Bool, ::Nothing, ::Bool, ::Array{String,1}, ::String, ::Char, ::Bool, ::Char, ::Nothing, ::Nothing, ::Char, ::Nothing, ::UInt8, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Dict{Int8,Int8}, ::Bool, ::Float64, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Type, ::String) at /home/odak6/.julia/packages/CSV/2IO2Z/src/CSV.jl:232
[5] (::getfield(Core, Symbol(“#kw#Type”)))(::NamedTuple{(:header, :delim),Tuple{Bool,Char}}, ::Type{CSV.File}, ::String) at ./none:0
[6] #read#63 at /home/odak6/.julia/packages/CSV/2IO2Z/src/CSV.jl:559 [inlined]
[7] #read at ./none:0 [inlined]
[8] mwe_memory_error() at /home/odak6/Evan/PP_VwF/MWE_memory_error/mwe_memory_error.jl:13
[9] top-level scope at none:0

tanhevg · June 13, 2019, 10:13am

Could be a locale issue, with different locales on the laptop and the desktop. For example, different locales can have different decimal separator characters, and this could be upsetting the parser. Can you give an example of what is in the files? Also, the locales, os versions, Julia versions and package versions of the desktop and the laptop.

bernhard · June 13, 2019, 10:32am

I am on my mobile, so maybe things are not displayed correctly.
But shouldn’t you have a closing single apostrophe to define the delimiter (in your code)?
Or is that not necessary?
I always tend to write ‘,’ for a Char.

e_johnson · June 13, 2019, 1:20pm

Thanks for these thoughts.

Locale: Actually, the locale was slightly different on the desktop, but after editing them so are all set to standard english, I get the same ReadOnlyMemoryError - so this did not fix the problem.

OS versions: both are on Ubuntu 16.04.6

Julia versions: both are on 1.1.1

CSV package: both are on v0.5.5

The file that fails to read, VwF_mwe.txt, looks like this, but there are 5000 x 5000 entries.
0.00140000 0.00200000 0.00210000 0.00290000 0.00270000 0.00280000 etc.

A couple other clues/thoughts:

If I open the 5000x5000 file in a text editor and delete many of the rows so it only has 3 rows with 5000 columns each, it does not give the ReadOnlyMemoryError, and it successfully completes. So it seems that it has to do with the fact that the file is large.
If I change the CSV.read line to instead use readdlm, it successfully reads in the full 5000x5000 dataset:
F = readdlm(all_VwF_name, ’ ', Float64, ‘\n’)

The readdlm satisfies me for the moment, but I’m still puzzled by the CSV.read error.

e_johnson · June 13, 2019, 1:22pm

Hi, I don’t follow what you’re saying exactly - maybe it’s a phone display issue?

bernhard · June 13, 2019, 1:24pm

This expression does not seem valid Julia syntax for me. What delimiter does your data have?
If your data is ‘space’ delimited, then shouldn’t your code be

delim=' '

bernhard · June 13, 2019, 1:32pm

Can you share the files?
What happens if you try CSV.File instead of CSV.read

e_johnson · June 13, 2019, 4:04pm

It is space delimited. It probably looks like there is not a space there because I didn’t format it as code (I wasn’t sure how to give the code the proper format so I just pasted it without) But there is definitely a space there which you can see if you select very carefully.

After some searching for how to do the format properly, here is the code with the formatting corrected, and you should more easily see the important space. Sorry for the poor formatting - I’m new but learning fast

function mwe_memory_error()

pos = Array{Float64}(undef, 5000, 3)
pos = CSV.read(“pos_mwe.txt”, header=false, delim=’ ')
println(“past position file read”)

F = Array{Float64}(undef, 5000, 5000)
F = CSV.read(“VwF_mwe.txt”, header=false, delim=’ ')
println(“past view factor file read”)

end;

e_johnson · June 14, 2019, 12:50pm

The text files are in the dropbox folder:
https://www.dropbox.com/sh/bqmvktqvyr5t2zz/AABI3-vl-BOD7KuNiC37DtdAa?dl=0

e_johnson · June 14, 2019, 12:59pm

Ok, so today I am trying this again (didn’t change the code) and the code runs without error!

Yesterday all day it gave the ReadOnlyMemoryError. And actually, before yesterday I’m pretty sure it was running fine. I did now try the CSV.File command, and not surprisingly, it works too.

So it seems to be something intermittent. Does that help point this in a certain direction?

Like I said, I switched to readdlm which seems to be working fine, but for the sake of others facing a similar problem in the future, if anybody has an explanation it could be helpful. Thanks!

bkamins · June 14, 2019, 1:01pm

@quinnj - just letting you know in case you have not seen this thread.

quinnj · June 14, 2019, 9:38pm

Thanks for sharing the files; I too don’t see any issues on OSX, same versions you mentioned. Do let me know if you see any other issues and feel free to open an issue on the CSV.jl repo.

bernhard · July 2, 2019, 11:58am

I certainly have run into this error with CSV.jl in the past, but I do not recall why. I will take a note next time this happens…
In any case, I can read both files just fine.