i have to read a 50,000,000 line file and it’s currently taking about 1.5s/million lines, which means it takes on the order of 80s to pull in the whole file (I do have to mention that my fairly non-trivial math on the 50M points takes about 1s )
data format on each line is
i’m using the following code to parse (this is in a loop of course)
line=readline(f) fields = split(line[4:end], ",") x[i] = parse(Float64, fields) y[i] = parse(Float64, fields)
Where x and y are pre-allocated.
Is it possible for me to do this any faster (unfortunately the values are NOT fixed length strings) ?
Thank you !