CSV Reading (rewrite in C?)


#24

That’s spending a fair amount of time updating a non-const global.

julia> @time begin; mystring="1.2"; mysum=0.0; for i=1:100_000_000; global mysum+= parse(Float32, mystring); end; end
 38.471735 seconds (200.02 M allocations: 2.981 GiB, 0.21% gc time)

julia> function f()
           mystring="1.2"
           mysum=0.0
           for i=1:100_000_000
               mysum += parse(Float32, mystring)
           end
           return mysum
       end
       @time f()
  9.861511 seconds (24.90 k allocations: 1.305 MiB)
1.2000000476837158e8

julia> @time f()
  9.834558 seconds (5 allocations: 176 bytes)
1.2000000476837158e8

#25

oh, one of those julia gotchas.

The global was a typed floating point, one storage location. I hope that julia will eventually recognize such situations.

ok, so we are down to a factor of 4 rather than 10. Still, for a fast csv reader, this is a killer.


#26
julia> f(s) = ccall((:atof, "libc"), Float64, (Ptr{UInt8},), pointer(s))
f (generic function with 2 methods)

julia> @btime f("1.23")
  27.176 ns (0 allocations: 0 bytes)
1.23

julia> f_safe(s) = ccall((:atof, "libc"), Float64, (Cstring,), s)
f_safe (generic function with 1 method)

julia> @btime f_safe("1.23")
  34.547 ns (0 allocations: 0 bytes)
1.23

julia> @btime parse(Float64, "1.23")
  31.142 ns (0 allocations: 0 bytes)
1.23

FWIW.


#27

Benchmarking _ in _ global _ scope _ is _ probably _ the _ most _ common _ mistake _ that _ Julia _ newbies _ make, _ and _ is _ the _ topic _ of _ countless _ posts. _ But _ I _ would _ think _ you _ already _ knew _ that _ considering _ THIS. :slight_smile:


#28

Use traceur to avoid such common performance issues.


#29

But _ I _ would _ think _ you _ already _ knew _ that _ considering _ THIS

yes I did know this. unfortunately, the emphasis was on the did and not on the know. when one gets old, memory has already been used up and is no longer freed up by the garbage collector.

heck, at least I do remember always to type my variables and functions, because this avoids a set of other gotchas.


#30

yes, for real code, traceur would save my behind.

but can I ask not just for the mechanics, but also for the why? is this a temporary problem, or something that is intrinsically unavoidable in Julia?

I mean, the variable is global and typed. The code is compiled and run, and when this happens, it can see that the variable exists and is typed. I understand why type variability can cause extra code and dispatch time, but why here?


#31

I am wondering: what is actually causing the slowing down of C-native 100m atof’s vs. julia 100m parses then? is it the C-Julia barrier, which is also crossed 100m times?

even after my global variable mishap (which, I apologize, sidetracked everyone), it still does not seem feasible to get a native julia csv decoder to the speed of fread, because the necessary string-float conversions already exceed the budget.


#32

In the CRSP daily data, the ret variable sometimes contains characters like “C”, which will force the whole column into string. You then need to convert it back to numeric, and replace these special observation with missing value. I think such task has been handled by most existing packages. If you write your own parser, a small change in the data structure may require a rewrite of code. Are you running the code in cloud?


#33

No it’s not typed.

No that’s undecidable.


#34

No, pointer. This is undefined. f_safe is also not safer than f (other than the wrong use of pointer there), it just checks for embedding NUL, which isn’t really a safety issue. The safety issue, i.e. if the string is NUL terminated, is guaranteed in both cases.


#35

I think I see. although the compiler could easily figure out the type in my example, in more general cases, another function could have the side effect of changing a global variable. this is not the case with variables defined in a stack frame.

in some sense, I would need the ability to lock a global variable to retain a certain type. and julia 1.0.0 does not yet have it.


#36

Do note that CSV.jl uses the native-Julia float parser in the Parsers.jl package, which is different from what’s provided in Base Julia (Base just calls the C function strtod). There’s a known performance issue when parsing floats at full precision, but if the float is rounded to even one less significant digit, the performance is comparable with C. A few of us have ideas on how to fix this, it’s just a matter of deciding what would be easiest/best for the fix.

Again, it would be most helpful if there are specific files that can be shared, even if just privately, to compare the performance of CSV.jl vs. other parsers.


#37

Hi, thx for the great work in CSV.jl, it’s very nice to have DataFrame interaction. What work flow do you recommend with large CSV file? Can we dump it to other format at the moment? or can we partially |> to DataFrame?


#38

Would it make sense to generate a big test data CSV (why not other formats as well) with all the known bad data features? Or maybe a package that generates the data file(s) locally for testing.


#39

Cf https://github.com/JuliaImages/TestImages.jl for image test data.


#40

As this question is orthogonal to the original, it may be worth starting a separate thread. When there are multiple conversations happening in a single these it can make it hard to follow, and make it harder for future reference to find.


#41

This had been discussed on slack before. I think it would be useful, but it’s not always practical since a lot of files have proprietary data and it’s not always obvious which feature of the table is causing issues.

But this file in particular seems straightforward, and others may be as well. Couldn’t hurt I’d say.


#42

BioJulia has a repo like this for all the weird file formats used in bioinformatics https://github.com/BioJulia/BioFmtSpecimens


#43

Rather than storing big CSV files for reference data, it might work better to have julia code that generates CSV files with the desired characteristics. That would also make it easier to ramp the file size as desired.