# CSV Reading (rewrite in C?)

**URL:** <https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537>\
**Category:** Internals & Design\
**Created:** [September 27, 2018, 1:39am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537 "2018-09-27T01:39:59Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 9:02pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/21 "2018-09-27T21:02:40Z")

</div>

David Sander’s question: [quote]First question is why don’t you just write it in Julia, [/quote]

> [@iwelch](#):
>
> a cat of the file to /dev/null from cache takes under a second. a conversion of 89 million strings to floats takes about 2-4 seconds. so about 25-50 seconds to convert all 12 fields. (a ‘wc’ takes about 15 seconds.)

some more numbers.

Reading my cd.csv file into a Julia string buffer takes about 3 seconds. Although this is much slower than C, it is a one-time operation and thus acceptable for many uses.

More problematic:

A `parse.(Float32, stringvector)` of 100 million strings takes about 20-25 seconds on my imac. A `@time begin; mysum=0.0; for i=1:n; global mysum+= parse(Float32, sv[1]); end; end` takes about 50 seconds.

in contrast, the C version is about 2-5 seconds. I am assuming that this is also why R’s fread is fast.

I do not know why Julia is so slow at the string-\>Float conversion, but it limits the efficiency of native julia code. just guessing, this is the bottleneck for CSV.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 27, 2018, 9:21pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/22 "2018-09-27T21:21:36Z")

</div>

Please post code when posting benchmarks if possible. Julia calls into C for parsing e.g. Float32s.

Trying on master I get

```julia
julia> strs = string.(1:100*10^6);

julia> @time parse.(Float32, strs);;
 10.913384 seconds (9 allocations: 381.470 MiB)

```

(Master, and 1.0.1 does have [improve performance for parsing Floats by KristofferC · Pull Request #27764 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/27764) which might explain why you get slower results).

> [@iwelch](#):
>
> A `@time begin; mysum=0.0; for i=1:n; global mysum+= parse(Float32, sv[1]); end; end` takes about 50 seconds.

For performance, don’t use `globals`. Fixing that this take (of course) just as long time as just parsing the strings.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 9:33pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/23 "2018-09-27T21:33:07Z")

</div>

I have no code for C to Julia. I only experimented with pure C and with pure Julia. Both code was trivial:

```julia
julia> @time begin; mystring="1.2"; mysum=0.0; for i=1:100_000_000; global mysum+= parse(Float32, mystring); end; end
 27.745808 seconds (200.02 M allocations: 2.981 GiB, 0.92% gc time)

```

25-50 seconds.

and quick-and-dirty C code:

```julia
#include <stdio.h>
#include <stdlib.h>

int main() {
  char s[100];
  int n=100000000;

  float mysum=0.0; sprintf(s, "%.f", 1.2);
  for (int i=0; i<n; ++i) {
    s[0]= '0'+i%10; ## make sure we are not optimizing it away
    mysum+= atof(s);
  }
  printf("%f\n", mysum);
}

```

2 seconds.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 27, 2018, 9:42pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/24 "2018-09-27T21:42:18Z")

</div>

That’s spending a fair amount of time updating a non-const global.

```julia
julia> @time begin; mystring="1.2"; mysum=0.0; for i=1:100_000_000; global mysum+= parse(Float32, mystring); end; end
 38.471735 seconds (200.02 M allocations: 2.981 GiB, 0.21% gc time)

julia> function f()
           mystring="1.2"
           mysum=0.0
           for i=1:100_000_000
               mysum += parse(Float32, mystring)
           end
           return mysum
       end
       @time f()
  9.861511 seconds (24.90 k allocations: 1.305 MiB)
1.2000000476837158e8

julia> @time f()
  9.834558 seconds (5 allocations: 176 bytes)
1.2000000476837158e8

```

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 10:06pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/25 "2018-09-27T22:06:12Z")

</div>

oh, one of those julia gotchas.

The global was a typed floating point, one storage location. I hope that julia will eventually recognize such situations.

ok, so we are down to a factor of 4 rather than 10. Still, for a fast csv reader, this is a killer.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 27, 2018, 10:25pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/26 "2018-09-27T22:25:16Z")

</div>

```julia
julia> f(s) = ccall((:atof, "libc"), Float64, (Ptr{UInt8},), pointer(s))
f (generic function with 2 methods)

julia> @btime f("1.23")
  27.176 ns (0 allocations: 0 bytes)
1.23

julia> f_safe(s) = ccall((:atof, "libc"), Float64, (Cstring,), s)
f_safe (generic function with 1 method)

julia> @btime f_safe("1.23")
  34.547 ns (0 allocations: 0 bytes)
1.23

julia> @btime parse(Float64, "1.23")
  31.142 ns (0 allocations: 0 bytes)
1.23

```

FWIW.

---

<div class="post-metadata">

**Author:** ![NiclasMattsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/niclasmattsson/32/21988_2.png) [@NiclasMattsson](https://discourse.julialang.org/u/NiclasMattsson)\
**Post date:** [September 27, 2018, 11:29pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/27 "2018-09-27T23:29:58Z")

</div>

> [@iwelch](#):
>
> oh, one of those julia gotchas.

[Benchmarking](https://discourse.julialang.org/t/with-missings-julia-is-slower-than-r/11838/2) \_ [in](https://discourse.julialang.org/t/why-is-this-code-so-slow-in-julia-compared-to-a-numpy-implementation/6660/4) \_ [global](https://discourse.julialang.org/t/sparse-matrix-vector-product-much-more-slow-than-matlab/7840/2) \_ [scope](https://discourse.julialang.org/t/3-different-ways-to-create-an-array-have-different-performance/9636/2) \_ [is](https://discourse.julialang.org/t/benchmark-matlab-julia-for-matrix-operations/2000/2) \_ [probably](https://discourse.julialang.org/t/vector-of-matrices-vs-multidimensional-arrays/9602/3) \_ [the](https://discourse.julialang.org/t/help-understanding-vectorization-or-lack-thereof/11518/2) \_ [most](https://discourse.julialang.org/t/generators-speed/3715/2) \_ [common](https://discourse.julialang.org/t/function-with-initialized-data/3908/2) \_ [mistake](https://discourse.julialang.org/t/linear-solver-a-b-performance-vs-matlab-a-b/2082/23) \_ [that](https://discourse.julialang.org/t/element-wise-vector-multiplication-and-fusing-dot/1228/2) \_ [Julia](https://discourse.julialang.org/t/anonymous-function-as-argument-cant-be-optimized/11814/2) \_ [newbies](https://discourse.julialang.org/t/performance-of-array-broadcasting/4724/5) \_ [make](https://discourse.julialang.org/t/spurious-allocation/3751/2), \_ [and](https://discourse.julialang.org/t/convert-tuple-to-array/2147/9) \_ [is](https://discourse.julialang.org/t/simple-parallel-examples-for-embarrassingly-simple-problems/8788/9) \_ [the](https://discourse.julialang.org/t/massive-performance-penalty-for-float16-compared-to-float32/6864/3) \_ [topic](https://discourse.julialang.org/t/performance-drop-for-explicit-indexing-in-fused-loops/4459/5) \_ [of](https://discourse.julialang.org/t/performance-difference-between-two-code/1557/2) \_ [countless](https://discourse.julialang.org/t/to-avoid-allocations-avoid-anonymous-functions/9458/4) \_ [posts](https://discourse.julialang.org/t/fusing-not-faster/8696/2). \_ [But](https://discourse.julialang.org/t/why-is-this-code-so-slow-in-julia-compared-to-a-numpy-implementation/6660/6) \_ [I](https://discourse.julialang.org/t/slow-matrix-multiplication-in-julia-compared-to-python-numpy/11015/4) \_ [would](https://discourse.julialang.org/t/types-and-gradients-including-forward-gradient/946/19) \_ [think](https://discourse.julialang.org/t/parallel-is-very-slow/9443/2) \_ [**you**](https://discourse.julialang.org/t/view-and-slices-comparison-of-speed/8647/5) \_ [already](https://discourse.julialang.org/t/improve-the-performance-of-multiplication-of-an-arbitrary-number-of-matrices/10835/2) \_ [knew](https://discourse.julialang.org/t/how-to-assign-real-imaginary-part-of-complex-array/10818/11) \_ [that](https://discourse.julialang.org/t/a-generator-with-two-for-keywords-is-slow/7407/3) \_ [considering](https://discourse.julialang.org/t/wip-faster-string-sort/7671/16) \_ [**THIS**](https://discourse.julialang.org/t/simple-for-loops-and-speed/7158/8). 🙂

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [September 27, 2018, 11:42pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/28 "2018-09-27T23:42:07Z")

</div>

Use traceur to avoid such common performance issues.

> **[GitHub - JunoLab/Traceur.jl](https://github.com/JunoLab/Traceur.jl)**
>
> Contribute to JunoLab/Traceur.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 11:46pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/29 "2018-09-27T23:46:14Z")

</div>

[But](https://discourse.julialang.org/t/why-is-this-code-so-slow-in-julia-compared-to-a-numpy-implementation/6660/6) \_ [I](https://discourse.julialang.org/t/slow-matrix-multiplication-in-julia-compared-to-python-numpy/11015/4) \_ [would](https://discourse.julialang.org/t/types-and-gradients-including-forward-gradient/946/19) \_ [think](https://discourse.julialang.org/t/parallel-is-very-slow/9443/2) \_ [**you**](https://discourse.julialang.org/t/view-and-slices-comparison-of-speed/8647/5) \_ [already](https://discourse.julialang.org/t/improve-the-performance-of-multiplication-of-an-arbitrary-number-of-matrices/10835/2) \_ [knew](https://discourse.julialang.org/t/how-to-assign-real-imaginary-part-of-complex-array/10818/11) \_ [that](https://discourse.julialang.org/t/a-generator-with-two-for-keywords-is-slow/7407/3) \_ [considering](https://discourse.julialang.org/t/wip-faster-string-sort/7671/16) \_ [**THIS**](https://discourse.julialang.org/t/simple-for-loops-and-speed/7158/8)

yes I did know this. unfortunately, the emphasis was on the _did_ and not on the _know_. when one gets old, memory has already been used up and is no longer freed up by the garbage collector.

heck, at least I do remember always to type my variables and functions, because this avoids a set of other gotchas.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 11:50pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/30 "2018-09-27T23:50:20Z")

</div>

yes, for real code, traceur would save my behind.

but can I ask not just for the mechanics, but also for the why? is this a temporary problem, or something that is intrinsically unavoidable in Julia?

I mean, the variable is global and typed. The code is compiled and run, and when this happens, it can see that the variable exists and is typed. I understand why type variability can cause extra code and dispatch time, but why here?

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 27, 2018, 11:55pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/31 "2018-09-27T23:55:19Z")

</div>

> [@kristoffer.carlsson](#):
>
> FWIW.

I am wondering: what is actually causing the slowing down of C-native 100m atof’s vs. julia 100m parses then? is it the C-Julia barrier, which is also crossed 100m times?

even after my global variable mishap (which, I apologize, sidetracked everyone), it still does not seem feasible to get a native julia csv decoder to the speed of fread, because the necessary string-float conversions already exceed the budget.

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [September 27, 2018, 11:57pm UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/32 "2018-09-27T23:57:22Z")

</div>

In the CRSP daily data, the ret variable sometimes contains characters like “C”, which will force the whole column into string. You then need to convert it back to numeric, and replace these special observation with missing value. I think such task has been handled by most existing packages. If you write your own parser, a small change in the data structure may require a rewrite of code. Are you running the code in cloud?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [September 28, 2018, 12:03am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/33 "2018-09-28T00:03:30Z")

</div>

> [@iwelch](#):
>
> I mean, the variable is global and typed.

No it’s not typed.

> [@iwelch](#):
>
> it can see that the variable exists and is typed

No that’s undecidable.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [September 28, 2018, 12:16am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/34 "2018-09-28T00:16:27Z")

</div>

> [@kristoffer.carlsson](#):
>
> `ccall((:atof, "libc"), Float64, (Ptr{UInt8},), pointer(s))`

No, `pointer`. This is undefined. `f_safe` is also not safer than `f` (other than the wrong use of `pointer` there), it just checks for embedding `NUL`, which isn’t really a safety issue. The safety issue, i.e. if the string is NUL terminated, is guaranteed in both cases.

---

<div class="post-metadata">

**Author:** ![iwelch](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Post date:** [September 28, 2018, 5:04am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/35 "2018-09-28T05:04:32Z")

</div>

I think I see. although the compiler could easily figure out the type in my example, in more general cases, another function could have the side effect of changing a global variable. this is not the case with variables defined in a stack frame.

in some sense, I would need the ability to lock a global variable to retain a certain type. and julia 1.0.0 does not yet have it.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 28, 2018, 6:16am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/36 "2018-09-28T06:16:06Z")

</div>

Do note that CSV.jl uses the native-Julia float parser in the Parsers.jl package, which is different from what’s provided in Base Julia (Base just calls the C function `strtod`). There’s a [known performance issue](https://github.com/JuliaData/Parsers.jl/issues/5) when parsing floats at full precision, but if the float is rounded to even one less significant digit, the performance is comparable with C. A few of us have ideas on how to fix this, it’s just a matter of deciding what would be easiest/best for the fix.

Again, it would be most helpful if there are specific files that can be shared, even if just privately, to compare the performance of CSV.jl vs. other parsers.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 28, 2018, 8:34am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/37 "2018-09-28T08:34:42Z")

</div>

Hi, thx for the great work in CSV.jl, it’s very nice to have DataFrame interaction. What work flow do you recommend with large CSV file? Can we dump it to other format at the moment? or can we partially `|>` to DataFrame?

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [September 28, 2018, 9:00am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/38 "2018-09-28T09:00:41Z")

</div>

Would it make sense to generate a big test data CSV (why not other formats as well) with all the known bad data features? Or maybe a package that generates the data file(s) locally for testing.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [September 28, 2018, 9:53am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/39 "2018-09-28T09:53:48Z")

</div>

Cf [GitHub - JuliaImages/TestImages.jl: commonly used test images](https://github.com/JuliaImages/TestImages.jl) for image test data.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [September 28, 2018, 10:53am UTC](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537/40 "2018-09-28T10:53:34Z")

</div>

As this question is orthogonal to the original, it may be worth starting a separate thread. When there are multiple conversations happening in a single these it can make it hard to follow, and make it harder for future reference to find.

[Previous page](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537.md?page=1)

[Next page](https://discourse.julialang.org/t/csv-reading-rewrite-in-c/15537.md?page=3)