# TTFX with DataFrames and CSV
**URL:** https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432
**Category:** General Usage
**Tags:** question, dataframes, csv, ttfp
**Created:** [January 29, 2022, 8:05pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432 "2022-01-29T20:05:54Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 30, 2022, 10:18pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/21 "2022-01-30T22:18:46Z")
This is a known issue with CSV.jl, still - it is good to keep track of it explicitly in the issue so thank you for opening it.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [January 30, 2022, 10:25pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/22 "2022-01-30T22:25:08Z")
> [@ufechner7](#):
>
> Is there a way to reduce this time?
Use DelimitedFiles
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 30, 2022, 10:31pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/23 "2022-01-30T22:31:35Z")
And how would I parse them into a data frame?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 30, 2022, 10:44pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/24 "2022-01-30T22:44:49Z")
```julia
julia> data, header = readdlm(IOBuffer(input), ',', header=true)
([1.0 25.7; 2.0 31.8], AbstractString["time" "ping"])
julia> DataFrame(data, vec(header))
2×2 DataFrame
Row │ time ping
│ Float64 Float64
─────┼──────────────────
1 │ 1.0 25.7
2 │ 2.0 31.8
```
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 30, 2022, 10:54pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/25 "2022-01-30T22:54:59Z")
and something I will blog post next week is:
```julia
julia> input="""
time,ping,name
1,25.7,a
2,,b
"""
"time,ping,name\n1,25.7,a\n2,,b\n"
julia> data, header = readdlm(IOBuffer(input), ',', header=true)
(Any[1 25.7 "a"; 2 "" "b"], AbstractString["time" "ping" "name"])
julia> identity.(DataFrame(ifelse.(data .== "", missing, data), vec(header)))
2×3 DataFrame
Row │ time ping name
│ Int64 Float64? SubStrin…
─────┼─────────────────────────────
1 │ 1 25.7 a
2 │ 2 missing b
```
(I have just realized how far we can go with plain `DelimitedFiles` and one line of code)
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [January 30, 2022, 11:47pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/26 "2022-01-30T23:47:16Z")
For OP’s input example, `@btime` of your code on a fresh **first run** shows `readlm()` taking ~0.12 s and `DataFrame()` taking ~0.25 s.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 31, 2022, 7:18am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/27 "2022-01-31T07:18:51Z")
Well, @btime cannot be used to determine the time-to-first-dataframe.
But the solution suggested by @rafael.guerra seams to solve my initial problem. Code:
```julia
@time using DataFrames, DelimitedFiles
const input="""
time,ping
1,25.7
2,31.8
"""
const input="""
time,ping
1,25.7
2,31.8
"""
function read_csv(inp)
@time data, header = readdlm(IOBuffer(inp), ',',header=true)
# @time df = DataFrame(data, vec(header))
@time df = identity.(DataFrame(ifelse.(data .== "", missing, data), vec(header)))
@time df[!,:time] = convert.(Int64,df[:,:time])
df
end
df = read_csv(input)
```
Output:
```julia
julia> @time include("bench5.jl")
0.837795 seconds (1.92 M allocations: 132.477 MiB, 4.66% gc time, 0.51% compilation time)
0.114726 seconds (90.08 k allocations: 4.882 MiB, 99.82% compilation time)
0.594612 seconds (1.70 M allocations: 93.770 MiB, 9.75% gc time, 99.64% compilation time)
0.080426 seconds (201.72 k allocations: 11.051 MiB, 99.58% compilation time)
2.109000 seconds (5.50 M allocations: 327.066 MiB, 8.52% gc time, 60.14% compilation time)
2×2 DataFrame
Row │ time ping
│ Int64 Float64
─────┼────────────────
1 │ 1 25.7
2 │ 2 31.8
```
Summary:
**Time-to-first-dataframe**
```julia
Python (Pandas): 0.3s
DelimitedFiles: 2.1s
CSV: 19.0s
```
DelimitedFiles does not support two features by default:
a. detecting different column types
b. detecting missing values
The code above handles this correctly for this toy example.
Open questions:
- is it possible to make CSV faster to avoid the need of two different solutions depending on the size of the problem?
- if CSV.jl cannot made fast, would it be good to have a package CSVlight.jl that has the same interfaces as CSV.jl and can serve as drop-in replacement?
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [January 31, 2022, 7:19am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/28 "2022-01-31T07:19:56Z")
> [@ufechner7](#):
>
> Well, @time cannot be used to determine the first-time-to-dataframe.
Why is that?
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 31, 2022, 7:55am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/29 "2022-01-31T07:55:43Z")
Sorry, that was a typo. I mean, @btime cannot be used, because it is explicitly written not to take the compilation overhead into account, but here I am mainly interested in the compilation and inference time, not in the runtime which is neglectable for small datasets.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [January 31, 2022, 11:28am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/30 "2022-01-31T11:28:45Z")
Just played around with this. Essentially the problem is huge functions with type instability.
Breaking up the functions so that unnecessary parts are not compiled is a _really_ easy win. Every `if` block with a bunch of code in it should just call multiple inner functions. Sections with known types should be separated out from larger functions so that only the method for known types has to be compiled, instead of everything being boxed.
Then we could even precompile a bunch of these for specific types.
Moving the ctx.threaded block out of the conditional to another function shaved 4s of the time for me, 16s to 12s. Things like that can be done pretty much everywhere in file.jl, and that’s where nearly all the time is spent.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [January 31, 2022, 11:57am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/31 "2022-01-31T11:57:25Z")
[https://github.com/JuliaData/CSV.jl/pull/975](https://github.com/JuliaData/CSV.jl/pull/975)
I’m stuck in the paris airport a few more hours, see how far I can get with this
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [January 31, 2022, 12:29pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/32 "2022-01-31T12:29:55Z")
Rooting for further delays on your inbound plane!
---
### Author:  [@lrnv](https://discourse.julialang.org/u/lrnv)
#### Post date: [January 31, 2022, 2:13pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/33 "2022-01-31T14:13:13Z")
I hope `plane(inport,outport)` has compilation issues and a looong TTFX.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [January 31, 2022, 7:53pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/34 "2022-01-31T19:53:50Z")
Didn’t get much further.
A problem with doing this is there is so much noise in timing changes that it’s hard to be sure of small improvements. We need a `@ctime` macro that will run code in a fresh session mulltiple times and take an average, although it’s going to take a long time to run.
It also seems like reorganising code is only effective for really large blocks.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 31, 2022, 9:35pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/35 "2022-01-31T21:35:49Z")
Well, I think laptops also change their speed depending on the CPU temperature more then desktops… Easier to do the benchmarking on a desktop.
---
### Author:  [@jules](https://discourse.julialang.org/u/jules)
#### Post date: [January 31, 2022, 10:27pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/36 "2022-01-31T22:27:23Z")
I was trying something like that with [https://github.com/jkrumbiegel/VersionBenchmarks.jl/](https://github.com/jkrumbiegel/VersionBenchmarks.jl/) to compare code across different versions or commits, with different Julia versions if that’s desired. For my test case of improving GridLayoutBase.jl latency, it has still been pretty noisy so far, however. More between trial variation than I would have liked. Maybe one needs to collect 10 runs or more so the average is meaningful.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [February 1, 2022, 8:47am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/37 "2022-02-01T08:47:54Z")
Interestingly, fixing type stability and reorganising things didn’t do that much for the timing. But _after_ doing that, adding some precompile methods had a large affect (where they had none previously) - I’m getting full TTFX including `using` of 8 seconds, and 7 if we remove `@refargs` macros. There are a few more patches of instability preventing further precompilation, but hopefully they are fixable and it can mostly precompile away.
---
### Author:  [@Raf](https://discourse.julialang.org/u/Raf)
#### Post date: [February 5, 2022, 1:56am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/38 "2022-02-05T01:56:48Z")
Finally, to round out this saga, nearly all of the time ends up being resolved by precompilaiton in Parsers.jl, making most of the other changes I made much less effective.
[https://github.com/JuliaData/Parsers.jl/pull/108](https://github.com/JuliaData/Parsers.jl/pull/108)
This is also the case for JSON.jl, and Blink.jl from the other TTFX thread. Its nearly all Parsers.jl.
[https://github.com/JuliaIO/JSON.jl/pull/337](https://github.com/JuliaIO/JSON.jl/pull/337)
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [February 5, 2022, 4:57am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/39 "2022-02-05T04:57:53Z")
Great job! Hopefully your PR gets merged soon! 😀
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [February 5, 2022, 5:56am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/40 "2022-02-05T05:56:09Z")
Wow, that’s really impressive! Turns out that lots of ttfx time improvements don’t even require any compiler optimizations.
[Previous page](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432.md?page=1)
[Next page](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432.md?page=3)