# TTFX with DataFrames and CSV
**URL:** https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432
**Category:** General Usage
**Tags:** question, dataframes, csv, ttfp
**Created:** [January 29, 2022, 8:05pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432 "2022-01-29T20:05:54Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 29, 2022, 8:05pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/1 "2022-01-29T20:05:54Z")
I have a very simple little program:
```julia
@time using CSV, DataFrames
input="""
time,ping
1,25.7
2,31.8
"""
io = IOBuffer(input)
@time df = CSV.read(io, DataFrame)
close(io)
df
```
Output:
```julia
julia> include("bench.jl")
3.062944 seconds (7.52 M allocations: 453.843 MiB, 5.73% gc time, 67.24% compilation time)
16.101461 seconds (49.23 M allocations: 2.039 GiB, 6.23% gc time, 99.98% compilation time)
2×2 DataFrame
Row │ time ping
│ Int64 Float64
─────┼────────────────
1 │ 1 25.7
2 │ 2 31.8
```
So it needs about 19s on my machine (Intel® Core™ i7-7700K on Linux).
Is there a way to reduce this time?
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [January 29, 2022, 9:41pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/2 "2022-01-29T21:41:32Z")
You can certainly help by profiling. Maybe you can narrow the problem down to either CSV.jl or DataFrames.jl separately?
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 29, 2022, 9:47pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/3 "2022-01-29T21:47:30Z")
How can I create a profile? Or where is that documented?
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 29, 2022, 9:52pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/4 "2022-01-29T21:52:54Z")
Still recommended: [The Juno.jl Front-End · Juno Documentation](https://docs.junolab.org/stable/man/juno_frontend/#Profiler)
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 29, 2022, 10:06pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/5 "2022-01-29T22:06:18Z")
Well, I thought @profile is only for profiling runtime performance and not to debug interference and compilation time. Am I wrong?
---
### Author:  [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [January 29, 2022, 10:07pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/6 "2022-01-29T22:07:31Z")
`@profile` can do both (but if you want to measure compiletime you need to run it in a fresh session)
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 29, 2022, 10:20pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/7 "2022-01-29T22:20:06Z")
> [@ufechner7](#):
>
> Well, I thought @profile is only for profiling runtime performance and not to debug interference and compilation time. Am I wrong?
Adding to Oscar’s answer: sometimes you have to be careful not to include compilation in your profile (so you run your code twice: first to compile and then to measure).
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 29, 2022, 10:21pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/8 "2022-01-29T22:21:32Z")
Thanks for your answers. Next issue:
julia\> using ProfileView
Gtk-Message: 23:20:15.044: Failed to load module “unity-gtk-module”
Gtk-Message: 23:20:15.061: Failed to load module “canberra-gtk-module”
Gtk-Message: 23:20:15.062: Failed to load module “canberra-gtk-module”
Any idea?
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 29, 2022, 10:24pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/9 "2022-01-29T22:24:58Z")
> [@ufechner7](#):
>
> julia\> using ProfileView
> Gtk-Message: 23:20:15.044: Failed to load module “unity-gtk-module”
> Gtk-Message: 23:20:15.061: Failed to load module “canberra-gtk-module”
> Gtk-Message: 23:20:15.062: Failed to load module “canberra-gtk-module”
Is this VS Code or Juno? (VS Code profiling still seems to be [lagging](https://github.com/julia-vscode/julia-vscode/issues/1545) behind Juno and that is the reason I’m using both IDEs).
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [January 29, 2022, 10:28pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/10 "2022-01-29T22:28:41Z")
You can install those packages via `apt-get` on linux. Not sure why they aren’t downloaded automatically.
But I mean more basic profiling. Timing `using CSV` and `using DataFrames` separetely. In the `CSV.read` call, separating out `CSV.File` and `DataFrame(...)`.
These are all things you can do to narrow down the problem without a complicated flame graph like ProfileView.jl
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 29, 2022, 10:33pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/11 "2022-01-29T22:33:28Z")
> [@pdeffebach](#):
>
> These are all things you can do to narrow down the problem without a complicated flame graph like ProfileView.jl
OK, and you can debug with `println` (which I do often enough). But IMHO it would be better if adequate tools were available and users educated to use them.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 29, 2022, 10:37pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/12 "2022-01-29T22:37:13Z")
I manged to create a profile view. The “Failed to load module…” messages are only warnings. I am not so much interested in the time for using CSV, DataFrames, because thats reasonable short and probably the package authors try to keep it low anyways.
New code:
```julia
@time using CSV, DataFrames
const input="""
time,ping
1,25.7
2,31.8
"""
function read_csv(in)
io = IOBuffer(in)
df = CSV.read(io, DataFrame)
close(io)
df
end
using ProfileView
@profview read_csv(input)
```
file:///home/ufechner/Bilder/Profile.png
In the left block if I hover I mainly see:
- abstractinterpretation.jl
- typeinfer.jl
In the right section I mainly see:
- file.jl
- CSV.jl
- loading.jl
But this is difficult to interpret for me…
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [January 29, 2022, 10:57pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/13 "2022-01-29T22:57:26Z")
Here is what I see with Atom/Juno using
```julia
@time using CSV, DataFrames
const input="""
time,ping
1,25.7
2,31.8
"""
function read_csv(in)
io = IOBuffer(in)
df = CSV.read(io, DataFrame)
close(io)
df
end
using Profile
@profile read_csv(input)
Juno.profiler()
```
in a fresh session. Output in console:
```julia
3.309279 seconds (7.74 M allocations: 456.073 MiB, 4.35% gc time, 60.60% compilation time)
```
Graphical output:
The dominating function calls are:
```julia
typeinf_ext_toplevel(mi::MethodInstance, world::UInt) = typeinf_ext_toplevel(NativeInterpreter(world), mi)
```
around 20 % of runtime and
```julia
df = CSV.read(io, DataFrame)
```
around 80 % of the runtime.
Edit: @ufechner7 the graph in Atom/Juno is interactive: you can click at a bar and the IDE navigates to the corresponding code.
---
### Author:  [@lawless-m](https://discourse.julialang.org/u/lawless-m)
#### Post date: [January 30, 2022, 10:47am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/14 "2022-01-30T10:47:24Z")
There’s a major TTFX discussion going on atm. There’s plenty of tips and discussion going on there
> [@Taking TTFX seriously: Can we make common packages faster to load and use](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949):
>
> So much work has been done to improve julia compilation and remove invalidations. Things really feel a lot snappier to use. But there is still the problem of packages that were written without much thought to startup time, and TTFX - the time-to-first whatever a package does most, is sometimes really slow. Some common problems I see are (reorganised to be in order of importance): Lack of type stability slowing down compilation. Not calling precompile for common methods, or just running them …
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [January 30, 2022, 10:51am UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/15 "2022-01-30T10:51:37Z")
The OP issue seems mostly related to CSV.jl, not to DataFrames. I don’t use the latter and still experience similar times-to-first-CSV-read with `CSV.read(file, rowtable)` or `columntable`.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 30, 2022, 1:06pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/16 "2022-01-30T13:06:39Z")
I run another test, and I agree with @aplavin : The issue is related to CSV.jl.
```julia
@time using CSV, DataFrames
const input="""
time,ping
1,25.7
2,31.8
"""
function read_csv(in)
io = IOBuffer(in)
@time file = CSV.File(io)
@time df = DataFrame(file)
close(io)
df
end
df = read_csv(input)
```
Output:
```julia
julia> @time include("bench2.jl")
3.071134 seconds (7.53 M allocations: 454.289 MiB, 5.26% gc time, 67.56% compilation time)
11.473243 seconds (2.32 M allocations: 96.563 MiB, 99.98% compilation time)
0.106111 seconds (166.72 k allocations: 9.127 MiB, 99.85% compilation time)
19.250656 seconds (56.72 M allocations: 2.479 GiB, 6.03% gc time, 94.79% compilation time)
2×2 DataFrame
Row │ time ping
│ Int64 Float64
─────┼────────────────
1 │ 1 25.7
2 │ 2 31.8
```
11.5 of the 19.25s total time are related to calling CSV.File .
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 30, 2022, 1:16pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/17 "2022-01-30T13:16:21Z")
Created an issue in CSV.jl: [https://github.com/JuliaData/CSV.jl/issues/974](https://github.com/JuliaData/CSV.jl/issues/974)
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [January 30, 2022, 5:33pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/18 "2022-01-30T17:33:15Z")
This doesn’t seem unusual, AFAIK CSV.File has to be compiled for the specific schema of your data (i.e. the specific types of your columns)
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 30, 2022, 9:57pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/19 "2022-01-30T21:57:55Z")
Well, but why? For small CSV files there should be a code path that avoids compilation of specialized code.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [January 30, 2022, 10:09pm UTC](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/20 "2022-01-30T22:09:50Z")
Equivalent Python code for comparison:
```julia
from io import StringIO
import pandas as pd
input = StringIO("""time,ping
1,25.7
2,31.8
""")
df = pd.read_csv(input)
print(df)
```
Output:
```julia
time python3 bench.py
time ping
0 1 25.7
1 2 31.8
real 0m0.295s
user 0m0.458s
sys 0m0.272s
```
Python: 0.3s
Julia: 19.3s
Doesn’t look good. If Julia would reach 3s I would already be very happy.
[Next page](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432.md?page=2)