I have a very simple little program: @time using CSV, DataFrames input=""" time,ping 1,25.7 2,31.8 """ io = IOBuffer(input) @time df = CSV.read(io, DataFrame) close(io) df Output: julia> include("bench.jl") 3.062944 seconds (7.52 M allocations: 453.843 MiB, 5.73% gc time, 67.24% compilation t…

Well, @btime cannot be used to determine the time-to-first-dataframe. But the solution suggested by @rafael.guerra seams to solve my initial problem. Code: @time using DataFrames, DelimitedFiles const input=""" time,ping 1,25.7 2,31.8 """ const input=""" time,ping 1,25.7 2,31.8 """ function rea…

[image] ufechner7: julia> using ProfileView Gtk-Message: 23:20:15.044: Failed to load module “unity-gtk-module” Gtk-Message: 23:20:15.061: Failed to load module “canberra-gtk-module” Gtk-Message: 23:20:15.062: Failed to load module “canberra-gtk-module” Is this VS Code or Juno? (VS Code pr…

You can install those packages via apt-get on linux. Not sure why they aren’t downloaded automatically. But I mean more basic profiling. Timing using CSV and using DataFrames separetely. In the CSV.read call, separating out CSV.File and DataFrame(...). These are all things you can do to narrow dow…

[image] pdeffebach: These are all things you can do to narrow down the problem without a complicated flame graph like ProfileView.jl OK, and you can debug with println (which I do often enough). But IMHO it would be better if adequate tools were available and users educated to use them.

I manged to create a profile view. The “Failed to load module…” messages are only warnings. I am not so much interested in the time for using CSV, DataFrames, because thats reasonable short and probably the package authors try to keep it low anyways. New code: @time using CSV, DataFrames const in…

Here is what I see with Atom/Juno using @time using CSV, DataFrames const input=""" time,ping 1,25.7 2,31.8 """ function read_csv(in) io = IOBuffer(in) df = CSV.read(io, DataFrame) close(io) df end using Profile @profile read_csv(input) Juno.profiler() in a fresh session. Outpu…

There’s a major TTFX discussion going on atm. There’s plenty of tips and discussion going on there [image] Taking TTFX seriously: Can we make common packages faster to load and use Performance So much work has been done to improve julia compilation and remove invalidation…

The OP issue seems mostly related to CSV.jl, not to DataFrames. I don’t use the latter and still experience similar times-to-first-CSV-read with CSV.read(file, rowtable) or columntable.

I run another test, and I agree with @aplavin : The issue is related to CSV.jl. @time using CSV, DataFrames const input=""" time,ping 1,25.7 2,31.8 """ function read_csv(in) io = IOBuffer(in) @time file = CSV.File(io) @time df = DataFrame(file) close(io) df end df = read_csv(…

Created an issue in CSV.jl: https://github.com/JuliaData/CSV.jl/issues/974

This doesn’t seem unusual, AFAIK CSV.File has to be compiled for the specific schema of your data (i.e. the specific types of your columns)

TTFX with DataFrames and CSV

General Usage

lawless-m January 30, 2022, 10:47am 14

There’s a major TTFX discussion going on atm. There’s plenty of tips and discussion going on there

Topic		Replies	Views
Julia v1.9.0-beta2 is fast Performance	44	6125	January 5, 2023
Using DataFrames: ~ 10 seconds General Usage	24	2357	March 19, 2018
CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R General Usage announcement	68	9521	March 23, 2022
CSV read performance vs Pandas General Usage	29	8423	May 6, 2019
CSV read in is too slow than other language General Usage performance	13	1561	June 21, 2023

TTFX with DataFrames and CSV

Related topics