Very Slow `DBInterface.execute` on large duckdb relative to R

floswald
Wed, 30 Oct 2024 08:26:18 +0000
Performance

Hi all,

I’ve made a clone of @grantmcdermott 's nice workshop on duckdb and polars - adding a chapter on julia and duckdb. He had pointed out some very slow julia timings in his slides, and actually created an issue on DBInterface.jl. We don’t have a MWE but we have a WE - this example only makes sense on the big (8.6Gb) dataset we are using. There are straightforward instructions for fast data download contained within the workshop website here. But first the result.

R

from here:

tic = Sys.time()
dat1 = dbGetQuery(
  con,
  "
  FROM 'nyc-taxi/**/*.parquet'
  SELECT
    passenger_count,
    AVG(tip_amount) AS mean_tip
  GROUP BY ALL
  ORDER BY ALL
  "
)
toc = Sys.time()
toc - tic

Time difference of 2.233316 secs

julia

from here:

time1 = @elapsed dat1 = DBInterface.execute(
    con,
    """
    FROM 'nyc-taxi/**/*.parquet'
    SELECT
        passenger_count,
        AVG(tip_amount) AS mean_tip
    GROUP BY ALL
    ORDER BY ALL
    """
    )

6.27071575

This is all the more surprising as the julia example does not even convert the result to a dataframe yet, while the dbGetQuery call in R does that as well. I find it really puzzling as I was expecting to obtain close to identical timing (given that the actual work is done by duckdb and neither R nor julia.)

Any insights on this one?

5 posts - 3 participants