Hello,
Is anyone else experiencing problems reading parquet files in Julia 1.9.0? I was checking how my code performed when upgrading my Julia version from 1.8.4 to 1.9.0, and noticed the first run being extremely slow. Anyone know why the compilation time is 9-10x slower in Julia 1.9.0 for reading parquet files?
The performance for the 2 versions for me is as follows.
I only loaded 1 parquet file in each version.
Julia 1.8.4, compile time
39.993073 seconds (85.53 M allocations: 3.422 GiB, 3.29% gc time, 99.58% compilation time)
Julia 1.9.0, compile time
365.131727 seconds (44.78 M allocations: 2.288 GiB, 0.24% gc time, 99.98% compilation time: <1% of which was recompilation)
using Parquet
using DataFrames
load parquet file to dataframe
function history(start_year,end_year)
df = DataFrame()
for year in start_year:end_year
path = “…”
dfn = DataFrame(read_parquet(string(path,,“something_$year.parquet”)))
df = vcat(df,dfn)
end
return df
end