# What's the latest and greatest in data in Julia
**URL:** https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271
**Category:** Data
**Created:** [July 20, 2024, 12:17pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271 "2024-07-20T12:17:31Z")
**Posts on this page:** 10
**Page:** 2
### Author:  [@Satvik](https://discourse.julialang.org/u/Satvik)
#### Post date: [July 30, 2024, 9:26pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/22 "2024-07-30T21:26:23Z")
I’ve also had issues on the other side, where I could read Arrow data in Julia but not pyarrow. Overall I’ve come to the conclusion that Arrow is just not a very mature format yet, even though it looks promising in some ways.
---
### Author:  [@adienes](https://discourse.julialang.org/u/adienes)
#### Post date: [July 30, 2024, 9:30pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/23 "2024-07-30T21:30:13Z")
I experience similar problems with Parquet, not just Arrow. see e.g. [LZ4 decompression failed (#48) · Issues · Expanding Man / Parquet2.jl · GitLab](https://gitlab.com/ExpandingMan/Parquet2.jl/-/issues/48)
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [July 30, 2024, 11:42pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/24 "2024-07-30T23:42:23Z")
> [@adienes](#):
>
> I experience similar problems with Parquet, not just Arrow
DuckDB directly supports Parquet, so I tried your last file and it can directly be read without any issues:
```julia
using QuackIO, Tables
read_parquet(columntable, "mwe2.parquet")
@btime read_parquet(columntable, "mwe2.parquet").x0 |> sum
# 683.894 ms (1810544 allocations: 396.49 MiB)
```
---
### Author:  [@drizk1](https://discourse.julialang.org/u/drizk1)
#### Post date: [July 31, 2024, 1:07am UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/25 "2024-07-31T01:07:19Z")
as pointed out above, parquet + duckdb work which means you could use TidierDB.jl to work on the file directly without reading it into memory
```julia
using TidierDB
db = connect(duckdb())
@chain db_table(db, "path_to_mwe2.parquet") begin
### your work here
@collect #to bring it to a local df
end
```
---
### Author:  [@nhz2](https://discourse.julialang.org/u/nhz2)
#### Post date: [July 31, 2024, 1:14am UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/26 "2024-07-31T01:14:44Z")
Here is an example comparing performance to OhMyThreads.jl and Mmap.jl
```julia-repl
julia> using QuackIO, Tables, OhMyThreads, Mmap, BenchmarkTools
julia> @btime read_parquet(columntable, "mwe2.parquet").x0 |> sum
893.883 ms (1810628 allocations: 396.49 MiB)
-8513239905662102228
julia> write("x0.bin", collect(Int64, read_parquet(columntable, "mwe2.parquet").x0));
julia> @btime sum(mmap("x0.bin", Vector{Int64}))
120.969 μs (16 allocations: 1.20 KiB)
-8513239905662102228
julia> @btime treduce(+, mmap("x0.bin", Vector{Int64}))
59.583 μs (362 allocations: 30.05 KiB)
-8513239905662102228
```
---
### Author:  [@dlakelan](https://discourse.julialang.org/u/dlakelan)
#### Post date: [July 31, 2024, 5:36am UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/27 "2024-07-31T05:36:57Z")
I love the explicitness of Julia, DataFrames, and DataFramesMeta and have no plans to go back to Tidy type stuff, but glad people have options.
DuckDB seems great and I like just writing SQL code for more complex queries. I used to use sqldf all the time in R.
I’ve only used arrow stuff occasionally but never had problems with it in Julia. But it was relatively simple stuff, like streaming a CSV from the census through a Julia filter and then streaming it out to an arrow file where I could reaccess the filtered data quickly later.
---
### Author:  [@greatpet](https://discourse.julialang.org/u/greatpet)
#### Post date: [August 1, 2024, 1:12pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/28 "2024-08-01T13:12:38Z")
The [DuckDB.jl](https://github.com/kimmolinna/DuckDB.jl) repository seems to be archived already. Is it working nicely anyway?
---
### Author:  [@DanielVandH](https://discourse.julialang.org/u/DanielVandH)
#### Post date: [August 1, 2024, 1:19pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/29 "2024-08-01T13:19:35Z")
The autolinked DuckDB.jl links to the old version. It’s now here [duckdb/tools/juliapkg at 6e4f15a3d6cee050ddc94b4bde5d27af4d83097a · duckdb/duckdb · GitHub](https://github.com/duckdb/duckdb/tree/6e4f15a3d6cee050ddc94b4bde5d27af4d83097a/tools/juliapkg)
---
### Author:  [@xgdgsc](https://discourse.julialang.org/u/xgdgsc)
#### Post date: [August 15, 2024, 5:12am UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/30 "2024-08-15T05:12:57Z")
What’ s the plan on JDF.jl , especially issues with large dataset like [Cannot write DataFrames with more than 2^31 rows · Issue #87 · xiaodaigh/JDF.jl (github.com)](https://github.com/xiaodaigh/JDF.jl/issues/87) ?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [August 15, 2024, 12:28pm UTC](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271/31 "2024-08-15T12:28:42Z")
I have not mentally recovered sufficiently to continue working. I still want to write something but chances are I wont be looking at github much soon.
[Previous page](https://discourse.julialang.org/t/whats-the-latest-and-greatest-in-data-in-julia/117271.md?page=1)