Today Julia Computing is excited to announce JuliaDB.jl ( https://github.com/JuliaComputing/JuliaDB.jl ), a package for working with large persistent data sets. It is still at a fairly early stage, but we wanted to release it as soon as we had meaningful functionality. JuliaDB ties together several e…

[image] davidanthoff: I’m not sure the name helps here, maybe at least add the word distributed somehow, that seems really the core idea here? That’s why it’s called Julia Distributed Bytes :slight_smile:

We’re working on benchmarks; some will be posted fairly soon. It would definitely be good to support more file formats, especially feather and parquet. So far we get a small amount of compression from PooledArrays (for columns with few unique values), but this is also something we’ll keep working o…

Don’t forget fst and feather. Any non-official previous results?

Don’t forget fst and feather. Fair enough, but they correspond only to a part of JuliaDB.jl’s functionality, namely serialization and deserialization.

Does fst exist as a file storage format outside of the R package http://www.fstpackage.org/ ? I got the impression from the documentation that the file storage format is not documented other than in the code and that it is subject to change. Specifically, the page states Note to users: The binary …

[image] shashi: The candidates for comparison we are considering are pandas.Series and xts, (maybe also data.table) which are columnar in-memory databases in the same design space as JuliaDB.jl, rather than sqlite. Comparing IndexedTables.jl with pandas seems to make a lot of sense, but the s…

Currently the way JuliaDB handles distributed datasets and storage is pretty tied to the index concept, so it’s related. I would be fine with a non-descriptive name if we can think of a good one.

Yes, we can only fairly compare single-process performance. The idea is to get to within reasonable speed of pandas including whatever overhead comes from wrapping IndexedTables.jl with Dagger.jl’s scheduler on a single process, and then demonstrate some speed ups vs single process performance with …

Why single-core? Do you mean single core or single thread?

Because I believe pandas, for example, doesn’t come with multi-processing/multi-threading.

ANN: JuliaDB.jl

Community

ChrisRackauckas May 8, 2017, 5:10pm 11

That’s why it’s called Julia Distributed Bytes

Topic		Replies	Views
The naming of JuliaDB.jl Offtopic	47	4729	June 7, 2017
JuliaDB, dataframes: Speculations over the future of data packages Data	24	7576	August 21, 2020
Why do you use JuliaDB? General Usage	9	2193	October 28, 2019
What's the latest and greatest in data in Julia Data	29	2462	August 15, 2024
Package for reading/writing ~100GB data files General Usage	10	2963	November 17, 2018

ANN: JuliaDB.jl

Related topics