Today Julia Computing is excited to announce JuliaDB.jl ( https://github.com/JuliaComputing/JuliaDB.jl ), a package for working with large persistent data sets. It is still at a fairly early stage, but we wanted to release it as soon as we had meaningful functionality. JuliaDB ties together several e…

This was written to solve a particular need (fast analytics on out of core datasets), and in doing so takes a certain design path – using indexing and Dagger. So I doubt that there is an intention to replace anything, just an expectation that this is useful to many users.

It looks very interesting! I’d also like to see how this compares with DataFrames. Something that I’m curious about and I’m not sure I’ve understood is whether this data structure will provide its own optimised ways of doing general data manipulation (e.g. by, groupby, join, etc.) that take advantag…

Just to second @avik , I took a look, and this does not at all look like a generic package that is meant to handle all data situations in julia that would replace things like DataFrame. I think it looks fantastic, but it also appears to target a very specific use case. I’m not sure the name helps her…

[image] davidanthoff: I’m not sure the name helps here, maybe at least add the word distributed somehow, that seems really the core idea here? That’s why it’s called Julia Distributed Bytes :slight_smile:

We’re working on benchmarks; some will be posted fairly soon. It would definitely be good to support more file formats, especially feather and parquet. So far we get a small amount of compression from PooledArrays (for columns with few unique values), but this is also something we’ll keep working o…

Don’t forget fst and feather. Any non-official previous results?

Don’t forget fst and feather. Fair enough, but they correspond only to a part of JuliaDB.jl’s functionality, namely serialization and deserialization.

Does fst exist as a file storage format outside of the R package http://www.fstpackage.org/ ? I got the impression from the documentation that the file storage format is not documented other than in the code and that it is subject to change. Specifically, the page states Note to users: The binary …

[image] shashi: The candidates for comparison we are considering are pandas.Series and xts, (maybe also data.table) which are columnar in-memory databases in the same design space as JuliaDB.jl, rather than sqlite. Comparing IndexedTables.jl with pandas seems to make a lot of sense, but the s…

ANN: JuliaDB.jl

Community

Topic		Replies	Views
The naming of JuliaDB.jl Offtopic	47	4747	June 7, 2017
JuliaDB, dataframes: Speculations over the future of data packages Data	24	7590	August 21, 2020
Why do you use JuliaDB? General Usage	9	2194	October 28, 2019
What's the latest and greatest in data in Julia Data	29	2476	August 15, 2024
Package for reading/writing ~100GB data files General Usage	10	2966	November 17, 2018

ANN: JuliaDB.jl

Related topics