Today Julia Computing is excited to announce JuliaDB.jl ( https://github.com/JuliaComputing/JuliaDB.jl ), a package for working with large persistent data sets. It is still at a fairly early stage, but we wanted to release it as soon as we had meaningful functionality. JuliaDB ties together several e…

Don’t forget fst and feather. Fair enough, but they correspond only to a part of JuliaDB.jl’s functionality, namely serialization and deserialization.

Does fst exist as a file storage format outside of the R package http://www.fstpackage.org/ ? I got the impression from the documentation that the file storage format is not documented other than in the code and that it is subject to change. Specifically, the page states Note to users: The binary …

[image] shashi: The candidates for comparison we are considering are pandas.Series and xts, (maybe also data.table) which are columnar in-memory databases in the same design space as JuliaDB.jl, rather than sqlite. Comparing IndexedTables.jl with pandas seems to make a lot of sense, but the s…

Currently the way JuliaDB handles distributed datasets and storage is pretty tied to the index concept, so it’s related. I would be fine with a non-descriptive name if we can think of a good one.

Yes, we can only fairly compare single-process performance. The idea is to get to within reasonable speed of pandas including whatever overhead comes from wrapping IndexedTables.jl with Dagger.jl’s scheduler on a single process, and then demonstrate some speed ups vs single process performance with …

Why single-core? Do you mean single core or single thread?

Because I believe pandas, for example, doesn’t come with multi-processing/multi-threading.

Could somebody please clarify what “distributed datasets” mean in this context? Coming from Hadoop and distributed databases I understand it as a set of files stored on multiple machines with the ability to run particular code or a query locally without copying data over a network. However, in a des…

It is a fully distributed model; it should be possible to use it with files stored on multiple machines. Also, we don’t load the data and then copy it to a remote machine in two steps, rather the remote machine that will handle a certain chunk does the loading itself.

ANN: JuliaDB.jl

Community

Juan May 8, 2017, 6:02pm 16

Don’t forget fst and feather.

Any non-official previous results?

Topic		Replies	Views
The naming of JuliaDB.jl Offtopic	46	5055	June 7, 2017
JuliaDB, dataframes: Speculations over the future of data packages Data	24	7782	August 21, 2020
Why do you use JuliaDB? General Usage	9	2266	October 28, 2019
What's the latest and greatest in data in Julia Data	29	2825	August 15, 2024
Package for reading/writing ~100GB data files General Usage	10	3077	November 17, 2018

ANN: JuliaDB.jl

Related topics