# ANN: JuliaDB.jl

**URL:** <https://discourse.julialang.org/t/ann-juliadb-jl/3564>\
**Category:** Community\
**Created:** [May 6, 2017, 7:23pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564 "2017-05-06T19:23:38Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [May 8, 2017, 7:28pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/29 "2017-05-08T19:28:22Z")

</div>

Because I believe `pandas`, for example, doesn’t come with multi-processing/multi-threading.

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [May 8, 2017, 8:00pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/31 "2017-05-08T20:00:03Z")

</div>

Could somebody please clarify what “distributed datasets” mean in this context? Coming from Hadoop and [distributed databases](https://en.wikipedia.org/wiki/Distributed_database) I understand it as a set of files _stored on multiple machines_ with the ability to run particular code or a query locally without copying data over a network. However, in a description of both - JuliaDB.jl and Dagger.jl - I can see only examples of loading data _from a local disk_ and maybe copying it to other machines for processing.

In other words, is JuliaDB.jl more similar to DataTables or to Hadoop?

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [May 8, 2017, 8:15pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/33 "2017-05-08T20:15:18Z")

</div>

It is a fully distributed model; it should be possible to use it with files stored on multiple machines. Also, we don’t load the data and then copy it to a remote machine in two steps, rather the remote machine that will handle a certain chunk does the loading itself.

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [May 8, 2017, 8:27pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/34 "2017-05-08T20:27:14Z")

</div>

Are there any examples of such a workflow? So far it sounds like it may become a _huge_ thing for machine learning / big data community because existing systems like Apache Spark aren’t really well-suited for scientific computations.

---

<div class="post-metadata">

**Author:** ![datnamer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datnamer/32/3471_2.png) [@datnamer](https://discourse.julialang.org/u/datnamer)\
**Post date:** [May 8, 2017, 8:54pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/35 "2017-05-08T20:54:05Z")

</div>

because they dont have n dim distributed array?

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [May 8, 2017, 11:01pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/37 "2017-05-08T23:01:34Z")

</div>

Why so much interest in python’s pandas?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [May 8, 2017, 11:03pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/38 "2017-05-08T23:03:18Z")

</div>

> [@datnamer](#):
>
> because they dont have n dim distributed array?

They have [distributed matrices](https://spark.apache.org/docs/2.1.0/mllib-data-types.html#distributed-matrix) if this is what you mean. Not sure about higher-order arrays, though. But there are many other issues such as:

1. Java and Scala aren’t really well-suited for data science or machine learning - too verbose, too little libraries, too complicated to extend for most users. Python and R impose a serious performance loss.
2. Due to the nature of HDFS, you cannot control data distribution, e.g. you cannot collocate data and use local optimizations.
3. Data on HDFS cannot be modified. If you want to modify data, you have to save it to a different location.
4. Asynchronous processing is not supported. For example, [SGD in Spark](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala#L181-L279) is implemented as synchronous iterations over a set of partitions, each being processed by a separate worker. If one of the partitions is significantly larger than the others, or if one worker is significantly slower, all the other workers will wait for him instead of taking the next (as e.g. Google’s [Downpour SGD](https://static.googleusercontent.com/media/research.google.com/en//archive/large_deep_networks_nips2012.pdf) does), etc.

If JuliaDB.jl can handle terabyte-sized datasets without imposing Spark’s and HDFS’s limitations, it can become a very attractive framework for a much larger audience than Julia users.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [May 9, 2017, 12:14pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/43 "2017-05-09T12:14:14Z")

</div>

Could I ask you to say a little more about the specific use case? I saw this line

> [@jeff.bezanson](#):
>
> standard choice for managing persistent array and tabular data for Julia

and the docs’ “Work with data, the Julia way”, which was had me thinking this was a replacement for DataFrames. I realize now that the keyword was “persistent”, but I’m still not completely sure where it slots into the data ecosystem. Is it the ability to work with data on several computer cores at once that sets it apart, or is this a type of database? I read the docs (of course) but I am still not completely sure.

Sorry for the naive question, but as a biologist (rather than a programmer) I feel like I am missing something here that is obvious to the other commenters on the thread. My rationale for asking is that I do work a lot with data and I’m thus very interested in the DataFrames/DataTables/IterableTables/IndexedTables debate.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 9, 2017, 1:25pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/47 "2017-05-09T13:25:26Z")

</div>

Can it read gzipped files somehow? I have some datasets for which this would be ideal, stored in compressed (`gzip -9`) format on my HD, but uncompressing them would take 10x the space. But possibly the mmap-based design precludes this.

---

<div class="post-metadata">

**Author:** ![hpoit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hpoit/32/245_2.png) [@hpoit](https://discourse.julialang.org/u/hpoit)\
**Post date:** [May 9, 2017, 1:34pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/48 "2017-05-09T13:34:12Z")

</div>

[https://github.com/JuliaComputing/JuliaDB.jl/issues/48](https://github.com/JuliaComputing/JuliaDB.jl/issues/48)  
I’m working on it right now Tamas.

---

<div class="post-metadata">

**Author:** ![adriano.vilela](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adriano.vilela/32/7596_2.png) [@adriano.vilela](https://discourse.julialang.org/u/adriano.vilela)\
**Post date:** [May 9, 2017, 1:52pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/50 "2017-05-09T13:52:37Z")

</div>

Do you guys know how this compares to Anaconda’s Blaze ([https://docs.continuum.io/docs\_oss/blaze/](https://docs.continuum.io/docs_oss/blaze/))? Are the long term objectives of the projects similar? I know nothing about distributed data sets so I apologize if this is a silly question.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [May 11, 2017, 3:54pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/75 "2017-05-11T15:54:50Z")

</div>

Can this naming discussion be split out in its own thread please?

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [May 11, 2017, 5:09pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/76 "2017-05-11T17:09:45Z")

</div>

One thing that would be amazing (to me) is a conversation about what exactly this package does, and how it relates to the other core data handling packages in the julia package ecosystem.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 11, 2017, 5:41pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/77 "2017-05-11T17:41:44Z")

</div>

Alright, things were fairly intertwined, but I did my best to split off the naming discussion into: [The naming of JuliaDB.jl](https://discourse.julialang.org/t/the-naming-of-juliadb-jl/3653/1)

One portion I was a little sad to see go was:

> [@The naming of JuliaDB.jl](https://discourse.julialang.org/t/the-naming-of-juliadb-jl/3653/2):
>
> An aspect of understanding JuliaDB is that, compared to traditional SQL databases, which are all about transactional storage, JuliaDB is more about persistent computation and bringing code to data efficiently and seamlessly – including being able to store any kind of Julia data without fuss. So the “Julia” part in the name can be thought of as putting a strong emphasis on code.

Let’s try to keep this thread focused the functionality of JuliaDB.

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [May 11, 2017, 5:50pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/78 "2017-05-11T17:50:05Z")

</div>

The goal of this package is to be more end-to-end, providing something at a similar level of abstraction as a traditional database: create, update, and query operations connected to a persistent data store. So for now you could see it as just a stand-alone thing. However eventually it should become an umbrella package, or perhaps just a set of standard APIs, that connect appropriate data structures, file formats, and operations into a standard workflow.

---

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [May 11, 2017, 5:51pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/79 "2017-05-11T17:51:28Z")

</div>

It would be great to support reading compressed files. I was thinking of going about this using FileIO.jl or similar abstraction to somehow read zip and gz files.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [May 11, 2017, 5:55pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/80 "2017-05-11T17:55:34Z")

</div>

Thanks - this sounds very promising and exciting.

---

<div class="post-metadata">

**Author:** ![hpoit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hpoit/32/245_2.png) [@hpoit](https://discourse.julialang.org/u/hpoit)\
**Post date:** [May 11, 2017, 6:10pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/81 "2017-05-11T18:10:36Z")

</div>

Why would this implementation not be straight forward? It sounds like a good plan.

---

<div class="post-metadata">

**Author:** ![js135005](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/js135005/32/8219_2.png) [@js135005](https://discourse.julialang.org/u/js135005)\
**Post date:** [May 11, 2017, 6:19pm UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/82 "2017-05-11T18:19:05Z")

</div>

It should be fairly straightforward to incorporate the buffered input streams from Libz. Behind the scenes, I am working on an alternative to ZipFile for zip archives that doesn’t have a 20x slowdown compared to reading uncompressed files directly. Unfortunately no guarantee as to when I might have something like that working.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [November 13, 2018, 1:59am UTC](https://discourse.julialang.org/t/ann-juliadb-jl/3564/84 "2018-11-13T01:59:54Z")

</div>

A post was split to a new topic: [Package for reading/writing ~100GB data files](https://discourse.julialang.org/t/package-for-reading-writing-100gb-data-files/17458)

[Previous page](https://discourse.julialang.org/t/ann-juliadb-jl/3564.md?page=1)

[Next page](https://discourse.julialang.org/t/ann-juliadb-jl/3564.md?page=3)