# Struggling with Julia and large datasets

**URL:** <https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916>\
**Category:** General Usage\
**Tags:** question, big-data\
**Created:** [March 10, 2021, 11:01pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916 "2021-03-10T23:01:16Z")\
**Posts on this page:** 8\
**Page:** 4

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [June 6, 2021, 3:16am UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/62 "2021-06-06T03:16:29Z")

</div>

> [@Satvik](#):
>
> This lets you create an array larger than memory, where only part of the array is loaded into memory at a time. See [Memory-mapped I/O · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Mmap/)
> 
> Another option is to look into using distributed Julia. In Julia, a “cluster” is just a bunch of machines with passwordless ssh access to each other, so it’s actually relatively easy to set up. I use KissCluster on AWS. Here are some relevant links:

Generally good advice. Hmmm, perhaps it’s time to seriously start on my DiskFrames.jl project after I finish updating JDF.jl. It will be Julia-implementation of R’s {disk.frame} for working with large datasets.

JuliaDB.jl had lots of promising, but it kinda fell away I think.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [June 6, 2021, 3:17am UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/63 "2021-06-06T03:17:39Z")

</div>

> [@jebej](#):
>
> Try converting the dataset to an HDF5 file. If you can’t create the data in one go because loading the whole csv is impossible, you can incrementally write to the file:

If you cross Julia-version compatibility is not an issue at the moment, then I JDF.jl works pretty well for me. I wrote it so I can save/load tabular data fast.

---

<div class="post-metadata">

**Author:** ![djholiver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/djholiver/32/50470_2.png) [@djholiver](https://discourse.julialang.org/u/djholiver)\
**Post date:** [June 6, 2021, 9:36am UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/64 "2021-06-06T09:36:52Z")

</div>

Hi,

If you havent done so already (I couldn’t find it) can I suggest that you do a package announcement for this - it’s pretty huge imo having just checked it. Including arrow comparability in your release is really impressive and forward thinking.

Have you chained the SQL querying with normal Julia after the fact i.e. query into dataframe then act with normal Julia? How does that work with collect - is the code distributed “as normal” with spark?

I may have a backend re write to do…

Regards,

---

<div class="post-metadata">

**Author:** ![sampope](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@sampope](https://discourse.julialang.org/u/sampope)\
**Post date:** [June 7, 2021, 4:31pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/65 "2021-06-07T16:31:55Z")

</div>

> [@propelledanalytics](#):
>
> Apache Spark might be more performant for your workload size than Apache Drill.

Thanks! I did look into Spark and it involves more setup and coding than I wanted to do for this prototype. What I liked about Drill is that it’s just filesystem and run queries, no real setup needed. As we move out of prototype and into something that needs real performance, we’ll have to re-evaluate the decisions made so far.

---

<div class="post-metadata">

**Author:** ![propelledanalytics](https://avatars.discourse-cdn.com/v4/letter/p/e47774/32.png) [@propelledanalytics](https://discourse.julialang.org/u/propelledanalytics)\
**Post date:** [June 11, 2021, 5:11pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/66 "2021-06-11T17:11:14Z")

</div>

> [@djholiver](#):
>
> Have you chained the SQL querying with normal Julia after the fact i.e. query into dataframe then act with normal Julia?

Hi djholiver,

SparkSQL.jl is designed to make Julia lang the easiest and most optimal way to use Apache Spark for tabular data. Your use case is works and there is a new tutorial on how to do that here:

> **[SparkSQL.jl Blog](https://propelledanalytics.github.io/Tutorials/#working-with-data-tutorial)**
>
> Tutorials on how to use the SparkSQL.jl Julia Package.

Here’s a quick example:

```julia
JuliaDataFrame = DataFrame(tickers = ["CRM", "IBM"])
onSpark = toSparkDS(sprk, JuliaDataFrame)
createOrReplaceTempView(onSpark, "julia_data")
query = sql(sprk, "SELECT * FROM spark_data WHERE TICKER IN (SELECT * FROM julia_data)")
results = toJuliaDF(query)
describe(results)

```

.  
In the above example we created a Julia DataFrame, used it in a query on Apache Spark, brought the query results back to a Julia DataFrame and then used the describe function in Julia.

The SparkSQL.jl official documentation:

> **[SparkSQL.jl](https://propelledanalytics.github.io/SparkSQL.jl/)**
>
> SparkSQL.jl enables Julia programs to work with Apache Spark data using just SQL.

---

<div class="post-metadata">

**Author:** ![propelledanalytics](https://avatars.discourse-cdn.com/v4/letter/p/e47774/32.png) [@propelledanalytics](https://discourse.julialang.org/u/propelledanalytics)\
**Post date:** [June 14, 2021, 5:52pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/67 "2021-06-14T17:52:48Z")

</div>

> [@sampope](#):
>
> Thanks! I did look into Spark and it involves more setup and coding than I wanted to do for this prototype. What I liked about Drill is that it’s just filesystem and run queries, no real setup needed. As we move out of prototype and into something that needs real performance, we’ll have to re-evaluate the decisions made so far.

Hi sampope,

SparkSQL.jl is the easiest way to use Apache Spark. There is no need to learn Scala, Python, or Java. All you need to know is Julia and SQL. The SparkSQL.jl package supports queries on the file system too:

```julia
file = sql(sprk, "SELECT _c0 AS TICKER, to_date(_c1, 'yyyymmdd') AS TRADING_DATE, CAST(_c5 AS double) AS CLOSE FROM CSV.`/tmp/sampleData.csv`;");
createOrReplaceTempView(file, "stocks");
write = sql(sprk, "CREATE TABLE IF NOT EXISTS stocks USING PARQUET LOCATION '/tmp/stocks.PARQUET' AS SELECT * FROM stocks;")

```

.

The above code converts a CSV file to Parquet format in just three lines. The full tutorial can be found here:

> **[SparkSQL.jl Blog](https://propelledanalytics.github.io/Tutorials/#working-with-data-tutorial)**
>
> Tutorials on how to use the SparkSQL.jl Julia Package.

As your requirements grow, SparkSQL.jl can help. The SparkSQL.jl package is designed to support many advanced features including Delta Lake. Delta Lake architecture is a best practice for multi-petabyte and trillion+ row datasets.

To learn more visit the official project page of SparkSQL.jl:

> **[GitHub - propelledanalytics/SparkSQL.jl: SparkSQL.jl enables Julia programs...](https://github.com/propelledanalytics/SparkSQL.jl)**
>
> SparkSQL.jl enables Julia programs to work with Apache Spark data using just SQL. - GitHub - propelledanalytics/SparkSQL.jl: SparkSQL.jl enables Julia programs to work with Apache Spark data using ...

---

<div class="post-metadata">

**Author:** ![blackeneth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/blackeneth/32/10353_2.png) [@blackeneth](https://discourse.julialang.org/u/blackeneth)\
**Post date:** [June 15, 2021, 6:03pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/68 "2021-06-15T18:03:42Z")

</div>

If you have discovered it yet, take a look into Floops.jl and the @Floops macro. Perhaps an easy way to get some parallel processing into your code.

Here is a video of Prof. Alan Edelman doing a demo of Floops:  
[Parallel processing on your own machine](https://youtu.be/dczkYlOM2sg)

One of the things I admire about Prof. Edelman is the enthusiasm for everything he does.

---

<div class="post-metadata">

**Author:** ![tdunning](https://avatars.discourse-cdn.com/v4/letter/t/13edae/32.png) [@tdunning](https://discourse.julialang.org/u/tdunning)\
**Post date:** [October 17, 2024, 11:42pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916/69 "2024-10-17T23:42:00Z")

</div>

It’s three years later, but this may help others with similar problems.

First, I was a co-founder of Apache Drill and I love that it provided value for what you needed. That said, I would strongly suggest DuckDB for this kind of problem. The level of complexity for the setup is very low and it should be able to make use of the cores available on a single machine in a very efficient way. Drill pays a penalty for transferring data (as would Spark) because it is designed to work across a relatively large number of machines. Duck makes the best use of a single machine and often can provide performance equal to what Drill, Spark or Presto achieve on a dozen machines.

In addition, it is important to store your data in a columnar form that provides good compression. Parquet is the right choice there, but you should investigate sorting your data to maximize row to row coherence. Typically, this means sorting by data source, then by time. This can lead to _massive_ compression (I have seen 300:1 on weather data) and that, in turn, leads to stunning performance for queries.

Duck also integrates directly into Julia very nicely. If your final results are small enough to fit into memory, then Duck query results can become DataFrames with very little overhead.

[Previous page](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916.md?page=3)