# Developing a Beginner's Roadmap to Learn Julia High Performance Computing for Data Science

**URL:** <https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970>\
**Category:** Data\
**Tags:** first-steps, performance, data, big-data\
**Created:** [July 3, 2021, 3:06am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970 "2021-07-03T03:06:22Z")\
**Posts on this page:** 11\
**Page:** 2

<div class="post-metadata">

**Author:** ![brenhinkeller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brenhinkeller/32/36648_2.png) [@brenhinkeller](https://discourse.julialang.org/u/brenhinkeller)\
**Post date:** [July 4, 2021, 7:45pm UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/21 "2021-07-04T19:45:08Z")

</div>

Celeste is an interesting case because they don’t actually have an explicit dependency on MPI.jl, and it’s a bit tricky to figure out exactly what they were doing from looking at the source, but they do mention in [interviews](https://cs.lbl.gov/news-media/news/2016/celeste-enhancements-create-new-opportunities-in-sky-surveys/)

> we have integrated the DTree scheduler and utilized MPI-3 one-sided communication primitives

so they must have been hooking in to the cluster’s MPI at some level.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [July 4, 2021, 8:27pm UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/22 "2021-07-04T20:27:22Z")

</div>

> [@ImreSamu](#):
>
> split the data into lots of small packets ( data frames, cells )

/ ~ data cubes / Multidimensional Tiling / ~ partitioning/splitting the data by dimensions

for example, we have only 3 Dimensions:

- Date/Time ( ~ like splitting by Month )
- Spatial ( [Quadtiles](https://wiki.openstreetmap.org/wiki/QuadTiles) ; [H3](https://eng.uber.com/h3/) ; [S2](https://s2geometry.io/devguide/s2cell_hierarchy.html) )
- Genus / Species / Taxon

And each cell has a ~ 0…10000 trees  
So we can split the data with small “data tiles” ( cubes )

so just generate a big tasklist to a `./bio_tasklist.sh`

```julia
./julia_bio_task.sh 2021-01 ADBA Tilia
./julia_bio_task.sh 2021-01 ADBA Populus
./julia_bio_task.sh 2021-01 ADBA Fraxinus
./julia_bio_task.sh 2021-02 ADBA Tilia
./julia_bio_task.sh 2021-02 ADBA Populus
./julia_bio_task.sh 2021-02 ADBA Fraxinus
...

```

And run parallel: the "–jobs $(nproc) "== parallel tasks based on cpu counts.

```julia
time parallel --delay 2 --jobs $(nproc) --results ./jobs/bio_tasks -k < ./bio_tasklist.sh

```

see:

- [Gnu Parallel - Parallelize Serial Command Line Programs Without Changing Them](https://www.biostars.org/p/63816/)
- [GNU Parallel - GNU Project - Free Software Foundation](https://www.gnu.org/software/parallel/)
- [Chapter 8 Parallel Pipelines | Data Science at the Command Line, 2e](https://www.datascienceatthecommandline.com/2e/chapter-8-parallel-pipelines.html#controlling-the-number-of-concurrent-jobs)

In the OSM there are some biological data - if you need 🙂

- [genus | Keys | OpenStreetMap Taginfo](https://taginfo.openstreetmap.org/keys/genus#values)
- [genus:en | Keys | OpenStreetMap Taginfo](https://taginfo.openstreetmap.org/keys/genus%3Aen#values)
- [Tag:natural=tree - OpenStreetMap Wiki](https://wiki.openstreetmap.org/wiki/Tag:natural%3Dtree)

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [July 5, 2021, 12:47am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/23 "2021-07-05T00:47:20Z")

</div>

> [@TheCedarPrince](#):
>
> Would you mind explaining a bit more about doing work “in RAM”? What do you mean by that in the context of a data science processing pipeline? Like reading in files, performing computation, etc?

What’s a processing pipeline? It’s a bunch of transformations and side-effects, like storing data into S3.

Data process is done by retrieval of data from somewhere e.g. DBMS, HDFS, etc. Then processed by something, which is often a Spark CLUSTER. So replace that Spark cluster with a single computer. So everything can happen in RAM. E.g. replace it with a robust tool. Myabe a single node Spark? Or even Dask, disk.frame, vaex or let the user choose.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [July 5, 2021, 6:19am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/24 "2021-07-05T06:19:30Z")

</div>

There are technologies now which make large memory machines available and very fast access to data over networks, and composable infrastructure  
Intel Optane persistent memory is a tier below RAM.

Dell can provide multi-terabyte single servers using persistent memory. I would have to check what the limits are.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [July 5, 2021, 6:21am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/25 "2021-07-05T06:21:41Z")

</div>

You can have composable infrastrucures - where you build a high performance server on demand with a set of CPUs, GPUs and high memory - which can be torn down and redistributed for the next trainign or analysis run.  
Dell work very closely with Liqid on this - have a look at this 12 Terabyte server

[https://www.liqid.com/dell-technologies/solution-bundles/application/liqid-composable-high-memory-appliance](https://www.liqid.com/dell-technologies/solution-bundles/application/liqid-composable-high-memory-appliance)

Also look at Bluefield

> **[NVIDIA BlueField Data Processing Units (DPUs)](https://www.nvidia.com/en-gb/networking/products/data-processing-unit/)**
>
> Discover more ways to transform your data center.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [July 5, 2021, 6:29am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/26 "2021-07-05T06:29:30Z")

</div>

Comments about time to access data from network storage versus local storage on a server are getting a bit out of date.

Look at the Data Accelerator at the Dell Centre of Excellence in Cambridge

> **[dell-data-accelerator-cambridge.pdf](https://www.delltechnologies.com/content/dam/digitalassets/active/en/unauth/white-papers/products/ready-solutions/dell-data-accelerator-cambridge.pdf)**
>
> 7.48 MB

[https://www.dell.com/support/kbdoc/en-us/000122853/dell-emc-data-accelerator-reference-architecture](https://www.dell.com/support/kbdoc/en-us/000122853/dell-emc-data-accelerator-reference-architecture)

Weka storage say they are FASTER than local disk

> **[Home](https://www.weka.io/)**
>
> The WEKA data platform helps organizations store, process and manage data in the cloud and on-prem to fuel next-generation workloads.

Also look at Intel DAOS  
[https://www.intel.co.uk/content/www/uk/en/high-performance-computing/daos-high-performance-storage-brief.html](https://www.intel.co.uk/content/www/uk/en/high-performance-computing/daos-high-performance-storage-brief.html)

---

<div class="post-metadata">

**Author:** ![brenhinkeller](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brenhinkeller/32/36648_2.png) [@brenhinkeller](https://discourse.julialang.org/u/brenhinkeller)\
**Post date:** [July 5, 2021, 6:32am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/27 "2021-07-05T06:32:49Z")

</div>

> [@johnh](#):
>
> have a look at this 12 Terabyte server

Oh so _that’s_ what they’re using Optane for! Yeah, makes sense.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [July 5, 2021, 7:32am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/28 "2021-07-05T07:32:57Z")

</div>

> [@johnh](#):
>
> 12 Terabyte server

that’s amazing! Althought it’s 12TB of Optane so it’s a bit slower than DDRRAM but faster than SSDs. Optane is definitely what it is to SSD as what SSDs were to spinning hard drives.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [July 5, 2021, 7:46am UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/29 "2021-07-05T07:46:00Z")

</div>

> [@johnh](#):
>
> network storage versus local storage on a server are getting a bit out of date.

you make it sound like the specialist architecture is ubiquitous of which it is not. Most spark cluster suffer from poor network performance vs fetching from disk. It will be many years before it can be considered “out-dated”.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [July 5, 2021, 12:59pm UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/30 "2021-07-05T12:59:54Z")

</div>

@xiaodai Servers have a mic of DRAM and Optane memory - not all Optane. So depending on how much data you access at a time performance may not be affected highly. As usual YMMV

This is a subject that I am not expert in, however if there is interest I Can track down someone from Intel to comment.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [July 5, 2021, 11:53pm UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970/31 "2021-07-05T23:53:19Z")

</div>

> [@johnh](#):
>
> Servers have a mic of DRAM and Optane memory - not all Optane.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/9/1/910c13f353968549e4dabbcb3f7b35fcecd53649.png)

Of course, all computers need DRAM to function due to the current architecture. So you mentioned optane only so I pointed to the optane part.

[Previous page](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970.md?page=1)