# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=8

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 9

---

## [Spark.jl: Can we abandon RDD API?](https://discourse.julialang.org/t/spark-jl-can-we-abandon-rdd-api/45063)

<div class="topic-metadata">

**Author:** [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Replies:** 1\
**Last updated:** [August 17, 2020, 1:01am UTC](https://discourse.julialang.org/t/spark-jl-can-we-abandon-rdd-api/45063 "2020-08-17T01:01:19Z")

</div>

Cross-posting from GitHub issue. Originally, Spark.jl was developed for Spark 1.x with its RDD interface, e.g. most useful functions were map , filter , etc. In Spark 2.0, the default interface changed to Dataset ( D…

---

## [Activating virtual environment on workers](https://discourse.julialang.org/t/activating-virtual-environment-on-workers/44806)

<div class="topic-metadata">

**Author:** [@martincornejo](https://discourse.julialang.org/u/martincornejo)\
**Replies:** 5\
**Last updated:** [August 12, 2020, 3:15pm UTC](https://discourse.julialang.org/t/activating-virtual-environment-on-workers/44806 "2020-08-12T15:15:37Z")

</div>

Say we want to use certain package (that is only available in a project-specific virtual environment) in a distributed manner. (@v1.5) pkg\> activate . Activating environment at \`C:\\Users\\Martin Cornejo\\MyProject\\Projec…

---

## [Run a julia application at large scale (on thousands of nodes)](https://discourse.julialang.org/t/run-a-julia-application-at-large-scale-on-thousands-of-nodes/23873)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 8\
**Last updated:** [August 10, 2020, 2:52pm UTC](https://discourse.julialang.org/t/run-a-julia-application-at-large-scale-on-thousands-of-nodes/23873 "2020-08-10T14:52:52Z")

</div>

Overall context I am preparing a weak scaling test of a distributed Julia application. The test will use up to a couple of thousands of compute nodes and will need to run multiple times for each configuration (to obtain…

---

## [Best Practices for Parallel implementation of Gibbs Sampler algorithm](https://discourse.julialang.org/t/best-practices-for-parallel-implementation-of-gibbs-sampler-algorithm/44503)

<div class="topic-metadata">

**Author:** [@deburgess](https://discourse.julialang.org/u/deburgess)\
**Replies:** 0\
**Last updated:** [August 7, 2020, 2:53pm UTC](https://discourse.julialang.org/t/best-practices-for-parallel-implementation-of-gibbs-sampler-algorithm/44503 "2020-08-07T14:53:05Z")

</div>

Thanks in advance for your comments and advice. Would someone comment on how to improve my coding? In particular: did I use @sync properly? Did I use Distributed arrays efficiently? How does random number generation w…

---

## [Surprising capture boxing behavior in closure](https://discourse.julialang.org/t/surprising-capture-boxing-behavior-in-closure/20254)

<div class="topic-metadata">

**Author:** [@NHDaly](https://discourse.julialang.org/u/NHDaly)\
**Replies:** 9\
**Last updated:** [August 4, 2020, 3:51am UTC](https://discourse.julialang.org/t/surprising-capture-boxing-behavior-in-closure/20254 "2020-08-04T03:51:31Z")

</div>

Hi, I was surprised by the boxing behavior of closure capture, and I’d love to hear about its motivation and discuss whether it’s still relevant. I came across this when I noticed how it can introduce a race condition i…

---

## [Simultaneous internode I/O and compute using DistributedArrays?](https://discourse.julialang.org/t/simultaneous-internode-i-o-and-compute-using-distributedarrays/42910)

<div class="topic-metadata">

**Author:** [@Matt\_Davis](https://discourse.julialang.org/u/Matt_Davis)\
**Replies:** 0\
**Last updated:** [July 12, 2020, 12:40am UTC](https://discourse.julialang.org/t/simultaneous-internode-i-o-and-compute-using-distributedarrays/42910 "2020-07-12T00:40:22Z")

</div>

I am attempting to implement multinode tensor contraction, and I am running in to some issues. My current approach is, given two input tensors A and B and an output tensor C (all DArrays), to spawn on each worker a loop…

---

## [Automatic Parallelization in Julia](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543)

<div class="topic-metadata">

**Author:** [@Honza9723](https://discourse.julialang.org/u/Honza9723)\
**Replies:** 5\
**Last updated:** [July 5, 2020, 3:05am UTC](https://discourse.julialang.org/t/automatic-parallelization-in-julia/42543 "2020-07-05T03:05:08Z")

</div>

Dear All, I would like to ask you, whether there is some automatic parallelization tool in Julia similar to auto parallelization capabilities of Intel/gcc compilers for Fortran/C++. It would be really awesome if Julia c…

---

## [Volcano - HPC workloads on Kubernetes](https://discourse.julialang.org/t/volcano-hpc-workloads-on-kubernetes/42187)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 1\
**Last updated:** [June 28, 2020, 11:05am UTC](https://discourse.julialang.org/t/volcano-hpc-workloads-on-kubernetes/42187 "2020-06-28T11:05:25Z")

</div>

I have not tried this but I like the concept. with Kubernetes being more and more deployed has anyone workedon Julia in such an environment? What do we think of the Volcano project as a way of running Julia in paralle…

---

## [Distributed startup on large clusters](https://discourse.julialang.org/t/distributed-startup-on-large-clusters/42186)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 0\
**Last updated:** [June 28, 2020, 5:21am UTC](https://discourse.julialang.org/t/distributed-startup-on-large-clusters/42186 "2020-06-28T05:21:27Z")

</div>

I note that in Julia 1.5 IPV6 is supported for launch\_on\_machine(). Which is a good thing. https://github.com/JuliaLang/julia/pull/34430 This leads me to revive and old topic. Julia uses ssh connections to start proce…

---

## [Gcc vs Threads.@threads vs Threads.@spawn for large loops](https://discourse.julialang.org/t/gcc-vs-threads-threads-vs-threads-spawn-for-large-loops/34273)

<div class="topic-metadata">

**Author:** [@j-fu](https://discourse.julialang.org/u/j-fu)\
**Replies:** 14\
**Last updated:** [June 26, 2020, 7:45pm UTC](https://discourse.julialang.org/t/gcc-vs-threads-threads-vs-threads-spawn-for-large-loops/34273 "2020-06-26T19:45:03Z")

</div>

Hi, Schönauer vector triad again, see also the benchmarking site of Georg Hager. This time we compare multithreading vs. scalar, and also compare to gcc. Here ist the generating code. We see that for scalar perform…

---

## [Threading two consecutive double loops](https://discourse.julialang.org/t/threading-two-consecutive-double-loops/41931)

<div class="topic-metadata">

**Author:** [@jcook](https://discourse.julialang.org/u/jcook)\
**Replies:** 0\
**Last updated:** [June 23, 2020, 11:35am UTC](https://discourse.julialang.org/t/threading-two-consecutive-double-loops/41931 "2020-06-23T11:35:08Z")

</div>

Hi All, I’m trying to improve the threaded performance of some code: I’ve written up the associated MWE. Essentially we have a matrix m, which is mutated in two consecutive double for loops. The first loop: The fir…

---

## [Pmap batch example](https://discourse.julialang.org/t/pmap-batch-example/41849)

<div class="topic-metadata">

**Author:** [@jamblejoe](https://discourse.julialang.org/u/jamblejoe)\
**Replies:** 0\
**Last updated:** [June 22, 2020, 9:48am UTC](https://discourse.julialang.org/t/pmap-batch-example/41849 "2020-06-22T09:48:40Z")

</div>

Hi, can someone provide me with a good example of using the batch\_size parameter of the pmap function? I found Parallelism - Understanding pmap and the batch\_size parameter which indicates a performance increase by incr…

---

## [Workflow tips for MPI-based code](https://discourse.julialang.org/t/workflow-tips-for-mpi-based-code/41131)

<div class="topic-metadata">

**Author:** [@fverdugo](https://discourse.julialang.org/u/fverdugo)\
**Replies:** 0\
**Last updated:** [June 10, 2020, 2:39pm UTC](https://discourse.julialang.org/t/workflow-tips-for-mpi-based-code/41131 "2020-06-10T14:39:24Z")

</div>

Hi! I would be grateful if someone can help to speed-up my workflow when developing code based on MPI. The problem is that the code needs to be executed from the terminal: $ mpirun -np 5 julia my\_julia\_mpi\_code.jl w…

---

## [How do I use multithreaded BLAS in each MPI process](https://discourse.julialang.org/t/how-do-i-use-multithreaded-blas-in-each-mpi-process/40587)

<div class="topic-metadata">

**Author:** [@shipengcheng1230](https://discourse.julialang.org/u/shipengcheng1230)\
**Replies:** 0\
**Last updated:** [June 1, 2020, 10:38pm UTC](https://discourse.julialang.org/t/how-do-i-use-multithreaded-blas-in-each-mpi-process/40587 "2020-06-01T22:38:25Z")

</div>

I would like to launch one MPI process on each node and perform multithreaded BLAS, the same as tested here, and discussed at combining-distributed-computing-multithreading. Specifically, I am trying to use Elemental.jl:…

---

## [Large-Scale HPC Project on Probabilistic Programming at Scale in Conjunction with Scientific Simulators](https://discourse.julialang.org/t/large-scale-hpc-project-on-probabilistic-programming-at-scale-in-conjunction-with-scientific-simulators/39416)

<div class="topic-metadata">

**Author:** [@ludgerpaehler](https://discourse.julialang.org/u/ludgerpaehler)\
**Replies:** 8\
**Last updated:** [May 31, 2020, 7:41pm UTC](https://discourse.julialang.org/t/large-scale-hpc-project-on-probabilistic-programming-at-scale-in-conjunction-with-scientific-simulators/39416 "2020-05-31T19:41:23Z")

</div>

Hi everyone, we recently got approval for our 250m CPU-hour GCS supercomputing project named “Bayesian Inference of the Reactive Shock-Bubble Interaction - Probabilistic Programming at Scale”. A rough sketch: The Baye…

---

## [Slowdown with multiple instances per node?](https://discourse.julialang.org/t/slowdown-with-multiple-instances-per-node/40274)

<div class="topic-metadata">

**Author:** [@sebastian-steiner](https://discourse.julialang.org/u/sebastian-steiner)\
**Replies:** 2\
**Last updated:** [May 27, 2020, 5:20pm UTC](https://discourse.julialang.org/t/slowdown-with-multiple-instances-per-node/40274 "2020-05-27T17:20:17Z")

</div>

I am currently comparing MPI performance between C and Julia and when I only have a single task per node, Julia’s performance is pretty much exactly the same as C’s. But as soon as I utilize all 32 cores on both sockets,…

---

## [How to set environment variable in machine-file?](https://discourse.julialang.org/t/how-to-set-environment-variable-in-machine-file/39909)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 1\
**Last updated:** [May 22, 2020, 5:21am UTC](https://discourse.julialang.org/t/how-to-set-environment-variable-in-machine-file/39909 "2020-05-22T05:21:00Z")

</div>

I have a Julia script which computes many linear programming problems using Gurobi+JuMP in parallel. When I test my code on HPC and submit the following slurm script: #!/bin/bash #SBATCH --job-name=xz49 #SBATCH --partit…

---

## [How to parallel Julia on multiple nodes on HPC (slurm)?](https://discourse.julialang.org/t/how-to-parallel-julia-on-multiple-nodes-on-hpc-slurm/39730)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 11\
**Last updated:** [May 20, 2020, 3:55am UTC](https://discourse.julialang.org/t/how-to-parallel-julia-on-multiple-nodes-on-hpc-slurm/39730 "2020-05-20T03:55:35Z")

</div>

The parallel part of my Julia script utilizes the Distributed.jl package and the remotecall\_fetch function. (I am not using MPI.) In short, it calculates many linear programming problems in parallel independently. (Here …

---

## [Issues using Stan.jl on a cluster to run embarrassingly parallelisable experiments](https://discourse.julialang.org/t/issues-using-stan-jl-on-a-cluster-to-run-embarrassingly-parallelisable-experiments/39529)

<div class="topic-metadata">

**Author:** [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Replies:** 0\
**Last updated:** [May 15, 2020, 3:01pm UTC](https://discourse.julialang.org/t/issues-using-stan-jl-on-a-cluster-to-run-embarrassingly-parallelisable-experiments/39529 "2020-05-15T15:01:07Z")

</div>

Hi, So I have a working experiment of some Stan code on my local machine. But when I do wrap the main part in a pmap with an interior where I load in a series of models, then do sampling and write evaluation metrics to …

---

## [Code that works fine locally causes an error on a cluster](https://discourse.julialang.org/t/code-that-works-fine-locally-causes-an-error-on-a-cluster/39452)

<div class="topic-metadata">

**Author:** [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Replies:** 3\
**Last updated:** [May 14, 2020, 10:54am UTC](https://discourse.julialang.org/t/code-that-works-fine-locally-causes-an-error-on-a-cluster/39452 "2020-05-14T10:54:06Z")

</div>

Hi, so I have a function in my code to calculate the ROC AUC following an experiment, this has worked fine for me and continues to work fine locally but on our cluster I get the following error: ERROR: LoadError: On wor…

---

## [Naming output files from processes spawned using addprocs\_slurm on a cluster](https://discourse.julialang.org/t/naming-output-files-from-processes-spawned-using-addprocs-slurm-on-a-cluster/39174)

<div class="topic-metadata">

**Author:** [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Replies:** 1\
**Last updated:** [May 12, 2020, 2:14pm UTC](https://discourse.julialang.org/t/naming-output-files-from-processes-spawned-using-addprocs-slurm-on-a-cluster/39174 "2020-05-12T14:14:00Z")

</div>

Hi, I am running some distributed code using ClusterManagers.jl’s addprocs\_slurm command to spawn processes after submitting an sbatch script on our cluster’s login node. I can happily rename the output file for the “hea…

---

## [Questions on parallel programming terminology](https://discourse.julialang.org/t/questions-on-parallel-programming-terminology/38996)

<div class="topic-metadata">

**Author:** [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Replies:** 7\
**Last updated:** [May 8, 2020, 10:20pm UTC](https://discourse.julialang.org/t/questions-on-parallel-programming-terminology/38996 "2020-05-08T22:20:26Z")

</div>

I am using this week to learn the parallel programming features of the language in more depth. I confess that I am a bit confused with the terminology used throughout the documentation, and would like to ask a few questi…

---

## [Does @asyc or Threads.@spawn affect performance](https://discourse.julialang.org/t/does-asyc-or-threads-spawn-affect-performance/39118)

<div class="topic-metadata">

**Author:** [@samuel\_okon](https://discourse.julialang.org/u/samuel_okon)\
**Replies:** 2\
**Last updated:** [May 8, 2020, 8:00pm UTC](https://discourse.julialang.org/t/does-asyc-or-threads-spawn-affect-performance/39118 "2020-05-08T20:00:00Z")

</div>

To my understanding @async creates a closure which may box a variable. And the docs say boxing affects performance. Or is there something am getting wrong. ? (Am aware of the interpolation operator in julia \>= 1.4).

---

## [Struggling to figure out how I should use shared arrays on a slurm cluster using remote workers](https://discourse.julialang.org/t/struggling-to-figure-out-how-i-should-use-shared-arrays-on-a-slurm-cluster-using-remote-workers/38970)

<div class="topic-metadata">

**Author:** [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Replies:** 11\
**Last updated:** [May 8, 2020, 10:22am UTC](https://discourse.julialang.org/t/struggling-to-figure-out-how-i-should-use-shared-arrays-on-a-slurm-cluster-using-remote-workers/38970 "2020-05-08T10:22:11Z")

</div>

Hi so I am using a large distributed map function like so: using ClusterManagers using Distributed addprocs\_slurm(parse(Int, ENV\["SLURM\_NTASKS"\])) ... SETUP EXPERIMENT results = SharedArray{Float64, 2}((total\_steps, 1…

---

## [Using a machine file on a cluster but also propagating environment to the remote workers](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948)

<div class="topic-metadata">

**Author:** [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Replies:** 15\
**Last updated:** [May 7, 2020, 3:49pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948 "2020-05-07T15:49:36Z")

</div>

Hi, I am trying to run some distributed code on a slurm cluster using an sbatch file as below: #!/bin/bash #SBATCH --nodes=1 #SBATCH --tasks-per-node=28 #SBATCH --mem-per-cpu=4571 #SBATCH --time=24:00:00 #SBATCH -o slur…

---

## [Propagation of available/assigned worker-IDs in hierarchical computations?](https://discourse.julialang.org/t/propagation-of-available-assigned-worker-ids-in-hierarchical-computations/38670)

<div class="topic-metadata">

**Author:** [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Replies:** 21\
**Last updated:** [May 7, 2020, 4:59am UTC](https://discourse.julialang.org/t/propagation-of-available-assigned-worker-ids-in-hierarchical-computations/38670 "2020-05-07T04:59:35Z")

</div>

I’ve been thinking about ways to propagate information about available workers (and possibly other resources) in scenarios with nested computations. Say we have 10000 workers available and want to run a high-level distri…

---

## [Slower @threads than serial for array computations](https://discourse.julialang.org/t/slower-threads-than-serial-for-array-computations/23522)

<div class="topic-metadata">

**Author:** [@dataDiver](https://discourse.julialang.org/u/dataDiver)\
**Replies:** 26\
**Last updated:** [May 7, 2020, 1:29am UTC](https://discourse.julialang.org/t/slower-threads-than-serial-for-array-computations/23522 "2020-05-07T01:29:17Z")

</div>

I am doing computations on an array using @threads in the Base.Threads package. I am getting VERY slow performance for the threaded version and when the matrix gets very large the threaded version is killed. Here is an e…

---

## [How to save parallel outputs to the same file on HPC cluster](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 6\
**Last updated:** [April 22, 2020, 10:40pm UTC](https://discourse.julialang.org/t/how-to-save-parallel-outputs-to-the-same-file-on-hpc-cluster/37993 "2020-04-22T22:40:10Z")

</div>

I am doing Monte Carlo study on HPC cluster. I parallel the 100 Monte Carlo replications using job array in Slurm. I want to save the results of each replication to the same file. Let’s say for replication ID n, I have t…

---

## [Two level distributed / parallel execution](https://discourse.julialang.org/t/two-level-distributed-parallel-execution/21292)

<div class="topic-metadata">

**Author:** [@orenbenkiki](https://discourse.julialang.org/u/orenbenkiki)\
**Replies:** 4\
**Last updated:** [April 22, 2020, 6:57am UTC](https://discourse.julialang.org/t/two-level-distributed-parallel-execution/21292 "2020-04-22T06:57:34Z")

</div>

Given a complex, large scale computation that spans a non trivial amount of data (Tens of GBs), I wish to split the work between multiple processes across several hosts in a cluster. Each host has dozens of processors an…

---

## [Addprocs() fails with IOError on SLURM](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951)

<div class="topic-metadata">

**Author:** [@dkiese](https://discourse.julialang.org/u/dkiese)\
**Replies:** 2\
**Last updated:** [April 21, 2020, 11:19am UTC](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951 "2020-04-21T11:19:57Z")

</div>

Hello everyone, I am currently trying to run hybrid shared-distributed memory code on a large computing cluster, which is configured with SLURM. My code is roughly structured like this using Distributed using ClusterMa…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=7)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=9)