# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=6

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 7

---

## [The ultimate guide to distributed computing](https://discourse.julialang.org/t/the-ultimate-guide-to-distributed-computing/41867)

<div class="topic-metadata">

**Author:** [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Replies:** 44\
**Last updated:** [June 21, 2021, 11:02am UTC](https://discourse.julialang.org/t/the-ultimate-guide-to-distributed-computing/41867 "2021-06-21T11:02:41Z")

</div>

HOWTO: Distributed computing In this thread I would like to put together relevant information for people interested in distributing simple function calls on multiple cluster nodes. Although the task is simple, there are …

---

## [How to automatically determine number of cores provided by cluster?](https://discourse.julialang.org/t/how-to-automatically-determine-number-of-cores-provided-by-cluster/62544)

<div class="topic-metadata">

**Author:** [@DenisTitov](https://discourse.julialang.org/u/DenisTitov)\
**Replies:** 9\
**Last updated:** [June 18, 2021, 8:00pm UTC](https://discourse.julialang.org/t/how-to-automatically-determine-number-of-cores-provided-by-cluster/62544 "2021-06-18T20:00:19Z")

</div>

I often run parallel jobs on many nodes of HPC. My university cluster uses slurm and I use ClusterManagers.jl to submit jobs to many nodes. I love that it’s so easy to modify my code to run on 100s of cores! For examp…

---

## [When will Julia compete with Spark?](https://discourse.julialang.org/t/when-will-julia-compete-with-spark/25218)

<div class="topic-metadata">

**Author:** [@merlin](https://discourse.julialang.org/u/merlin)\
**Replies:** 16\
**Last updated:** [June 5, 2021, 5:09pm UTC](https://discourse.julialang.org/t/when-will-julia-compete-with-spark/25218 "2021-06-05T17:09:45Z")

</div>

My org uses Spark + EMR to query and transform static files on S3 and output to files that either get loaded to a DB or used as input for analysis. It seems like Julia could also do this and do it well, of course Spark …

---

## [Is AMD CPU more sensitive to false-sharing than Intel and Power?](https://discourse.julialang.org/t/is-amd-cpu-more-sensitive-to-false-sharing-than-intel-and-power/61679)

<div class="topic-metadata">

**Author:** [@tkf](https://discourse.julialang.org/u/tkf)\
**Replies:** 1\
**Last updated:** [May 23, 2021, 3:52pm UTC](https://discourse.julialang.org/t/is-amd-cpu-more-sensitive-to-false-sharing-than-intel-and-power/61679 "2021-05-23T15:52:50Z")

</div>

I tried to come up with a robust and minimal false sharing example (so that I can play with perf c2c). But it turned out the following example (from the previous discussion in Zulip) only shows the effect of false sharin…

---

## [Basic question on distributing modules](https://discourse.julialang.org/t/basic-question-on-distributing-modules/61542)

<div class="topic-metadata">

**Author:** [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Replies:** 2\
**Last updated:** [May 21, 2021, 12:38pm UTC](https://discourse.julialang.org/t/basic-question-on-distributing-modules/61542 "2021-05-21T12:38:14Z")

</div>

I am using Julia 1.6.1 with Distributed. Q: Must I first install a module on every node in a cluster to use it? Or is there a way to “push” a module to where it’s needed at runtime? Apparently, if my reading is any g…

---

## [Loading modules on multipole nodes](https://discourse.julialang.org/t/loading-modules-on-multipole-nodes/61569)

<div class="topic-metadata">

**Author:** [@Frisus95](https://discourse.julialang.org/u/Frisus95)\
**Replies:** 0\
**Last updated:** [May 21, 2021, 9:42am UTC](https://discourse.julialang.org/t/loading-modules-on-multipole-nodes/61569 "2021-05-21T09:42:27Z")

</div>

Good day everyone! As the title suggests, I have a question about how to run a parallel computation on multiple nodes in a cluster. My julia code calls an external library, written in C, and then parallelizes across mu…

---

## [Pkg.instantiate() takes a very long time on a remote worker](https://discourse.julialang.org/t/pkg-instantiate-takes-a-very-long-time-on-a-remote-worker/61498)

<div class="topic-metadata">

**Author:** [@gobs](https://discourse.julialang.org/u/gobs)\
**Replies:** 2\
**Last updated:** [May 20, 2021, 3:56pm UTC](https://discourse.julialang.org/t/pkg-instantiate-takes-a-very-long-time-on-a-remote-worker/61498 "2021-05-20T15:56:33Z")

</div>

I’m trying to use multiple nodes for a parallel algorithm, but currently Pkg.instantiate() takes far too long, as in 20+ minutes, so long that my 30 minute job timed out. Is there a way to speed this up? Is this normal?

---

## [Setting up distributed workers on seperate nodes of cluster - PBS and OpenMPI](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013)

<div class="topic-metadata">

**Author:** [@gobs](https://discourse.julialang.org/u/gobs)\
**Replies:** 10\
**Last updated:** [May 19, 2021, 8:38pm UTC](https://discourse.julialang.org/t/setting-up-distributed-workers-on-seperate-nodes-of-cluster-pbs-and-openmpi/61013 "2021-05-19T20:38:40Z")

</div>

I’m quite new to distributed computing and high performance computing so please be kind. My university has access to an HPC which uses PBS/Torque. I’m currently running the ProgressiveHedging.jl package using all the co…

---

## [Occasional SharedArray errors on slurm](https://discourse.julialang.org/t/occasional-sharedarray-errors-on-slurm/61385)

<div class="topic-metadata">

**Author:** [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Replies:** 1\
**Last updated:** [May 18, 2021, 9:42pm UTC](https://discourse.julialang.org/t/occasional-sharedarray-errors-on-slurm/61385 "2021-05-18T21:42:52Z")

</div>

I’m having trouble \[minimally\] reproducing this locally. It happens for maybe 0.1-5% of array tasks which execute initShared() from unregistered package pqADMM and causes the slurm error shown below for that respective …

---

## [Cannon's or SUMA algorithm implementation?](https://discourse.julialang.org/t/cannons-or-suma-algorithm-implementation/61034)

<div class="topic-metadata">

**Author:** [@Chase\_Roberts](https://discourse.julialang.org/u/Chase_Roberts)\
**Replies:** 0\
**Last updated:** [May 12, 2021, 5:22pm UTC](https://discourse.julialang.org/t/cannons-or-suma-algorithm-implementation/61034 "2021-05-12T17:22:57Z")

</div>

Are there any examples of doing distributed matmuls with Cannon’s or SUMA algorithm in Julia?

---

## [Controlling thread affinity for distributed multithreading](https://discourse.julialang.org/t/controlling-thread-affinity-for-distributed-multithreading/53869)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 3\
**Last updated:** [May 12, 2021, 12:44pm UTC](https://discourse.julialang.org/t/controlling-thread-affinity-for-distributed-multithreading/53869 "2021-05-12T12:44:49Z")

</div>

I have a question regarding the setup of hybrid parallelism which combines multithreading and distributed computations. Let’s say I want to run some computation on a machine with Non-Uniform Memory Access. In this case,…

---

## [Scattering and gathering arrays](https://discourse.julialang.org/t/scattering-and-gathering-arrays/60345)

<div class="topic-metadata">

**Author:** [@francispoulin](https://discourse.julialang.org/u/francispoulin)\
**Replies:** 5\
**Last updated:** [May 3, 2021, 1:27pm UTC](https://discourse.julialang.org/t/scattering-and-gathering-arrays/60345 "2021-05-03T13:27:34Z")

</div>

I am trying to translate a python script that scatters aand gathers 2D arrays, as can be found here. python - Along what axis does mpi4py Scatterv function split a numpy array? - Stack Overflow I have managed to get som…

---

## [Simple Parallel Examples for Embarrassingly Simple Problems](https://discourse.julialang.org/t/simple-parallel-examples-for-embarrassingly-simple-problems/8788)

<div class="topic-metadata">

**Author:** [@ivo\_welch](https://discourse.julialang.org/u/ivo_welch)\
**Replies:** 29\
**Last updated:** [April 23, 2021, 3:45pm UTC](https://discourse.julialang.org/t/simple-parallel-examples-for-embarrassingly-simple-problems/8788 "2021-04-23T15:45:18Z")

</div>

macOS, julia 0.6.2, Feb 2018, captain’s log: I have been playing around with trying to understand (the docs on) parallelism. as an early example, I just want to create, in parallel, a list of function logp1(x::Int64):…

---

## [Setup topology with ClusterManagers](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 8\
**Last updated:** [April 19, 2021, 3:12pm UTC](https://discourse.julialang.org/t/setup-topology-with-clustermanagers/59591 "2021-04-19T15:12:48Z")

</div>

How does one pass topology to a cluster manager? Julia docs mention the setup of topology in the case when addprocs dispatches on machines, but not on a cluster manager. I am trying to do the following thing. I would …

---

## [Parallel computing for eigs() function](https://discourse.julialang.org/t/parallel-computing-for-eigs-function/22926)

<div class="topic-metadata">

**Author:** [@saeed](https://discourse.julialang.org/u/saeed)\
**Replies:** 10\
**Last updated:** [April 14, 2021, 12:45pm UTC](https://discourse.julialang.org/t/parallel-computing-for-eigs-function/22926 "2021-04-14T12:45:46Z")

</div>

Hello, I am new to julia and parallel computing. I want to use parallel computing capability in julia in doing shift-invert calculations. My problem is that I have a couple of sparse matrices, and want to find specific e…

---

## [LIKWID profiler](https://discourse.julialang.org/t/likwid-profiler/58543)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 1\
**Last updated:** [April 4, 2021, 8:15pm UTC](https://discourse.julialang.org/t/likwid-profiler/58543 "2021-04-04T20:15:29Z")

</div>

Definitely worth reading if you are interested in finding detail on the CPU or GPU system you are working with. If this belongs in Off Topic I will happily move it.

---

## [DArray in Dagger vs DistributedArrays](https://discourse.julialang.org/t/darray-in-dagger-vs-distributedarrays/58330)

<div class="topic-metadata">

**Author:** [@nucklass](https://discourse.julialang.org/u/nucklass)\
**Replies:** 0\
**Last updated:** [March 31, 2021, 9:23pm UTC](https://discourse.julialang.org/t/darray-in-dagger-vs-distributedarrays/58330 "2021-03-31T21:23:50Z")

</div>

Hi all, Was looking over some options for making a project of mine run in parallel and happened to notice both Dagger and DistributedArrays export a DArray type. Anyone have any insights as to the difference between the…

---

## [Singularity, ERROR: Unable to find compatible target in system image](https://discourse.julialang.org/t/singularity-error-unable-to-find-compatible-target-in-system-image/57619)

<div class="topic-metadata">

**Author:** [@Perrin\_Meyer](https://discourse.julialang.org/u/Perrin_Meyer)\
**Replies:** 3\
**Last updated:** [March 21, 2021, 12:00am UTC](https://discourse.julialang.org/t/singularity-error-unable-to-find-compatible-target-in-system-image/57619 "2021-03-21T00:00:55Z")

</div>

I’m trying to compile a Singularity container for running on a HPC cluster. My container works on my laptop (where I build it), but when I transfer it over to the cluster, I get a: ERROR: Unable to find compatible targ…

---

## [Distributed, AllReduce, and Distributed Training](https://discourse.julialang.org/t/distributed-allreduce-and-distributed-training/56110)

<div class="topic-metadata">

**Author:** [@asaxton](https://discourse.julialang.org/u/asaxton)\
**Replies:** 1\
**Last updated:** [March 18, 2021, 9:56am UTC](https://discourse.julialang.org/t/distributed-allreduce-and-distributed-training/56110 "2021-03-18T09:56:51Z")

</div>

Continuing the discussion from Julia Distributed, AllReduce, and Distributed Training: I’m working with Distributed Julia on multiple hosts and want to write some code that does distributed training with a machine learn…

---

## [How to run Julia on Cluster?](https://discourse.julialang.org/t/how-to-run-julia-on-cluster/56551)

<div class="topic-metadata">

**Author:** [@Shazman](https://discourse.julialang.org/u/Shazman)\
**Replies:** 11\
**Last updated:** [March 16, 2021, 1:34pm UTC](https://discourse.julialang.org/t/how-to-run-julia-on-cluster/56551 "2021-03-16T13:34:48Z")

</div>

Hi, I am trying to run Julia on the Cluster. the main problem for me is to write the job script to tell the cluster to run my Julia commands. the cluster is using Slurm. anyone have an idea how to do it?

---

## [Dependencies when doing distributed](https://discourse.julialang.org/t/dependencies-when-doing-distributed/56736)

<div class="topic-metadata">

**Author:** [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Replies:** 0\
**Last updated:** [March 8, 2021, 2:41pm UTC](https://discourse.julialang.org/t/dependencies-when-doing-distributed/56736 "2021-03-08T14:41:09Z")

</div>

I have a line of code like this: @everywhere include("encounters.jl") The file in question, encounters.jl has in it a line using AAUtils The module AAUtils is installed both locally and remotely. Q: From where is AA…

---

## [Is there a way to lock the version of python packages](https://discourse.julialang.org/t/is-there-a-way-to-lock-the-version-of-python-packages/35784)

<div class="topic-metadata">

**Author:** [@Shih\_Shen](https://discourse.julialang.org/u/Shih_Shen)\
**Replies:** 5\
**Last updated:** [March 1, 2021, 6:42am UTC](https://discourse.julialang.org/t/is-there-a-way-to-lock-the-version-of-python-packages/35784 "2021-03-01T06:42:30Z")

</div>

Julia has provided a way to lock the version of Julia packages which is great. However, I found I still need python packages to finish my works sometimes. And in order to use Julia in production, locking the version of d…

---

## [Simplest way to convert a program for parallel (multithreaded) runs on multiple servers/cores?](https://discourse.julialang.org/t/simplest-way-to-convert-a-program-for-parallel-multithreaded-runs-on-multiple-servers-cores/55363)

<div class="topic-metadata">

**Author:** [@CosmoProf](https://discourse.julialang.org/u/CosmoProf)\
**Replies:** 26\
**Last updated:** [February 22, 2021, 2:00am UTC](https://discourse.julialang.org/t/simplest-way-to-convert-a-program-for-parallel-multithreaded-runs-on-multiple-servers-cores/55363 "2021-02-22T02:00:25Z")

</div>

Hi again all, I am running a collection of programs doing intensive numerical calculations (e.g., ~1-2 days of runtime on a system with 64 GB of memory and CPU \> 2.1 GHz). I have an opportunity to save tremendous resour…

---

## [Nice way to do parallel matrix multiplication](https://discourse.julialang.org/t/nice-way-to-do-parallel-matrix-multiplication/55310)

<div class="topic-metadata">

**Author:** [@Ribeiro](https://discourse.julialang.org/u/Ribeiro)\
**Replies:** 0\
**Last updated:** [February 15, 2021, 12:14pm UTC](https://discourse.julialang.org/t/nice-way-to-do-parallel-matrix-multiplication/55310 "2021-02-15T12:14:18Z")

</div>

Hello, I’m trying to do parallel matrix multiplication in a for loop. I’m using Threads.@threads, but this also applies to a GPU version of the code and maybe a distributed (MPI-like) version in the future. Within my c…

---

## [Checkpointing with Julia](https://discourse.julialang.org/t/checkpointing-with-julia/24026)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 8\
**Last updated:** [February 15, 2021, 7:50am UTC](https://discourse.julialang.org/t/checkpointing-with-julia/24026 "2021-02-15T07:50:59Z")

</div>

The post today by @Xing\_Shi\_Cai leads me to ask - has anyone worked with checkpointing in Julia? A bit of Googling would suggest DMTCP would work http://dmtcp.sourceforge.net/ I guess the real answer is - why don’t I…

---

## [Pmap using fewer workers than expected after some time](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715)

<div class="topic-metadata">

**Author:** [@Luapulu](https://discourse.julialang.org/u/Luapulu)\
**Replies:** 5\
**Last updated:** [February 10, 2021, 12:29pm UTC](https://discourse.julialang.org/t/pmap-using-fewer-workers-than-expected-after-some-time/46715 "2021-02-10T12:29:31Z")

</div>

I’m running pmap with 16 workers and batch\_size=1 to process hundreds of files. Each file takes over an hour to work through and the files are relatively homogeneous but don’t take exactly the same time. For the first 14…

---

## [Very simple MPI implementation question](https://discourse.julialang.org/t/very-simple-mpi-implementation-question/53641)

<div class="topic-metadata">

**Author:** [@abader](https://discourse.julialang.org/u/abader)\
**Replies:** 3\
**Last updated:** [February 7, 2021, 11:20am UTC](https://discourse.julialang.org/t/very-simple-mpi-implementation-question/53641 "2021-02-07T11:20:34Z")

</div>

I’m trying to do some simple MPI calculations but have had some difficulty finding good tutorials online. For this implementation we cannot use threading, because of some interface issues with other codes (most advice I…

---

## [How to close a remote Timer?](https://discourse.julialang.org/t/how-to-close-a-remote-timer/53295)

<div class="topic-metadata">

**Author:** [@Christian\_Dengler](https://discourse.julialang.org/u/Christian_Dengler)\
**Replies:** 3\
**Last updated:** [January 14, 2021, 7:47am UTC](https://discourse.julialang.org/t/how-to-close-a-remote-timer/53295 "2021-01-14T07:47:27Z")

</div>

I want to run mutliple procedures in parallel on a server in fixed time ntervals. The idea was to ,e.g., simply use “@spawnat :any” together with “Timer”, however, I cannot find a way to stop the Timer once its started. …

---

## [Sending functions within structs to workers leads to world age problem](https://discourse.julialang.org/t/sending-functions-within-structs-to-workers-leads-to-world-age-problem/52717)

<div class="topic-metadata">

**Author:** [@vancleve](https://discourse.julialang.org/u/vancleve)\
**Replies:** 4\
**Last updated:** [January 4, 2021, 3:34am UTC](https://discourse.julialang.org/t/sending-functions-within-structs-to-workers-leads-to-world-age-problem/52717 "2021-01-04T03:34:25Z")

</div>

I have a simulation where I pass functions as parameters by putting them in a parameter object. However, when I try to run the simulation using Distributed, the functions do not run in the workers due to their world age …

---

## [Basic remote procedure call via ssh from Linux to Windows](https://discourse.julialang.org/t/basic-remote-procedure-call-via-ssh-from-linux-to-windows/18743)

<div class="topic-metadata">

**Author:** [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Replies:** 8\
**Last updated:** [December 29, 2020, 12:52pm UTC](https://discourse.julialang.org/t/basic-remote-procedure-call-via-ssh-from-linux-to-windows/18743 "2020-12-29T12:52:48Z")

</div>

I have a Julia application running under Linux. I would like to make a couple of remote function calls to another Julia process running on a specific Windows machine, because that Windows machine has special data-acquisi…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=5)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=7)