# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=4

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 5

---

## [Edited video of course “GPU Programming with Julia” at the Swiss National Supercomputing Centre (CSCS) now available](https://discourse.julialang.org/t/edited-video-of-course-gpu-programming-with-julia-at-the-swiss-national-supercomputing-centre-cscs-now-available/80293)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 1\
**Last updated:** [May 1, 2022, 6:46pm UTC](https://discourse.julialang.org/t/edited-video-of-course-gpu-programming-with-julia-at-the-swiss-national-supercomputing-centre-cscs-now-available/80293 "2022-05-01T18:46:29Z")

</div>

Edited video of course “GPU Programming with Julia” at the Swiss National Supercomputing Centre (CSCS) now available!

---

## [Using pmap: Do all workers have to be equally capable?](https://discourse.julialang.org/t/using-pmap-do-all-workers-have-to-be-equally-capable/74122)

<div class="topic-metadata">

**Author:** [@chelseas](https://discourse.julialang.org/u/chelseas)\
**Replies:** 5\
**Last updated:** [April 29, 2022, 12:18pm UTC](https://discourse.julialang.org/t/using-pmap-do-all-workers-have-to-be-equally-capable/74122 "2022-04-29T12:18:32Z")

</div>

I am using pmap with workers from many computers. Some of those computers have more resources than others ~ does pmap know this? And distribute tasks accordingly? e.g. if I have 10 workers on one machine vs 2 workers on …

---

## [Introducing Julius: low code and scalable graph computing in Julia](https://discourse.julialang.org/t/introducing-julius-low-code-and-scalable-graph-computing-in-julia/80129)

<div class="topic-metadata">

**Author:** [@Yadong\_Li](https://discourse.julialang.org/u/Yadong_Li)\
**Replies:** 2\
**Last updated:** [April 27, 2022, 5:34pm UTC](https://discourse.julialang.org/t/introducing-julius-low-code-and-scalable-graph-computing-in-julia/80129 "2022-04-27T17:34:59Z")

</div>

We are a group of researchers and engineers passionate about graph computing. At Julius Technologies, we have built a truly innovative graph computing solution that allows developers to build sophisticated, scalable and …

---

## [How can I improve this parallel routine on mutually independent tasks?](https://discourse.julialang.org/t/how-can-i-improve-this-parallel-routine-on-mutually-independent-tasks/79669)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 1\
**Last updated:** [April 19, 2022, 6:04am UTC](https://discourse.julialang.org/t/how-can-i-improve-this-parallel-routine-on-mutually-independent-tasks/79669 "2022-04-19T06:04:23Z")

</div>

I intend to solve N mutually independent tasks in parallel and each task is solved on a single CPU. I find an example code online and revised it as in the following MWE. I was wondering: Why there is a for loop over a…

---

## [Strange multithreaded behaviour on ARM Ampere A1 (Oracle Cloud)](https://discourse.julialang.org/t/strange-multithreaded-behaviour-on-arm-ampere-a1-oracle-cloud/79490)

<div class="topic-metadata">

**Author:** [@Dandan](https://discourse.julialang.org/u/Dandan)\
**Replies:** 10\
**Last updated:** [April 14, 2022, 6:56pm UTC](https://discourse.julialang.org/t/strange-multithreaded-behaviour-on-arm-ampere-a1-oracle-cloud/79490 "2022-04-14T18:56:17Z")

</div>

I’ve tried to run the following simple benchmark in Oracle Cloud VM on an instance with 4 vCPU of ARM Ampere A1 and 24GB ram: using LinearAlgebra function foo() Threads.@threads for i in 1:10000 H = rand(100, 100) +…

---

## [Mapping Hwloc logical index to OS index](https://discourse.julialang.org/t/mapping-hwloc-logical-index-to-os-index/79377)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 5\
**Last updated:** [April 12, 2022, 3:12pm UTC](https://discourse.julialang.org/t/mapping-hwloc-logical-index-to-os-index/79377 "2022-04-12T15:12:29Z")

</div>

The docs for Hwloc library state that the logical indices of Processor units used by Hwloc are not the same as the indices used by OS. How does one map Hwloc logical indices to OS indices? Is this functionality exposed…

---

## [The purpose of ThreadingUtilities.jl and Polyester.jl](https://discourse.julialang.org/t/the-purpose-of-threadingutilities-jl-and-polyester-jl/79226)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 9\
**Last updated:** [April 10, 2022, 2:12pm UTC](https://discourse.julialang.org/t/the-purpose-of-threadingutilities-jl-and-polyester-jl/79226 "2022-04-10T14:12:34Z")

</div>

As far as I understand, both ThreadingUtilities.jl and Polyster.jl provide the threads with low overhead. Could someone (@Elrod) elaborate the precise difference between the packages and when one should be preferred ove…

---

## [Error when I use the function distribute(array) from the libray DistributedArrays](https://discourse.julialang.org/t/error-when-i-use-the-function-distribute-array-from-the-libray-distributedarrays/16015)

<div class="topic-metadata">

**Author:** [@dsHitman](https://discourse.julialang.org/u/dsHitman)\
**Replies:** 3\
**Last updated:** [April 7, 2022, 9:57pm UTC](https://discourse.julialang.org/t/error-when-i-use-the-function-distribute-array-from-the-libray-distributedarrays/16015 "2022-04-07T21:57:05Z")

</div>

When I create an array and after I use the distribute() function from the Distributed Arrays library, I get this error: ERROR: On worker 2: KeyError: key DistributedArrays \[aaf54ef3-cdf8-58ed-94cc-d582ad619b94\] not fou…

---

## [ParallelStencil w/MPI](https://discourse.julialang.org/t/parallelstencil-w-mpi/79077)

<div class="topic-metadata">

**Author:** [@wkharold](https://discourse.julialang.org/u/wkharold)\
**Replies:** 2\
**Last updated:** [April 6, 2022, 9:41am UTC](https://discourse.julialang.org/t/parallelstencil-w-mpi/79077 "2022-04-06T09:41:47Z")

</div>

Is the CPU parallelism in the ParallelStencil.jl Concise single/multi-XPU miniapps limited to threads, or since it’s built on/with ImplicitGlobalGrid can it do MPI as well? I don’t see anything that looks like MPI suppo…

---

## [Options for large-scale dense linear algebra](https://discourse.julialang.org/t/options-for-large-scale-dense-linear-algebra/78227)

<div class="topic-metadata">

**Author:** [@wcwitt](https://discourse.julialang.org/u/wcwitt)\
**Replies:** 1\
**Last updated:** [March 22, 2022, 10:18am UTC](https://discourse.julialang.org/t/options-for-large-scale-dense-linear-algebra/78227 "2022-03-22T10:18:06Z")

</div>

My team is trying to identify options for dense-matrix distributed linear algebra on ~100s-1000s of CPU cores spread over multiple nodes. We are hoping for at least QR and Cholesky factorization, as well as linear equati…

---

## [How to debug remotecall\_wait/deserialization failures?](https://discourse.julialang.org/t/how-to-debug-remotecall-wait-deserialization-failures/77864)

<div class="topic-metadata">

**Author:** [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Replies:** 3\
**Last updated:** [March 15, 2022, 6:48pm UTC](https://discourse.julialang.org/t/how-to-debug-remotecall-wait-deserialization-failures/77864 "2022-03-15T18:48:00Z")

</div>

When running code on 450 cores / 29 nodes in a Slurm environment (using the handy SlurmClusterManager.jl), sometimes everything works fine, sometimes I get failures on 1-7 workers. How do I investigate/debug/fix this? M…

---

## [Distributed Julia in the cloud](https://discourse.julialang.org/t/distributed-julia-in-the-cloud/77627)

<div class="topic-metadata">

**Author:** [@niczky12](https://discourse.julialang.org/u/niczky12)\
**Replies:** 18\
**Last updated:** [March 10, 2022, 4:06pm UTC](https://discourse.julialang.org/t/distributed-julia-in-the-cloud/77627 "2022-03-10T16:06:13Z")

</div>

I know that JuliaPro is doing a good job of running Julia across a cluster in Azure. However, I have not seen an easy to follow guide to spin up a cluster in AWS/GCP/Azure without the paid version. I am really fed up wi…

---

## [MPI.jl test\_io\_shared failing](https://discourse.julialang.org/t/mpi-jl-test-io-shared-failing/77530)

<div class="topic-metadata">

**Author:** [@RonRahaman](https://discourse.julialang.org/u/RonRahaman)\
**Replies:** 2\
**Last updated:** [March 9, 2022, 8:02pm UTC](https://discourse.julialang.org/t/mpi-jl-test-io-shared-failing/77530 "2022-03-09T20:02:19Z")

</div>

Hi all, I’m testing my MPI.jl build, which has been built with the cluster’s MPI library (OpenMPI 3.1.6 compiled with GCC 8.3.0). The tests (\]test MPI) succeed except for test\_io\_shared. The error message is below. I …

---

## [Introducing graph computing and programming in Julia](https://discourse.julialang.org/t/introducing-graph-computing-and-programming-in-julia/77629)

<div class="topic-metadata">

**Author:** [@Yadong\_Li](https://discourse.julialang.org/u/Yadong_Li)\
**Replies:** 0\
**Last updated:** [March 9, 2022, 11:07am UTC](https://discourse.julialang.org/t/introducing-graph-computing-and-programming-in-julia/77629 "2022-03-09T11:07:03Z")

</div>

Dear Fellow Julia Developers: I want to share the following introductory blog on graph computing and its enormous potential. Julia is the perfect language for building generic, transparent and scalable graph computin…

---

## [Addprocs behavior changed in recent update -- have to enter password now](https://discourse.julialang.org/t/addprocs-behavior-changed-in-recent-update-have-to-enter-password-now/77537)

<div class="topic-metadata">

**Author:** [@chelseas](https://discourse.julialang.org/u/chelseas)\
**Replies:** 2\
**Last updated:** [March 8, 2022, 9:46pm UTC](https://discourse.julialang.org/t/addprocs-behavior-changed-in-recent-update-have-to-enter-password-now/77537 "2022-03-08T21:46:59Z")

</div>

I have passwordless ssh login setup to my server like: ssh csidrane@myserver But then when I go into my terminal and I type: using Distributed addprocs(\[(“myserver”, 1)\]) it used to work just fine! About a month ago…

---

## [MPI on Raspberry pi distributed computer](https://discourse.julialang.org/t/mpi-on-raspberry-pi-distributed-computer/77588)

<div class="topic-metadata">

**Author:** [@Isonoglu](https://discourse.julialang.org/u/Isonoglu)\
**Replies:** 1\
**Last updated:** [March 8, 2022, 3:28pm UTC](https://discourse.julialang.org/t/mpi-on-raspberry-pi-distributed-computer/77588 "2022-03-08T15:28:51Z")

</div>

Hello everyone. I’m a total beginner in Julia and cluster computing. I have already made a Raspberry pi distributed computer with 4 nodes. Any node can SSH to every other. Every node should perform some calculations “loc…

---

## [How to set up / use a Kubernetes-cluster for distributed-computing?](https://discourse.julialang.org/t/how-to-set-up-use-a-kubernetes-cluster-for-distributed-computing/76615)

<div class="topic-metadata">

**Author:** [@tim-hilt](https://discourse.julialang.org/u/tim-hilt)\
**Replies:** 2\
**Last updated:** [February 17, 2022, 9:18am UTC](https://discourse.julialang.org/t/how-to-set-up-use-a-kubernetes-cluster-for-distributed-computing/76615 "2022-02-17T09:18:29Z")

</div>

I want to use the nodes of a Kubernetes-cluster to act as remote workers for my Julia script. Coming from a workplace, where we mostly develop enterprise-software, I thought that virtually every cluster is based on Kube…

---

## [Getting the length of remote channels](https://discourse.julialang.org/t/getting-the-length-of-remote-channels/76554)

<div class="topic-metadata">

**Author:** [@RogerP](https://discourse.julialang.org/u/RogerP)\
**Replies:** 0\
**Last updated:** [February 16, 2022, 12:14pm UTC](https://discourse.julialang.org/t/getting-the-length-of-remote-channels/76554 "2022-02-16T12:14:53Z")

</div>

A somewhat similar question to this was raised here Channel Monitoring Statistics. I want to use remote channels as if they were regular buffers, e.g. Circular Buffers from DataStructures.jl, however those are difficult …

---

## [Excessive memory usage parallel Julia job](https://discourse.julialang.org/t/excessive-memory-usage-parallel-julia-job/75256)

<div class="topic-metadata">

**Author:** [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Replies:** 6\
**Last updated:** [February 15, 2022, 2:05pm UTC](https://discourse.julialang.org/t/excessive-memory-usage-parallel-julia-job/75256 "2022-02-15T14:05:52Z")

</div>

I have ported the dynamical core of our C++ CFD code to Julia (https://github.com/Chiil/MicroHH.jl) and have been implementing MPI.jl over the past days. I ran my first few large test jobs (1024 cores on 8 EPYC nodes wit…

---

## [Migrating from SharedArrays to DistributedArrays](https://discourse.julialang.org/t/migrating-from-sharedarrays-to-distributedarrays/76376)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 0\
**Last updated:** [February 13, 2022, 9:52pm UTC](https://discourse.julialang.org/t/migrating-from-sharedarrays-to-distributedarrays/76376 "2022-02-13T21:52:26Z")

</div>

I have some code that currently works in a shared memory environment with SharedArrays structures that I want to migrate to using DistributedArrays for a multi-node/distributed memory machine. Roughly, the key part of t…

---

## [Distributed Computing with Slurm and Julia](https://discourse.julialang.org/t/distributed-computing-with-slurm-and-julia/76181)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 9\
**Last updated:** [February 10, 2022, 7:25pm UTC](https://discourse.julialang.org/t/distributed-computing-with-slurm-and-julia/76181 "2022-02-10T19:25:01Z")

</div>

I’m looking for some guidance on how to get started with the following scenario. I have a cluster with multiple nodes, each node has 48 cores. The job scheduling is done with slurm. Currently, I’ve been running single…

---

## [Poor performance of garbage collection in multi-threaded application](https://discourse.julialang.org/t/poor-performance-of-garbage-collection-in-multi-threaded-application/75493)

<div class="topic-metadata">

**Author:** [@racinmat](https://discourse.julialang.org/u/racinmat)\
**Replies:** 22\
**Last updated:** [February 3, 2022, 11:35pm UTC](https://discourse.julialang.org/t/poor-performance-of-garbage-collection-in-multi-threaded-application/75493 "2022-02-03T23:35:15Z")

</div>

Hi, I have a tricky problem bothering me: I am using julia 1.5 with HTTP.jl and Mux.jl, I added multi-threading support in a manner as MusicAlbums.jl/Workers.jl at master · quinnj/MusicAlbums.jl (github.com), but compa…

---

## [Differences between @spawnat, remotecall\_wait, remotecall\_fetch](https://discourse.julialang.org/t/differences-between-spawnat-remotecall-wait-remotecall-fetch/75649)

<div class="topic-metadata">

**Author:** [@shce](https://discourse.julialang.org/u/shce)\
**Replies:** 0\
**Last updated:** [February 2, 2022, 12:46pm UTC](https://discourse.julialang.org/t/differences-between-spawnat-remotecall-wait-remotecall-fetch/75649 "2022-02-02T12:46:53Z")

</div>

Hello! I was wondering what is the difference between @spawn, remotecall\_wait and remotecall\_fetch. For instance, consider the following MWE: using Distributed @everywhere using DistributedArrays @everywhere function …

---

## [Add package on remote worker](https://discourse.julialang.org/t/add-package-on-remote-worker/75489)

<div class="topic-metadata">

**Author:** [@guyvdb](https://discourse.julialang.org/u/guyvdb)\
**Replies:** 1\
**Last updated:** [January 31, 2022, 6:50am UTC](https://discourse.julialang.org/t/add-package-on-remote-worker/75489 "2022-01-31T06:50:27Z")

</div>

I am trying to install packages on a Distributed worker, but get errors during precompilation that don’t occur when running the same commands on the server manually. using Distributed pids = addprocs(\["server.edu"\]) rem…

---

## [HDF5 collective read using MPI goes out of memory, but write works perfectly](https://discourse.julialang.org/t/hdf5-collective-read-using-mpi-goes-out-of-memory-but-write-works-perfectly/75158)

<div class="topic-metadata">

**Author:** [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Replies:** 9\
**Last updated:** [January 26, 2022, 10:23am UTC](https://discourse.julialang.org/t/hdf5-collective-read-using-mpi-goes-out-of-memory-but-write-works-perfectly/75158 "2022-01-26T10:23:51Z")

</div>

I have a test script that uses parallel HDF5 for reading and writing. I am testing this on a node of our cluster with ample memory available, yet the read goes out of memory. Interestingly, the write allocates only ~200…

---

## [CUDA aware MPI works on system but not for Julia](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060)

<div class="topic-metadata">

**Author:** [@Chiil](https://discourse.julialang.org/u/Chiil)\
**Replies:** 30\
**Last updated:** [January 24, 2022, 10:15pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-works-on-system-but-not-for-julia/75060 "2022-01-24T22:15:10Z")

</div>

I have problems getting CUDA-aware MPI running in Julia. I have a C++ example that works perfectly if I run it with mpiexec -n 2 ./alltoall\_test. The equivalent Julia code run with mpiexec -n 2 julia alltoall\_test.jl fai…

---

## [Ssh cluster julia host address issue](https://discourse.julialang.org/t/ssh-cluster-julia-host-address-issue/63814)

<div class="topic-metadata">

**Author:** [@grandemundo82](https://discourse.julialang.org/u/grandemundo82)\
**Replies:** 7\
**Last updated:** [January 19, 2022, 10:09pm UTC](https://discourse.julialang.org/t/ssh-cluster-julia-host-address-issue/63814 "2022-01-19T22:09:06Z")

</div>

I am getting some error adding processors to a university cluster. When I ssh to the cluster and start Julia, when I run using Distributed addprocs(\[("username@compute-6-300",1)\], tunnel=true, max\_parallel=1, exename="…

---

## [Error while parallelizing](https://discourse.julialang.org/t/error-while-parallelizing/74049)

<div class="topic-metadata">

**Author:** [@Sumit\_Malbari](https://discourse.julialang.org/u/Sumit_Malbari)\
**Replies:** 0\
**Last updated:** [January 4, 2022, 5:43pm UTC](https://discourse.julialang.org/t/error-while-parallelizing/74049 "2022-01-04T17:43:30Z")

</div>

Hi All, I ran JULIA\_COPY\_STACKS=yes julia -e ‘using Pkg;Pkg.add(Pkg.PackageSpec(;name=“Spark”, version=“0.5.1”));Pkg.build(“Spark”);using Spark;Spark.init();sc = SparkContext(master=“yarn”);’ all this worked fine. C…

---

## [Setting up Julia on Spark on AWS EMR](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781)

<div class="topic-metadata">

**Author:** [@Sumit\_Malbari](https://discourse.julialang.org/u/Sumit_Malbari)\
**Replies:** 24\
**Last updated:** [January 4, 2022, 3:13pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781 "2022-01-04T15:13:50Z")

</div>

Hi folks, I want to use Julia on Spark on AWS EMR as this is one of the requirement at my workplace. I tried to follow the steps from Julia & Spark but I was not able to succeed. If anyone has implemented it on AWS EM…

---

## [Package install on (offline) Cluster - Build needs internet](https://discourse.julialang.org/t/package-install-on-offline-cluster-build-needs-internet/19862)

<div class="topic-metadata">

**Author:** [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Replies:** 7\
**Last updated:** [December 28, 2021, 7:25am UTC](https://discourse.julialang.org/t/package-install-on-offline-cluster-build-needs-internet/19862 "2021-12-28T07:25:10Z")

</div>

Hey everyone, I’m trying to use julia 1.0 on our cluster. It is not connected to the internet (login via login-server) so I can’t use the standard procedure to install the packages I need. I read here that one option…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=3)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=5)