# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=7

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 8

---

## [Julia crashes inside @threads with MPI](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400)

<div class="topic-metadata">

**Author:** [@Jae-Mo\_Lihm](https://discourse.julialang.org/u/Jae-Mo_Lihm)\
**Replies:** 5\
**Last updated:** [December 29, 2020, 7:41am UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400 "2020-12-29T07:41:52Z")

</div>

Hi, I am trying to implement a program with hybrid MPI + Threads parallelization. In the following example, if I comment out all MPI-related lines, the code runs without any problem. However, if I run the code with the …

---

## [BoundsError with SharedArrays in remote node](https://discourse.julialang.org/t/boundserror-with-sharedarrays-in-remote-node/51670)

<div class="topic-metadata">

**Author:** [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Replies:** 1\
**Last updated:** [December 17, 2020, 1:28am UTC](https://discourse.julialang.org/t/boundserror-with-sharedarrays-in-remote-node/51670 "2020-12-17T01:28:34Z")

</div>

I want to broadcast a matrix into some remote nodes to avoid communication. My best guess is to use SharedArrays, but, when I try to copy the data on remote node, I got a error. Here an example: args = (exeflags=\`--thr…

---

## [Warning: macos 10.12.\* bug with SharedArray: shm\_open() failed](https://discourse.julialang.org/t/warning-macos-10-12-bug-with-sharedarray-shm-open-failed/10471)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 3\
**Last updated:** [December 13, 2020, 10:23pm UTC](https://discourse.julialang.org/t/warning-macos-10-12-bug-with-sharedarray-shm-open-failed/10471 "2020-12-13T22:23:38Z")

</div>

The following program triggers a bug in macos 10.12.\*, but not in 10.13.\* : (nprocs()\>1) || addprocs(1) function sharedarray\_parallel\_sum() sa = SharedArray{Float64}(5) @sync @parallel for i=1:5; sa\[i\] = timing…

---

## [Network application framework in Julia (i.e. netty, NIO, MINA)](https://discourse.julialang.org/t/network-application-framework-in-julia-i-e-netty-nio-mina/50889)

<div class="topic-metadata">

**Author:** [@mks](https://discourse.julialang.org/u/mks)\
**Replies:** 0\
**Last updated:** [November 28, 2020, 6:24am UTC](https://discourse.julialang.org/t/network-application-framework-in-julia-i-e-netty-nio-mina/50889 "2020-11-28T06:24:19Z")

</div>

Hello, is there work going on to come up with a network application framework in Julia like netty, NIO, MINA? https://netty.io/ It would be nice to be able to use gRPC and Protobuf. I know there is protobuf.jl which …

---

## [Printing Diagnostic Info in Parallel](https://discourse.julialang.org/t/printing-diagnostic-info-in-parallel/47787)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 1\
**Last updated:** [November 27, 2020, 12:03pm UTC](https://discourse.julialang.org/t/printing-diagnostic-info-in-parallel/47787 "2020-11-27T12:03:20Z")

</div>

If I am using Distributed, is there a way to print diagnostic information from the different workers? I’m thinking of something like this: @everywhere x = rand(); and I would like to be able to print what value of x e…

---

## [Parallel simulations on multiple cores on Azure](https://discourse.julialang.org/t/parallel-simulations-on-multiple-cores-on-azure/50680)

<div class="topic-metadata">

**Author:** [@maxpiot](https://discourse.julialang.org/u/maxpiot)\
**Replies:** 0\
**Last updated:** [November 24, 2020, 11:46am UTC](https://discourse.julialang.org/t/parallel-simulations-on-multiple-cores-on-azure/50680 "2020-11-24T11:46:11Z")

</div>

I have some fairly computationally intensive simulation code that I would like to run across multiple cores, (independently, only combining data at the end of a simulation). Currently I run this on my PC, using the pmap …

---

## [\`Using\` with Distributed](https://discourse.julialang.org/t/using-with-distributed/50633)

<div class="topic-metadata">

**Author:** [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Replies:** 1\
**Last updated:** [November 23, 2020, 5:38pm UTC](https://discourse.julialang.org/t/using-with-distributed/50633 "2020-11-23T17:38:44Z")

</div>

I noticed recently that when I try to do using for a package with worker processes, it appears to try loading it on all workers simultaneously, which creates some issues. When I removed the worker processes and ran the s…

---

## [How to query Slurm Array Task ID in Julia?](https://discourse.julialang.org/t/how-to-query-slurm-array-task-id-in-julia/36771)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 15\
**Last updated:** [November 19, 2020, 5:19am UTC](https://discourse.julialang.org/t/how-to-query-slurm-array-task-id-in-julia/36771 "2020-11-19T05:19:05Z")

</div>

I want to parallel 100 Monte Carlo study on 100 compute nodes. Within each node, it should execute the Julia script attached with a unique random seed (so that it will generate a unique fake dataset). Also, within each n…

---

## [Julia 1.5.2 Distributed seeking workaround](https://discourse.julialang.org/t/julia-1-5-2-distributed-seeking-workaround/49757)

<div class="topic-metadata">

**Author:** [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Replies:** 2\
**Last updated:** [November 8, 2020, 2:57pm UTC](https://discourse.julialang.org/t/julia-1-5-2-distributed-seeking-workaround/49757 "2020-11-08T14:57:25Z")

</div>

On macOS Julia 1.5.2. I am trying to run addprocs where the remote machine is linux. I get the following error: sh: 1: cd: can’t cd to /Users/ Apparently, something on the remote is trying to cd to the same path as my…

---

## [Julia libdl\_find\_library in cluster](https://discourse.julialang.org/t/julia-libdl-find-library-in-cluster/49700)

<div class="topic-metadata">

**Author:** [@j\_tanu](https://discourse.julialang.org/u/j_tanu)\
**Replies:** 15\
**Last updated:** [November 7, 2020, 11:48am UTC](https://discourse.julialang.org/t/julia-libdl-find-library-in-cluster/49700 "2020-11-07T11:48:11Z")

</div>

Hi I am trying to run the Julia in my university cluster In the cluster, Julia is not installed as a module so I am running the .jl file by giving the exact location of the Julia executable. As of now, My code was singl…

---

## [Error/segfault in basic test of CUDA-aware MPI](https://discourse.julialang.org/t/error-segfault-in-basic-test-of-cuda-aware-mpi/31755)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 10\
**Last updated:** [November 6, 2020, 3:35pm UTC](https://discourse.julialang.org/t/error-segfault-in-basic-test-of-cuda-aware-mpi/31755 "2020-11-06T15:35:59Z")

</div>

Problem A basic test of CUDA-aware MPI with MPI.jl fails on both our Cray supercomputer and on another cluster: 1) On Cray system 1.1) Test and error: omlins@dom101:~\> export MPICH\_RDMA\_ENABLED\_CUDA=1 omlins@nid00002:…

---

## [ANN: MPI.jl v0.10.0: new build process and CUDA-aware support](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601)

<div class="topic-metadata">

**Author:** [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Replies:** 26\
**Last updated:** [October 31, 2020, 5:16am UTC](https://discourse.julialang.org/t/ann-mpi-jl-v0-10-0-new-build-process-and-cuda-aware-support/27601 "2020-10-31T05:16:06Z")

</div>

I have just tagged a new version of MPI.jl. Though the user-facing interface is largely the same, there has been extensive work underneath to internally use the C API (instead of the Fortran one). As a result, the build …

---

## [\[ANN\] MPI.jl 0.14 changes](https://discourse.julialang.org/t/ann-mpi-jl-0-14-changes/39126)

<div class="topic-metadata">

**Author:** [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Replies:** 6\
**Last updated:** [October 27, 2020, 8:30pm UTC](https://discourse.julialang.org/t/ann-mpi-jl-0-14-changes/39126 "2020-10-27T20:30:30Z")

</div>

I’ve just tagged MPI.jl 0.14: the major change in this release is that it will now use a BinaryBuilder-provided MPI binary (Microsoft MPI on Windows, MPICH on other platforms). If you are running on a HPC system you wil…

---

## [Building Julia from source on old Cray XE/XK system](https://discourse.julialang.org/t/building-julia-from-source-on-old-cray-xe-xk-system/48899)

<div class="topic-metadata">

**Author:** [@asaxton](https://discourse.julialang.org/u/asaxton)\
**Replies:** 0\
**Last updated:** [October 23, 2020, 6:04pm UTC](https://discourse.julialang.org/t/building-julia-from-source-on-old-cray-xe-xk-system/48899 "2020-10-23T18:04:00Z")

</div>

Hi, as the title says, I’m building Julia from source on a Cray XE/XK system (Blue Waters). I keep getting compiler errors similar to "error: expected ')' before 'PRIx32'". They have to do with macros defined under intty…

---

## [Tools for cloud computing (AWS, Azure, Google Cloud) with Julia (known to me as of Oct 2020)](https://discourse.julialang.org/t/tools-for-cloud-computing-aws-azure-google-cloud-with-julia-known-to-me-as-of-oct-2020/48368)

<div class="topic-metadata">

**Author:** [@nickeubank](https://discourse.julialang.org/u/nickeubank)\
**Replies:** 0\
**Last updated:** [October 14, 2020, 2:26pm UTC](https://discourse.julialang.org/t/tools-for-cloud-computing-aws-azure-google-cloud-with-julia-known-to-me-as-of-oct-2020/48368 "2020-10-14T14:26:53Z")

</div>

Azure: Azure API interface in Julia: like Azure CLI, but in Julia. Best set of Azure tools I’ve found (thanks Chevron!): AzManagers (standup VMs and scaled clusters), AzSessions (manage authentication into Azure se…

---

## [HPC / Julia, MPI / big data](https://discourse.julialang.org/t/hpc-julia-mpi-big-data/48236)

<div class="topic-metadata">

**Author:** [@oddodaoddo](https://discourse.julialang.org/u/oddodaoddo)\
**Replies:** 15\
**Last updated:** [October 13, 2020, 1:37pm UTC](https://discourse.julialang.org/t/hpc-julia-mpi-big-data/48236 "2020-10-13T13:37:38Z")

</div>

Hello. Many years ago I was in HPC (as in university/govt supercomputers). We used to write code in C/pthreads, MPI, PVM etc. Then I left and went to the private sector and then big data came with Spark, Scala and that w…

---

## [When to use QRSHManager?](https://discourse.julialang.org/t/when-to-use-qrshmanager/47154)

<div class="topic-metadata">

**Author:** [@jzr](https://discourse.julialang.org/u/jzr)\
**Replies:** 1\
**Last updated:** [October 11, 2020, 1:20pm UTC](https://discourse.julialang.org/t/when-to-use-qrshmanager/47154 "2020-10-11T13:20:36Z")

</div>

@bjarthur mentioned that QRSHManager can increase reliability on some systems: Julia workers can frequently timeout waiting for the standard output files to appear. In this case, it’s better to use the QRSHManager, whi…

---

## [Parallelism - Understanding pmap and the batch\_size parameter](https://discourse.julialang.org/t/parallelism-understanding-pmap-and-the-batch-size-parameter/15604)

<div class="topic-metadata">

**Author:** [@mdsalerno](https://discourse.julialang.org/u/mdsalerno)\
**Replies:** 1\
**Last updated:** [October 8, 2020, 3:22pm UTC](https://discourse.julialang.org/t/parallelism-understanding-pmap-and-the-batch-size-parameter/15604 "2020-10-08T15:22:12Z")

</div>

I am running something that looks like the following: addprocs(Sys.CPU\_THREADS) pmap(x -\> f(x, A, B, C), args) I have 48 CPU cores and thus 48 workers after running addprocs. A, B, and C are DataFrames, the largest h…

---

## [Standing up cluster on Azure](https://discourse.julialang.org/t/standing-up-cluster-on-azure/47942)

<div class="topic-metadata">

**Author:** [@nickeubank](https://discourse.julialang.org/u/nickeubank)\
**Replies:** 1\
**Last updated:** [October 7, 2020, 8:08pm UTC](https://discourse.julialang.org/t/standing-up-cluster-on-azure/47942 "2020-10-07T20:08:48Z")

</div>

Hey All! My program is trying to migrate a lot of our teaching and research towards using Azure instead of our on-site cluster (for which ClusterManager.jl has been amazing), and I was wondering if anyone had experience…

---

## [Help for basic usage of @distributed for](https://discourse.julialang.org/t/help-for-basic-usage-of-distributed-for/31666)

<div class="topic-metadata">

**Author:** [@Massimiliano\_Comin](https://discourse.julialang.org/u/Massimiliano_Comin)\
**Replies:** 1\
**Last updated:** [October 1, 2020, 9:27am UTC](https://discourse.julialang.org/t/help-for-basic-usage-of-distributed-for/31666 "2020-10-01T09:27:12Z")

</div>

Hello, I have a computation that I want to perform several times for several realizations of disorder. For this I wrote a serial script doing only one computation, and a separate script that calls it and distributes the…

---

## [How to get more info on w ∈ workers()](https://discourse.julialang.org/t/how-to-get-more-info-on-w-workers/34290)

<div class="topic-metadata">

**Author:** [@healyp](https://discourse.julialang.org/u/healyp)\
**Replies:** 1\
**Last updated:** [October 1, 2020, 9:20am UTC](https://discourse.julialang.org/t/how-to-get-more-info-on-w-workers/34290 "2020-10-01T09:20:39Z")

</div>

I’m getting the following error From worker 30: ErrorException("Process(30) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(30) - Invalid connection credentials sent by remote…

---

## [How to do distributed multithreading?](https://discourse.julialang.org/t/how-to-do-distributed-multithreading/47373)

<div class="topic-metadata">

**Author:** [@vkv](https://discourse.julialang.org/u/vkv)\
**Replies:** 1\
**Last updated:** [September 27, 2020, 10:30pm UTC](https://discourse.julialang.org/t/how-to-do-distributed-multithreading/47373 "2020-09-27T22:30:31Z")

</div>

I’ve a function that uses multithreading to compute a function f with each thread value and stores the value as an array element. function multithread() num\_threads = 8 output = Array{Float64}(num\_thread…

---

## [Getting started with distributed Julia computations on a cluster](https://discourse.julialang.org/t/getting-started-with-distributed-julia-computations-on-a-cluster/47366)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 1\
**Last updated:** [September 27, 2020, 7:06pm UTC](https://discourse.julialang.org/t/getting-started-with-distributed-julia-computations-on-a-cluster/47366 "2020-09-27T19:06:45Z")

</div>

Thus far, my parallel Julia experiences have involved running jobs on a single node on a cluster (shared memory environment) using the Distributed.jl module for computations like: using SharedArrays a = SharedArray{Flo…

---

## [Adding (remote) workers to ongoing Dagger job](https://discourse.julialang.org/t/adding-remote-workers-to-ongoing-dagger-job/47196)

<div class="topic-metadata">

**Author:** [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Replies:** 4\
**Last updated:** [September 26, 2020, 7:12pm UTC](https://discourse.julialang.org/t/adding-remote-workers-to-ongoing-dagger-job/47196 "2020-09-26T19:12:19Z")

</div>

Use case is that on LSF (and probably many other cluster systems) jobs to fire up workers are queued and started individually at the whim of the cluster scheduler. This basically means that the time between the first wor…

---

## [How to configure which port workers listen](https://discourse.julialang.org/t/how-to-configure-which-port-workers-listen/46759)

<div class="topic-metadata">

**Author:** [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Replies:** 8\
**Last updated:** [September 24, 2020, 3:49pm UTC](https://discourse.julialang.org/t/how-to-configure-which-port-workers-listen/46759 "2020-09-24T15:49:10Z")

</div>

I’m trying to use ClusterManagers.jl to distribute jobs on an LSF cluster but I have ran into issues as the ports used by workers (9000-10000?) are not open. Is it possible to specify a port range when the --worker argu…

---

## [Concurrency violation on interplay between Distributed and Base.Threads](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080)

<div class="topic-metadata">

**Author:** [@jonas-schulze](https://discourse.julialang.org/u/jonas-schulze)\
**Replies:** 4\
**Last updated:** [September 23, 2020, 7:24am UTC](https://discourse.julialang.org/t/concurrency-violation-on-interplay-between-distributed-and-base-threads/47080 "2020-09-23T07:24:03Z")

</div>

I’m working on a distributed pipeline algorithm that uses several stages per worker process. IIRC tasks cannot hop between threads once they’ve been scheduled. Since I want my stages to potentially run in parallel, I tri…

---

## [Spindle : Scalable Parallel Input Network for Dynamic Load Environments](https://discourse.julialang.org/t/spindle-scalable-parallel-input-network-for-dynamic-load-environments/47013)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 0\
**Last updated:** [September 21, 2020, 2:37pm UTC](https://discourse.julialang.org/t/spindle-scalable-parallel-input-network-for-dynamic-load-environments/47013 "2020-09-21T14:37:08Z")

</div>

From Todd Gamblin on Twitter: I keep reading papers saying containers are the only path to scalable python/shared lib loading. Large HPC jobs can DDOS filesystems by loading many libs/.py files. Spindle solves this gene…

---

## [Always encounter this error after installing new packages on HPC?](https://discourse.julialang.org/t/always-encounter-this-error-after-installing-new-packages-on-hpc/44093)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 2\
**Last updated:** [August 28, 2020, 11:45am UTC](https://discourse.julialang.org/t/always-encounter-this-error-after-installing-new-packages-on-hpc/44093 "2020-08-28T11:45:24Z")

</div>

Each time after I add a new Julia package on HPC, if I do not remove all the compiled packages in the folder “.julia/compiled” and then recompile, I will run into some random bugs. This is really annoying and I wonder wh…

---

## [Modifying arrays by views on workers](https://discourse.julialang.org/t/modifying-arrays-by-views-on-workers/45198)

<div class="topic-metadata">

**Author:** [@tansongchen](https://discourse.julialang.org/u/tansongchen)\
**Replies:** 4\
**Last updated:** [August 20, 2020, 1:54am UTC](https://discourse.julialang.org/t/modifying-arrays-by-views-on-workers/45198 "2020-08-20T01:54:17Z")

</div>

Julia has been good in my parallel computing works, however I encounter an issue recently. Let’s say I have an array a and I would like a @view as a shortcut to modify or refer to part of a, so I make a data structure ab…

---

## [Arrays, MPI, and broadcasting](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098)

<div class="topic-metadata">

**Author:** [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Replies:** 5\
**Last updated:** [August 17, 2020, 2:03pm UTC](https://discourse.julialang.org/t/arrays-mpi-and-broadcasting/45098 "2020-08-17T14:03:46Z")

</div>

I would like to use some kind of AbstractArray based on MPI such that broadcasting can be used for things like +, \*, muladd, which are necessary for simple explicit Runge-Kutta methods (e.g. from OrdinaryDiffEq.jl) cus…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=6)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=8)