# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=5

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 6

---

## [When to use pmap vs Threads.@threads?](https://discourse.julialang.org/t/when-to-use-pmap-vs-threads-threads/72805)

<div class="topic-metadata">

**Author:** [@chelseas](https://discourse.julialang.org/u/chelseas)\
**Replies:** 11\
**Last updated:** [December 16, 2021, 8:28am UTC](https://discourse.julialang.org/t/when-to-use-pmap-vs-threads-threads/72805 "2021-12-16T08:28:31Z")

</div>

Some of my processes may take a minute, and others miliseconds, so load balancing seems like a good idea. Do both do this? Any other things I should know about when using pmap vs. @threads? I’ve tagged jump because I w…

---

## [Ccall Fortran OpenBLAS multithreading issues with Julia 1.7](https://discourse.julialang.org/t/ccall-fortran-openblas-multithreading-issues-with-julia-1-7/72913)

<div class="topic-metadata">

**Author:** [@Perrin\_Meyer](https://discourse.julialang.org/u/Perrin_Meyer)\
**Replies:** 7\
**Last updated:** [December 14, 2021, 4:06pm UTC](https://discourse.julialang.org/t/ccall-fortran-openblas-multithreading-issues-with-julia-1-7/72913 "2021-12-14T16:06:28Z")

</div>

I am using Julia to drive a Fortran based Boundary Element code. The Fortran code uses OpenMP and OpenBLAS for multithreading. When I call into the BEM code from Julia 1.7, I now only get single-threaded performance. O…

---

## [Who is using Dagger?](https://discourse.julialang.org/t/who-is-using-dagger/71970)

<div class="topic-metadata">

**Author:** [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Replies:** 12\
**Last updated:** [December 7, 2021, 5:19pm UTC](https://discourse.julialang.org/t/who-is-using-dagger/71970 "2021-12-07T17:19:19Z")

</div>

Hey all! I’m interested in finding out who in the community is using Dagger, or has used Dagger in the past. If so, I would appreciate if you could provide answers to any of the following questions: What kind of proble…

---

## [PBS cluster: "Syntax error: redirection unexpected" and "nested task error: Unable to read host:port string from worker."](https://discourse.julialang.org/t/pbs-cluster-syntax-error-redirection-unexpected-and-nested-task-error-unable-to-read-host-port-string-from-worker/72620)

<div class="topic-metadata">

**Author:** [@j\_u](https://discourse.julialang.org/u/j_u)\
**Replies:** 1\
**Last updated:** [December 6, 2021, 2:50pm UTC](https://discourse.julialang.org/t/pbs-cluster-syntax-error-redirection-unexpected-and-nested-task-error-unable-to-read-host-port-string-from-worker/72620 "2021-12-06T14:50:42Z")

</div>

Hi, I am new to Distributed computing. Everything works fine with Distributed on one machine. It seems that everything works fine on two machines with MPI.jl (it seems to be slower than Distributed on one machine but wo…

---

## [ Why can't I use julia addprocs?(ECONNREFUSED)](https://discourse.julialang.org/t/why-cant-i-use-julia-addprocs-econnrefused/72477)

<div class="topic-metadata">

**Author:** [@GalloLuigi](https://discourse.julialang.org/u/GalloLuigi)\
**Replies:** 19\
**Last updated:** [December 5, 2021, 8:51pm UTC](https://discourse.julialang.org/t/why-cant-i-use-julia-addprocs-econnrefused/72477 "2021-12-05T20:51:09Z")

</div>

I’m writing a program that should connect to remote virtual machines and then do distributed execution. I have this warning: “Warning: Identity file /home/luigal/.ssh/id\_rsa not accessible: No such file or directory.”.(…

---

## [JuliaHPC Meeting](https://discourse.julialang.org/t/juliahpc-meeting/49093)

<div class="topic-metadata">

**Author:** [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Replies:** 10\
**Last updated:** [November 23, 2021, 3:07pm UTC](https://discourse.julialang.org/t/juliahpc-meeting/49093 "2021-11-23T15:07:11Z")

</div>

Several of us have started a roughly-monthly Julia on HPC-related discussion. The next one is to be held tomorrow (for me at least, probably today for almost everyone else): 18:00 UTC (2pm US Eastern), Tuesday October 2…

---

## [Mpi timing](https://discourse.julialang.org/t/mpi-timing/35953)

<div class="topic-metadata">

**Author:** [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Replies:** 2\
**Last updated:** [November 19, 2021, 12:58pm UTC](https://discourse.julialang.org/t/mpi-timing/35953 "2021-11-19T12:58:01Z")

</div>

Hi, I am learning the MPI.jl package. At some point I thought I understand how JIT works, but maybe not. I may need someone help me clarify some basic concepts regarding the MPI usages and Julia: By executing command…

---

## [SC21: "Comparing Julia to Performance Portable Parallel Programming Models for HPC"](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686)

<div class="topic-metadata">

**Author:** [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Replies:** 2\
**Last updated:** [November 17, 2021, 10:01pm UTC](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686 "2021-11-17T22:01:36Z")

</div>

SCI21 paper: Comparing Julia to Performance Portable Parallel Programming Models for HPC \[pdf\] Index Terms—Julia, OpenMP, OpenCL, Kokkos, CUDA, HIP, Performance Portability, Programming Models, GPUs context: Present…

---

## [How to Maximize CPU Utilization - @spawn Assigning to Busy Workers - Use pmap Instead](https://discourse.julialang.org/t/how-to-maximize-cpu-utilization-spawn-assigning-to-busy-workers-use-pmap-instead/53648)

<div class="topic-metadata">

**Author:** [@dburch](https://discourse.julialang.org/u/dburch)\
**Replies:** 17\
**Last updated:** [November 17, 2021, 12:29pm UTC](https://discourse.julialang.org/t/how-to-maximize-cpu-utilization-spawn-assigning-to-busy-workers-use-pmap-instead/53648 "2021-11-17T12:29:31Z")

</div>

This is partially a performance question. I have a parallel process that is working. However, by using @spawn, all of the tasks are spawned to the available workers before any of the tasks start running. It doesn’t seem …

---

## [How to access S3 compatible CEPH bucket?](https://discourse.julialang.org/t/how-to-access-s3-compatible-ceph-bucket/70186)

<div class="topic-metadata">

**Author:** [@j\_u](https://discourse.julialang.org/u/j_u)\
**Replies:** 4\
**Last updated:** [November 2, 2021, 1:31am UTC](https://discourse.julialang.org/t/how-to-access-s3-compatible-ceph-bucket/70186 "2021-11-02T01:31:11Z")

</div>

Hi, Is there any recommended way to access S3 compatible CEPH bucket and send and receive a file? What would be the best way to do it? AWS.jl, or are there maybe any other packages? Are there maybe any examples apart to …

---

## [Performance issues with parallel Julia code](https://discourse.julialang.org/t/performance-issues-with-parallel-julia-code/68919)

<div class="topic-metadata">

**Author:** [@kfrb](https://discourse.julialang.org/u/kfrb)\
**Replies:** 2\
**Last updated:** [October 29, 2021, 10:16pm UTC](https://discourse.julialang.org/t/performance-issues-with-parallel-julia-code/68919 "2021-10-29T22:16:11Z")

</div>

I am working on a scientific code that is experiencing issues with parallelization. The parallel version is slower than the serial one and I am not sure if the right approaches are used for this application. How can I …

---

## [Multinode scaling](https://discourse.julialang.org/t/multinode-scaling/70503)

<div class="topic-metadata">

**Author:** [@Antonis\_Skourtis](https://discourse.julialang.org/u/Antonis_Skourtis)\
**Replies:** 2\
**Last updated:** [October 28, 2021, 8:25am UTC](https://discourse.julialang.org/t/multinode-scaling/70503 "2021-10-28T08:25:50Z")

</div>

Hello, While I was experimenting with the Distributed library I ran into some issue. I am trying to utilize multiple nodes on a given cluster using Slurm but as of yet I cannot get it working. My script.jl is quite sim…

---

## [MPI.jl warning with system provided MPI](https://discourse.julialang.org/t/mpi-jl-warning-with-system-provided-mpi/70063)

<div class="topic-metadata">

**Author:** [@hideakiv](https://discourse.julialang.org/u/hideakiv)\
**Replies:** 7\
**Last updated:** [October 25, 2021, 6:06pm UTC](https://discourse.julialang.org/t/mpi-jl-warning-with-system-provided-mpi/70063 "2021-10-25T18:06:01Z")

</div>

I am using MPI.jl on an HPC cluster. I got a warning ┌ Warning: You appear to be using MPI.jl with the default MPI binary on a cluster. │ We recommend using the system-provided MPI, see the Configuration section of th…

---

## [Is it possible to pass a struct in MPI.jl?](https://discourse.julialang.org/t/is-it-possible-to-pass-a-struct-in-mpi-jl/70137)

<div class="topic-metadata">

**Author:** [@giancarloantonucci](https://discourse.julialang.org/u/giancarloantonucci)\
**Replies:** 9\
**Last updated:** [October 22, 2021, 4:23pm UTC](https://discourse.julialang.org/t/is-it-possible-to-pass-a-struct-in-mpi-jl/70137 "2021-10-22T16:23:47Z")

</div>

I have something like struct Struct a::Vector b::Vector i::Integer end mystruct = Struct(rand(10), rand(5), 1) How can I pass mystruct around with MPI.jl? If it’s not possible as it is, what’s the easiest wor…

---

## ['Stale File Handle' Error When Submitting Job Array on SLURM](https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108)

<div class="topic-metadata">

**Author:** [@dleather](https://discourse.julialang.org/u/dleather)\
**Replies:** 1\
**Last updated:** [October 20, 2021, 9:20pm UTC](https://discourse.julialang.org/t/stale-file-handle-error-when-submitting-job-array-on-slurm/70108 "2021-10-20T21:20:48Z")

</div>

I’m attempting to submit a large number of jobs on an HPC running SLURM. The code runs fine when submitting a single job at a time, but I am submitting 4000 jobs at once. The error is as follows: ERROR: SystemError: cl…

---

## [MPI Jupyter kernel & cluster in Julia (similar to dask-mpi / ipyparallel)?](https://discourse.julialang.org/t/mpi-jupyter-kernel-cluster-in-julia-similar-to-dask-mpi-ipyparallel/67240)

<div class="topic-metadata">

**Author:** [@marius311](https://discourse.julialang.org/u/marius311)\
**Replies:** 9\
**Last updated:** [September 30, 2021, 4:51am UTC](https://discourse.julialang.org/t/mpi-jupyter-kernel-cluster-in-julia-similar-to-dask-mpi-ipyparallel/67240 "2021-09-30T04:51:13Z")

</div>

I’m on a cluster where MPI transport is significantly faster than TCP transport, and I’d like to keep my interactive Jupyter-based workflow for parallel work. A key thing is I don’t want to be doing any intercommunicatio…

---

## [One-sided MPI with Subarrays](https://discourse.julialang.org/t/one-sided-mpi-with-subarrays/63937)

<div class="topic-metadata">

**Author:** [@smillerc](https://discourse.julialang.org/u/smillerc)\
**Replies:** 4\
**Last updated:** [September 22, 2021, 7:13pm UTC](https://discourse.julialang.org/t/one-sided-mpi-with-subarrays/63937 "2021-09-22T19:13:05Z")

</div>

I’m having some issues getting a one-sided subarray MPI.Put to work. I’m trying to do a 2d halo exchange with an arbitrary number of halo cells. I’d like to move the jlo\_edge subarray on processor 1 to the jhi\_halo\_edge …

---

## [Setting up a small compute cluster](https://discourse.julialang.org/t/setting-up-a-small-compute-cluster/50622)

<div class="topic-metadata">

**Author:** [@maxfreu](https://discourse.julialang.org/u/maxfreu)\
**Replies:** 16\
**Last updated:** [September 17, 2021, 9:54am UTC](https://discourse.julialang.org/t/setting-up-a-small-compute-cluster/50622 "2021-09-17T09:54:59Z")

</div>

Hi everybody! I would like to set up a “compute cluster” with 4-5 computers in my workgroup. It should be able to run more or less arbitrary code, so it’s maybe not a totally julia-specific question, but in the end I wo…

---

## [Why don't remotecalls to a WorkerPool block when all workers are busy?](https://discourse.julialang.org/t/why-dont-remotecalls-to-a-workerpool-block-when-all-workers-are-busy/22059)

<div class="topic-metadata">

**Author:** [@Cody-G](https://discourse.julialang.org/u/Cody-G)\
**Replies:** 6\
**Last updated:** [September 3, 2021, 7:09pm UTC](https://discourse.julialang.org/t/why-dont-remotecalls-to-a-workerpool-block-when-all-workers-are-busy/22059 "2021-09-03T19:09:20Z")

</div>

Based on the docs for remotecall I expect the WorkerPool variant to block until a worker in the pool is available for a new job. From docs: remotecall(f, pool::AbstractWorkerPool, args...; kwargs...) -\> Future Work…

---

## [PyCall error on lsf cluster](https://discourse.julialang.org/t/pycall-error-on-lsf-cluster/66837)

<div class="topic-metadata">

**Author:** [@marcobonici](https://discourse.julialang.org/u/marcobonici)\
**Replies:** 1\
**Last updated:** [August 23, 2021, 1:50pm UTC](https://discourse.julialang.org/t/pycall-error-on-lsf-cluster/66837 "2021-08-23T13:50:04Z")

</div>

Dear all, I am having an issue importing a Python package (with PyCall) on the lsf cluster of my university. I want to perform an MCMC analysis using Turing on the Cluster. I have been able to use Turing on my local ma…

---

## [Solving A\\b in parallel](https://discourse.julialang.org/t/solving-a-b-in-parallel/10011)

<div class="topic-metadata">

**Author:** [@wsshin](https://discourse.julialang.org/u/wsshin)\
**Replies:** 13\
**Last updated:** [August 21, 2021, 10:25am UTC](https://discourse.julialang.org/t/solving-a-b-in-parallel/10011 "2021-08-21T10:25:09Z")

</div>

I have an access to a large shared-memory machine. It has memory in the order of TB, and has 64 CPU cores. If I want to perform A\\b for a square matrix A and column vector b on this shared-memory machine using multiple…

---

## [Scaling for multi-threading](https://discourse.julialang.org/t/scaling-for-multi-threading/65301)

<div class="topic-metadata">

**Author:** [@francispoulin](https://discourse.julialang.org/u/francispoulin)\
**Replies:** 10\
**Last updated:** [July 28, 2021, 4:14pm UTC](https://discourse.julialang.org/t/scaling-for-multi-threading/65301 "2021-07-28T16:14:35Z")

</div>

A question for @luraess and @samo . In the mini-course you gave here you give a lot of different code to solve the diffusion equation in 3D, which was a lot of fun. I know that you have shown that multi-GPU can be effic…

---

## [Error building 'MPI\`, libmpi could not be found](https://discourse.julialang.org/t/error-building-mpi-libmpi-could-not-be-found/63785)

<div class="topic-metadata">

**Author:** [@francispoulin](https://discourse.julialang.org/u/francispoulin)\
**Replies:** 28\
**Last updated:** [July 22, 2021, 1:17am UTC](https://discourse.julialang.org/t/error-building-mpi-libmpi-could-not-be-found/63785 "2021-07-22T01:17:37Z")

</div>

I am trying to build MPI on a cluster. I asked for instructions here but it was suggested that I go to Discourse. I have specified the variables that I need to find libmpi but when I try building I get an error. To tr…

---

## [CUDA aware MPI fails but runs on multiple GPUs](https://discourse.julialang.org/t/cuda-aware-mpi-fails-but-runs-on-multiple-gpus/65059)

<div class="topic-metadata">

**Author:** [@francispoulin](https://discourse.julialang.org/u/francispoulin)\
**Replies:** 5\
**Last updated:** [July 21, 2021, 9:32pm UTC](https://discourse.julialang.org/t/cuda-aware-mpi-fails-but-runs-on-multiple-gpus/65059 "2021-07-21T21:32:10Z")

</div>

I am running the library ImplicitGlobalGrid.jl on a server to learn about multi GPU computing. The server has MPI and CUDA but when I try MPI.has\_cuda() I get false, which tells me that the MPI is not CUDA aware. Howeve…

---

## [What do you need from a distributed computation framework?](https://discourse.julialang.org/t/what-do-you-need-from-a-distributed-computation-framework/55676)

<div class="topic-metadata">

**Author:** [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Replies:** 1\
**Last updated:** [July 20, 2021, 11:30pm UTC](https://discourse.julialang.org/t/what-do-you-need-from-a-distributed-computation-framework/55676 "2021-07-20T23:30:26Z")

</div>

Recently I was thinking about adding UDF support to Spark.jl - a Julia bindings for Apache Spark. UDFs is how you run custom functions on distributed datasets in Python and Scala nowadays. However, adding them to Julia t…

---

## [How do I use Plots with MPIClusterManagers in "only workers" mode?](https://discourse.julialang.org/t/how-do-i-use-plots-with-mpiclustermanagers-in-only-workers-mode/64286)

<div class="topic-metadata">

**Author:** [@giancarloantonucci](https://discourse.julialang.org/u/giancarloantonucci)\
**Replies:** 2\
**Last updated:** [July 19, 2021, 10:30pm UTC](https://discourse.julialang.org/t/how-do-i-use-plots-with-mpiclustermanagers-in-only-workers-mode/64286 "2021-07-19T22:30:12Z")

</div>

I find that @mpi\_do from MPIClusterManagers.jl is great with Jupiter notebooks. However, I’m not able to plot anything from within @mpi\_do: using MPIClusterManagers using Distributed manager = MPIManager(np = 4) addproc…

---

## [Julia: how to run embarrassingly parallel jobs with nested for loops?](https://discourse.julialang.org/t/julia-how-to-run-embarrassingly-parallel-jobs-with-nested-for-loops/64410)

<div class="topic-metadata">

**Author:** [@erwanlecarpentier](https://discourse.julialang.org/u/erwanlecarpentier)\
**Replies:** 6\
**Last updated:** [July 13, 2021, 9:22am UTC](https://discourse.julialang.org/t/julia-how-to-run-embarrassingly-parallel-jobs-with-nested-for-loops/64410 "2021-07-13T09:22:03Z")

</div>

Hey! I’d like to run 4 independent tasks in parallel for a certain number of times using 4 cores. Here is the script, say test.jl, that I run with the command julia --threads 4 test.jl: using Distributed function remot…

---

## [Learning and benchmarking resources](https://discourse.julialang.org/t/learning-and-benchmarking-resources/64018)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 0\
**Last updated:** [July 4, 2021, 4:16am UTC](https://discourse.julialang.org/t/learning-and-benchmarking-resources/64018 "2021-07-04T04:16:30Z")

</div>

I saw a post in the Data domain saying it was hard to get resources to learn MPI programming. First off I am not promoting the company I work for (Dell). I like to connect people and to inspire people with possibilities …

---

## [Addprocs with SSH and a ProxyJump times out](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647)

<div class="topic-metadata">

**Author:** [@ararslan](https://discourse.julialang.org/u/ararslan)\
**Replies:** 7\
**Last updated:** [June 22, 2021, 6:21am UTC](https://discourse.julialang.org/t/addprocs-with-ssh-and-a-proxyjump-times-out/26647 "2021-06-22T06:21:00Z")

</div>

In my SSH config file, I have Host a User me HostName a.example.com Host b User me HostName b.example.com ProxyJump a I want to add b as a worker process from my local machine using addprocs. I’m …

---

## [What does this error mean?](https://discourse.julialang.org/t/what-does-this-error-mean/49796)

<div class="topic-metadata">

**Author:** [@threadedblue](https://discourse.julialang.org/u/threadedblue)\
**Replies:** 10\
**Last updated:** [June 21, 2021, 3:52pm UTC](https://discourse.julialang.org/t/what-does-this-error-mean/49796 "2021-06-21T15:52:33Z")

</div>

I am attempting to use the Distributed package from: macOS to: Ubuntu. Julia 1.5.2 on both machines. The remote machine has ssh on port 2222 and not the usual 22. I attempted at one point to provide a bind address of …

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=4)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=6)