# \#distributed

**URL:** https://discourse.julialang.org/tag/distributed/185.md?page=2

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 3

---

## [How to convert a thread-parallelized code into a core-parallelized code?](https://discourse.julialang.org/t/how-to-convert-a-thread-parallelized-code-into-a-core-parallelized-code/114406)

<div class="topic-metadata">

**Author:** [@Uranium238](https://discourse.julialang.org/u/Uranium238)\
**Replies:** 3\
**Last updated:** [May 19, 2024, 7:06am UTC](https://discourse.julialang.org/t/how-to-convert-a-thread-parallelized-code-into-a-core-parallelized-code/114406 "2024-05-19T07:06:44Z")

</div>

This post can be thought as a successor to an old series of posts that I had made where the motive was to improve the performance and eventually parallelize a code that consisted primarily of matrix operations, the end g…

---

## [Is this the right way of distributing the system of differential equations over multiple nodes on a cluster?](https://discourse.julialang.org/t/is-this-the-right-way-of-distributing-the-system-of-differential-equations-over-multiple-nodes-on-a-cluster/113612)

<div class="topic-metadata">

**Author:** [@Shashank](https://discourse.julialang.org/u/Shashank)\
**Replies:** 2\
**Last updated:** [May 6, 2024, 11:51am UTC](https://discourse.julialang.org/t/is-this-the-right-way-of-distributing-the-system-of-differential-equations-over-multiple-nodes-on-a-cluster/113612 "2024-05-06T11:51:40Z")

</div>

I have a large system of ordinary differential equations, and the right-hand side of the equation is coupled and time-consuming. I have written code that parallelizes the evaluation of the right-hand side over multiple t…

---

## [Problems with distributed package](https://discourse.julialang.org/t/problems-with-distributed-package/113893)

<div class="topic-metadata">

**Author:** [@greivin](https://discourse.julialang.org/u/greivin)\
**Replies:** 0\
**Last updated:** [May 6, 2024, 10:18am UTC](https://discourse.julialang.org/t/problems-with-distributed-package/113893 "2024-05-06T10:18:37Z")

</div>

I have been having troubles using the Distributed package. I have one function that returns a 100 x 4 array. This array is produced by averaging over some dimensions of a SharedArray that is constructed with a distribute…

---

## [Is it possible to tell which processes are currently free?](https://discourse.julialang.org/t/is-it-possible-to-tell-which-processes-are-currently-free/113305)

<div class="topic-metadata">

**Author:** [@mikel8](https://discourse.julialang.org/u/mikel8)\
**Replies:** 0\
**Last updated:** [April 21, 2024, 3:57pm UTC](https://discourse.julialang.org/t/is-it-possible-to-tell-which-processes-are-currently-free/113305 "2024-04-21T15:57:17Z")

</div>

Hi all, I’m currently writing a distributed program with several workers. At one point in the code, I am trying to have all free workers split up a loop and execute a function for different input data. While I can make …

---

## [Parallel resampling columnwise](https://discourse.julialang.org/t/parallel-resampling-columnwise/113203)

<div class="topic-metadata">

**Author:** [@jafagervik](https://discourse.julialang.org/u/jafagervik)\
**Replies:** 2\
**Last updated:** [April 20, 2024, 10:48pm UTC](https://discourse.julialang.org/t/parallel-resampling-columnwise/113203 "2024-04-20T22:48:43Z")

</div>

Hi, I have a large matrix I want to resample. Typical size is about 20\_000 \* 500. The code I have for now looks like this: using DSP: resample function resample\_parallel(signal::Matrix{Float32}, rate::Real) result …

---

## [Unable to create parallel package](https://discourse.julialang.org/t/unable-to-create-parallel-package/113207)

<div class="topic-metadata">

**Author:** [@jafagervik](https://discourse.julialang.org/u/jafagervik)\
**Replies:** 1\
**Last updated:** [April 19, 2024, 5:52am UTC](https://discourse.julialang.org/t/unable-to-create-parallel-package/113207 "2024-04-19T05:52:55Z")

</div>

Hi, I’m trying to make a package in which I use Distributed.jl a bit. Lets call it Intro.jl, and I have two files in src directory. Intro.jl contains this code: module Intro using Distributed export mad, meap include…

---

## [Addprocs: specify both multithreading and active environment](https://discourse.julialang.org/t/addprocs-specify-both-multithreading-and-active-environment/80427)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 2\
**Last updated:** [April 16, 2024, 5:11pm UTC](https://discourse.julialang.org/t/addprocs-specify-both-multithreading-and-active-environment/80427 "2024-04-16T17:11:02Z")

</div>

Let’s say I want to add a process with 8 threads. This is easy: using Distributed addprocs(1, exeflags="-t 8") One can check that @fetchfrom 2 Threads.nthreads() returns 8 as expected. But, if I try to specify the env…

---

## [Maintaining structures across workers](https://discourse.julialang.org/t/maintaining-structures-across-workers/112820)

<div class="topic-metadata">

**Author:** [@Utkarsh\_Sharma](https://discourse.julialang.org/u/Utkarsh_Sharma)\
**Replies:** 1\
**Last updated:** [April 12, 2024, 4:07pm UTC](https://discourse.julialang.org/t/maintaining-structures-across-workers/112820 "2024-04-12T16:07:29Z")

</div>

I created a struct params to be used across different workers as - @everywhere params = parameters(Int64\[\], ...) I initialise params.mapping\_ping as params.mapping\_ping = rand(1:params.q\_datacenters, dimensions.num\_fu…

---

## [Running Julia in a SLURM Cluster](https://discourse.julialang.org/t/running-julia-in-a-slurm-cluster/67614)

<div class="topic-metadata">

**Author:** [@Noah\_Guzman](https://discourse.julialang.org/u/Noah_Guzman)\
**Replies:** 6\
**Last updated:** [April 11, 2024, 11:11am UTC](https://discourse.julialang.org/t/running-julia-in-a-slurm-cluster/67614 "2024-04-11T11:11:15Z")

</div>

I’m looking to provide an explicit explanation of the simplest way to get a Julia program running in parallel across multiple nodes in a SLURM cluster (basically a minimal working example that illustrates the logic). My…

---

## [Understanding Distributed Programming in Julia](https://discourse.julialang.org/t/understanding-distributed-programming-in-julia/112801)

<div class="topic-metadata">

**Author:** [@Utkarsh\_Sharma](https://discourse.julialang.org/u/Utkarsh_Sharma)\
**Replies:** 3\
**Last updated:** [April 11, 2024, 10:16am UTC](https://discourse.julialang.org/t/understanding-distributed-programming-in-julia/112801 "2024-04-11T10:16:35Z")

</div>

According to the documentation, fetch() will - Wait for a \[\`Task’\] (Tasks · The Julia Language) to finish, then return its result value. For the code and its output given below, I have the following questions - How d…

---

## [Multiple Worker Processes with Different Numbers of Threads](https://discourse.julialang.org/t/multiple-worker-processes-with-different-numbers-of-threads/112428)

<div class="topic-metadata">

**Author:** [@mikel8](https://discourse.julialang.org/u/mikel8)\
**Replies:** 2\
**Last updated:** [April 2, 2024, 5:02pm UTC](https://discourse.julialang.org/t/multiple-worker-processes-with-different-numbers-of-threads/112428 "2024-04-02T17:02:33Z")

</div>

Hi all, I’m trying to write a program which needs to solve three large problems and about 50 small problems in parallel. I’m planning to create workers for each process, but need 8 threads for the large problems and one…

---

## [Make variable available on all workers](https://discourse.julialang.org/t/make-variable-available-on-all-workers/112302)

<div class="topic-metadata">

**Author:** [@Igor\_Douven](https://discourse.julialang.org/u/Igor_Douven)\
**Replies:** 0\
**Last updated:** [March 29, 2024, 4:31pm UTC](https://discourse.julialang.org/t/make-variable-available-on-all-workers/112302 "2024-03-29T16:31:43Z")

</div>

Hi all, I’d like to run an evolutionary algorithm which starts with a population of agents and then lets that evolve for a number of generations. To make this not too slow, parts of the algorithm are parallelized. If I …

---

## [Should you use @spawnat/@everywhere within a package? (Distributed.jl)](https://discourse.julialang.org/t/should-you-use-spawnat-everywhere-within-a-package-distributed-jl/112134)

<div class="topic-metadata">

**Author:** [@Rutger0000](https://discourse.julialang.org/u/Rutger0000)\
**Replies:** 1\
**Last updated:** [March 26, 2024, 2:39pm UTC](https://discourse.julialang.org/t/should-you-use-spawnat-everywhere-within-a-package-distributed-jl/112134 "2024-03-26T14:39:34Z")

</div>

How should a package use Distributed.jl? The goal is that you have a package with some functions, and then you execute MyPackage.run() in your main process and the package in the background does all the ‘clever’ parallel…

---

## [Flush stdout from workers](https://discourse.julialang.org/t/flush-stdout-from-workers/111669)

<div class="topic-metadata">

**Author:** [@AAL](https://discourse.julialang.org/u/AAL)\
**Replies:** 0\
**Last updated:** [March 15, 2024, 4:38pm UTC](https://discourse.julialang.org/t/flush-stdout-from-workers/111669 "2024-03-15T16:38:48Z")

</div>

I am running simulations in parallel processes that take many hours to run, so I print the number of replications to keep track of the progress. Here is a MWE: # test.jl using Distributed addprocs(2) @everywhere functi…

---

## [Failed to remove pidfile on close](https://discourse.julialang.org/t/failed-to-remove-pidfile-on-close/111497)

<div class="topic-metadata">

**Author:** [@math\_opt](https://discourse.julialang.org/u/math_opt)\
**Replies:** 0\
**Last updated:** [March 12, 2024, 1:38am UTC](https://discourse.julialang.org/t/failed-to-remove-pidfile-on-close/111497 "2024-03-12T01:38:53Z")

</div>

I am trying to run a Julia script on a Linux cluster, and I see the following warning/error. I use Distributed.jl package to addprocs. It would be helpful if someone could point at the exact cause and help resolve it! …

---

## [Is @distributed using local threads to further distribute?](https://discourse.julialang.org/t/is-distributed-using-local-threads-to-further-distribute/111130)

<div class="topic-metadata">

**Author:** [@schlichtanders](https://discourse.julialang.org/u/schlichtanders)\
**Replies:** 2\
**Last updated:** [March 4, 2024, 3:39pm UTC](https://discourse.julialang.org/t/is-distributed-using-local-threads-to-further-distribute/111130 "2024-03-04T15:39:05Z")

</div>

I am looking for a @distributed reducer for ... version which simply parallelizes the function as large as possible. I.e. using both available machines and available threads. Is @distributed already doing so? if not, i…

---

## [Module scope and distributed computing](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014)

<div class="topic-metadata">

**Author:** [@lgo](https://discourse.julialang.org/u/lgo)\
**Replies:** 2\
**Last updated:** [March 3, 2024, 3:49pm UTC](https://discourse.julialang.org/t/module-scope-and-distributed-computing/111014 "2024-03-03T15:49:13Z")

</div>

I have a rather extensive and inaccessible algorithm that deploys distributed computing. To make it more accessible, I want to wrap as much of the algorithm into the functions of my package. I manage to initialize my va…

---

## [Distributed computing using addprocs](https://discourse.julialang.org/t/distributed-computing-using-addprocs/110984)

<div class="topic-metadata">

**Author:** [@math\_opt](https://discourse.julialang.org/u/math_opt)\
**Replies:** 4\
**Last updated:** [March 1, 2024, 8:02pm UTC](https://discourse.julialang.org/t/distributed-computing-using-addprocs/110984 "2024-03-01T20:02:31Z")

</div>

I have a simple question on using Distributed.jl. I am working on an 8-core machine. Now, I have a simple loop that I can parallelize. So I carry out the following commands: using Distributed addprocs(30) @everywhere …

---

## [How to redirect messages from worker procecces into \`stderr\` on the master procecces](https://discourse.julialang.org/t/how-to-redirect-messages-from-worker-procecces-into-stderr-on-the-master-procecces/111000)

<div class="topic-metadata">

**Author:** [@mdyuki](https://discourse.julialang.org/u/mdyuki)\
**Replies:** 0\
**Last updated:** [March 1, 2024, 8:30am UTC](https://discourse.julialang.org/t/how-to-redirect-messages-from-worker-procecces-into-stderr-on-the-master-procecces/111000 "2024-03-01T08:30:40Z")

</div>

Hello, I am writing a program that uses Distributed package for parallelization. I have a question about messages passed from the workers to master. It seems to be printed on the stdout, even if the message is originally…

---

## [ParallelDataTransfer.jl with local variables](https://discourse.julialang.org/t/paralleldatatransfer-jl-with-local-variables/110916)

<div class="topic-metadata">

**Author:** [@lgo](https://discourse.julialang.org/u/lgo)\
**Replies:** 1\
**Last updated:** [February 28, 2024, 7:49pm UTC](https://discourse.julialang.org/t/paralleldatatransfer-jl-with-local-variables/110916 "2024-02-28T19:49:13Z")

</div>

I want to pass a variable to my workers inside a function (see minimum example below) but I get stuck on passobj. I assume this is because passobj is evaluated at the global scope where the variable is not defined? Is th…

---

## [Efficient ways to implement a (distributed) matrix-matrix product?](https://discourse.julialang.org/t/efficient-ways-to-implement-a-distributed-matrix-matrix-product/79466)

<div class="topic-metadata">

**Author:** [@Neo](https://discourse.julialang.org/u/Neo)\
**Replies:** 2\
**Last updated:** [February 27, 2024, 4:30pm UTC](https://discourse.julialang.org/t/efficient-ways-to-implement-a-distributed-matrix-matrix-product/79466 "2024-02-27T16:30:46Z")

</div>

There are a huge sparse matrix A and a tall-and-skinny matrix B (a set of multiple vectors). I want partition them into pieces for parallel distributed-memory matrix-matrix (matrix-vectors) multiplication. Is there any w…

---

## [How to implement hybrid parallelized programs](https://discourse.julialang.org/t/how-to-implement-hybrid-parallelized-programs/109604)

<div class="topic-metadata">

**Author:** [@mdyuki](https://discourse.julialang.org/u/mdyuki)\
**Replies:** 29\
**Last updated:** [February 12, 2024, 8:31am UTC](https://discourse.julialang.org/t/how-to-implement-hybrid-parallelized-programs/109604 "2024-02-12T08:31:30Z")

</div>

Hello. I have scientific computing codes that contain nested loops. I have parallelized them so far with multithreading one of the loops. This improved performance, but I need to reduce more computational time. To increa…

---

## [I am unable to run a simple distributed.jl code on my slurm cluster](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892)

<div class="topic-metadata">

**Author:** [@Shashank](https://discourse.julialang.org/u/Shashank)\
**Replies:** 11\
**Last updated:** [February 10, 2024, 4:30pm UTC](https://discourse.julialang.org/t/i-am-unable-to-run-a-simple-distributed-jl-code-on-my-slurm-cluster/109892 "2024-02-10T16:30:36Z")

</div>

I am trying to run a simple file that is pasted below: using Distributed addprocs(4) println("Number of processes: ", nprocs()) println("Number of workers: ", nworkers()) @sync @distributed for i in 1:4 sleep(1) …

---

## [Distributed: Passing views of an array for read access to workers (using pmap)](https://discourse.julialang.org/t/distributed-passing-views-of-an-array-for-read-access-to-workers-using-pmap/109379)

<div class="topic-metadata">

**Author:** [@james](https://discourse.julialang.org/u/james)\
**Replies:** 8\
**Last updated:** [January 31, 2024, 9:42am UTC](https://discourse.julialang.org/t/distributed-passing-views-of-an-array-for-read-access-to-workers-using-pmap/109379 "2024-01-31T09:42:42Z")

</div>

I have a large array and want multiple workers to have read access to that array in parallel. Each worker works on a disjoint interval of the array. My plan was to create a view of each of the sections of the array, and…

---

## [Distributed Arrays: Using multiprocessing increaes memory allocations by factor of 200 and slows down](https://discourse.julialang.org/t/distributed-arrays-using-multiprocessing-increaes-memory-allocations-by-factor-of-200-and-slows-down/109402)

<div class="topic-metadata">

**Author:** [@james](https://discourse.julialang.org/u/james)\
**Replies:** 0\
**Last updated:** [January 29, 2024, 2:45pm UTC](https://discourse.julialang.org/t/distributed-arrays-using-multiprocessing-increaes-memory-allocations-by-factor-of-200-and-slows-down/109402 "2024-01-29T14:45:49Z")

</div>

I have 4 processes, each working on a separate part of an array in parallel. Following a previous thread, I use the DistributedArrays package so that every process gets their own part of the array. In the MWE below, I r…

---

## [Stopping pmap when one of the node hit an error](https://discourse.julialang.org/t/stopping-pmap-when-one-of-the-node-hit-an-error/109389)

<div class="topic-metadata">

**Author:** [@muuusso](https://discourse.julialang.org/u/muuusso)\
**Replies:** 0\
**Last updated:** [January 29, 2024, 11:05am UTC](https://discourse.julialang.org/t/stopping-pmap-when-one-of-the-node-hit-an-error/109389 "2024-01-29T11:05:36Z")

</div>

I’m using pmap on my cluster university to distribute computation between different nodes. I’d like to be able to stop all the workers when one of them fails/throw an error. I’ve been looking for a while and I couldn’t f…

---

## [How to Collect Data from a Async Call?](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825)

<div class="topic-metadata">

**Author:** [@DataPsycho](https://discourse.julialang.org/u/DataPsycho)\
**Replies:** 5\
**Last updated:** [November 28, 2020, 5:46am UTC](https://discourse.julialang.org/t/how-to-collect-data-from-a-async-call/50825 "2020-11-28T05:46:54Z")

</div>

Hi, I have build a application where I have a function which collect data from a API call then store it to a Array as follows: using DotEnv, HTTP, JSON3 DotEnv.config(); # Function to collect data function collect\_dat…

---

## [Code after @sync gets executed while @async blocks are still running](https://discourse.julialang.org/t/code-after-sync-gets-executed-while-async-blocks-are-still-running/108854)

<div class="topic-metadata">

**Author:** [@novalex98](https://discourse.julialang.org/u/novalex98)\
**Replies:** 3\
**Last updated:** [January 16, 2024, 9:15am UTC](https://discourse.julialang.org/t/code-after-sync-gets-executed-while-async-blocks-are-still-running/108854 "2024-01-16T09:15:43Z")

</div>

I am using Distributed.jl and trying to parallelize a for loop on a slurm cluster (Hyak klone at the University of Washington). I mark the loop with the @sync macro, then I enclose its insides in an @async macro. There’s…

---

## [What is the best practice for parallel dictionary comprehensions?](https://discourse.julialang.org/t/what-is-the-best-practice-for-parallel-dictionary-comprehensions/108786)

<div class="topic-metadata">

**Author:** [@cdawg](https://discourse.julialang.org/u/cdawg)\
**Replies:** 3\
**Last updated:** [January 14, 2024, 12:51pm UTC](https://discourse.julialang.org/t/what-is-the-best-practice-for-parallel-dictionary-comprehensions/108786 "2024-01-14T12:51:14Z")

</div>

I’m trying to work out a good template for the parallel part of my experiments. I’m looking for rock-solid example code to copy and and extend based on the current ecosystem. Usually, I’m in the embarrassingly || space…

---

## [Setting up worker local buffers when using Distributed.jl](https://discourse.julialang.org/t/setting-up-worker-local-buffers-when-using-distributed-jl/108671)

<div class="topic-metadata">

**Author:** [@RomeoV](https://discourse.julialang.org/u/RomeoV)\
**Replies:** 2\
**Last updated:** [January 11, 2024, 11:20am UTC](https://discourse.julialang.org/t/setting-up-worker-local-buffers-when-using-distributed-jl/108671 "2024-01-11T11:20:10Z")

</div>

Hello all, I am in the process of incrementally optimizing an algorithm, having focused first on single-core performance, then threading, and now want to scale to Distributed workers. One key element of my optimization…

[Previous page](https://discourse.julialang.org/tag/distributed/185.md?match_all_tags=true&page=1&tags%5B%5D=distributed)

[Next page](https://discourse.julialang.org/tag/distributed/185.md?match_all_tags=true&page=3&tags%5B%5D=distributed)
