# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=11

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 12

---

## [@distributed fails for many workers?](https://discourse.julialang.org/t/distributed-fails-for-many-workers/24145)

<div class="topic-metadata">

**Author:** [@dkiese](https://discourse.julialang.org/u/dkiese)\
**Replies:** 21\
**Last updated:** [May 14, 2019, 10:10am UTC](https://discourse.julialang.org/t/distributed-fails-for-many-workers/24145 "2019-05-14T10:10:19Z")

</div>

Hello folks, I am currently running some data analysis on a cluster with 68 cores per node. To this end I set up a SharedArray for 68 workers and let them operate on it via @sync @distributed. For small datasets this wo…

---

## [How to launch workers on specific allocation node?](https://discourse.julialang.org/t/how-to-launch-workers-on-specific-allocation-node/24336)

<div class="topic-metadata">

**Author:** [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Replies:** 1\
**Last updated:** [May 17, 2019, 11:02pm UTC](https://discourse.julialang.org/t/how-to-launch-workers-on-specific-allocation-node/24336 "2019-05-17T23:02:01Z")

</div>

Hi! I am doing some process that uses a ton of memory. The way I thought of doing it is to request 2 nodes via slurm and then send the memory-intensive part to a second node. My process looks like: salloc --nodes 2 -…

---

## [State of distributed processing in Julia](https://discourse.julialang.org/t/state-of-distributed-processing-in-julia/24191)

<div class="topic-metadata">

**Author:** [@felipenoris](https://discourse.julialang.org/u/felipenoris)\
**Replies:** 3\
**Last updated:** [May 14, 2019, 4:49pm UTC](https://discourse.julialang.org/t/state-of-distributed-processing-in-julia/24191 "2019-05-14T16:49:42Z")

</div>

I’ll be working on a Hadoop cluster (hdfs+yarn). I’m checking the best options to integrate Julia in this environment to handle big data distributed processing. Any ideas on the best options? Spark.jl? Elly.jl? Mount hd…

---

## [Intel MKL with Distributed fails](https://discourse.julialang.org/t/intel-mkl-with-distributed-fails/24053)

<div class="topic-metadata">

**Author:** [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Replies:** 2\
**Last updated:** [May 12, 2019, 7:29pm UTC](https://discourse.julialang.org/t/intel-mkl-with-distributed-fails/24053 "2019-05-12T19:29:24Z")

</div>

I’m attempting to use Intel MKL over multiple remote workers. I am using KissCluster.jl to set up a lightweight cluster on AWS and a machinefile to configure via addprocs(machines, enable\_threaded\_blas=true, topology=:m…

---

## [@threads in array comprehension](https://discourse.julialang.org/t/threads-in-array-comprehension/24072)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 11\
**Last updated:** [May 11, 2019, 3:48pm UTC](https://discourse.julialang.org/t/threads-in-array-comprehension/24072 "2019-05-11T15:48:28Z")

</div>

This is just a feature suggestion. It would be nice it the @threads macro would work also inside array comprehensions as: \[i @threads for i=1:4\] I.e. that example should produce the same result as: julia\> \[i for i=1:4…

---

## [How to distribute array by column?](https://discourse.julialang.org/t/how-to-distribute-array-by-column/24091)

<div class="topic-metadata">

**Author:** [@Carol](https://discourse.julialang.org/u/Carol)\
**Replies:** 4\
**Last updated:** [May 10, 2019, 10:23pm UTC](https://discourse.julialang.org/t/how-to-distribute-array-by-column/24091 "2019-05-10T22:23:46Z")

</div>

Background: using Distributed @everywhere using DistributedArrays addprocs(4) A=\[1 1 2 2; 1 1 2 2; 3 3 4 4; 3 3 4 4\] adist = distribute(A) I found A is distributed in a checkerboard block way, which means f…

---

## [OOM Killer on Linux](https://discourse.julialang.org/t/oom-killer-on-linux/22722)

<div class="topic-metadata">

**Author:** [@versipellis](https://discourse.julialang.org/u/versipellis)\
**Replies:** 5\
**Last updated:** [May 9, 2019, 3:57pm UTC](https://discourse.julialang.org/t/oom-killer-on-linux/22722 "2019-05-09T15:57:57Z")

</div>

I’m curious if anyone else has run into this. I’m on Ubuntu and trying to run an analysis on some really large datasets (40gigs binary, 1.5billion rows, 1600 chunks) using JuliaDB. I’ve frequently run into the issue whe…

---

## [Why do use @spawnat and fetch?](https://discourse.julialang.org/t/why-do-use-spawnat-and-fetch/24014)

<div class="topic-metadata">

**Author:** [@Carol](https://discourse.julialang.org/u/Carol)\
**Replies:** 3\
**Last updated:** [May 9, 2019, 8:02am UTC](https://discourse.julialang.org/t/why-do-use-spawnat-and-fetch/24014 "2019-05-09T08:02:41Z")

</div>

Hi, I am learning parallel in Julia. And I saw below code of summing an array. Background: using Distributed using DistributedArrays addprocs(4) @everywhere using DistributedArrays a = rand(10^7) adist = distribute(a…

---

## [Reshape DistributedArrays](https://discourse.julialang.org/t/reshape-distributedarrays/23622)

<div class="topic-metadata">

**Author:** [@monty](https://discourse.julialang.org/u/monty)\
**Replies:** 1\
**Last updated:** [May 7, 2019, 2:39pm UTC](https://discourse.julialang.org/t/reshape-distributedarrays/23622 "2019-05-07T14:39:36Z")

</div>

From an existing DistributedArrays dr = DistributedArrays.distribute(rand(2,3,4)) I want to generate a new DistributedArrays which is flattened version of dr with dimensions 3 x 12 by “vcat” out of the following slices…

---

## [Remote access arrays in a numerical grid](https://discourse.julialang.org/t/remote-access-arrays-in-a-numerical-grid/23281)

<div class="topic-metadata">

**Author:** [@ArjunNarayanan](https://discourse.julialang.org/u/ArjunNarayanan)\
**Replies:** 6\
**Last updated:** [April 19, 2019, 11:05pm UTC](https://discourse.julialang.org/t/remote-access-arrays-in-a-numerical-grid/23281 "2019-04-19T23:05:36Z")

</div>

I am learning the distributed computing capabilities of Julia as applied to numerical methods. I am currently interested in learning how to use the standard Distributed capabilities rather than solutions from MPI.jl and …

---

## [Parallel for breaks in SDL application](https://discourse.julialang.org/t/parallel-for-breaks-in-sdl-application/22937)

<div class="topic-metadata">

**Author:** [@xor0110](https://discourse.julialang.org/u/xor0110)\
**Replies:** 4\
**Last updated:** [April 14, 2019, 12:50pm UTC](https://discourse.julialang.org/t/parallel-for-breaks-in-sdl-application/22937 "2019-04-14T12:50:21Z")

</div>

I am doing my first attempt ever at Julia with multi-threading. I wrote a program to draw a fractal using SDL and SIMD instructions. That works well. To render each frame I just do a loop over all pixels, calculate a val…

---

## [Running a function with multiple threads inside @btime never ends](https://discourse.julialang.org/t/running-a-function-with-multiple-threads-inside-btime-never-ends/23047)

<div class="topic-metadata">

**Author:** [@xor0110](https://discourse.julialang.org/u/xor0110)\
**Replies:** 1\
**Last updated:** [April 11, 2019, 6:25pm UTC](https://discourse.julialang.org/t/running-a-function-with-multiple-threads-inside-btime-never-ends/23047 "2019-04-11T18:25:52Z")

</div>

I am trying to debug an SDL application using the experimental multi-threading. It appears to conflict with a parallel for I wrote. I separated this parallel for part just to test it and benchmark separately, and I am ha…

---

## [Empty remotechannel and resizing slurm jobs](https://discourse.julialang.org/t/empty-remotechannel-and-resizing-slurm-jobs/22816)

<div class="topic-metadata">

**Author:** [@vancleve](https://discourse.julialang.org/u/vancleve)\
**Replies:** 0\
**Last updated:** [April 5, 2019, 8:41pm UTC](https://discourse.julialang.org/t/empty-remotechannel-and-resizing-slurm-jobs/22816 "2019-04-05T20:41:17Z")

</div>

I am using RemoteChannels and Distributed for a multicore simulation with the template provided here: https://docs.julialang.org/en/v1/manual/parallel-computing/index.html#Channels-and-RemoteChannels-1 Each process can…

---

## [Memory blow-up when passing DataFrame to function inside @threads loop](https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643)

<div class="topic-metadata">

**Author:** [@ruthmarx1](https://discourse.julialang.org/u/ruthmarx1)\
**Replies:** 1\
**Last updated:** [April 2, 2019, 5:06pm UTC](https://discourse.julialang.org/t/memory-blow-up-when-passing-dataframe-to-function-inside-threads-loop/22643 "2019-04-02T17:06:15Z")

</div>

When I run the code below, the memory usage blows up: function func(df::DataFrame) X = df\[:time\] indices = findall(X .\> 0) end # read in R data rds = "blablab.rds" objs = load(rds); params = collect(0.5:0.005:…

---

## [Dynamic scheduling in @sync @distributed for constructions](https://discourse.julialang.org/t/dynamic-scheduling-in-sync-distributed-for-constructions/22301)

<div class="topic-metadata">

**Author:** [@MMalakhov](https://discourse.julialang.org/u/MMalakhov)\
**Replies:** 6\
**Last updated:** [March 27, 2019, 11:08am UTC](https://discourse.julialang.org/t/dynamic-scheduling-in-sync-distributed-for-constructions/22301 "2019-03-27T11:08:25Z")

</div>

Hello! Maybe it is a too simple question, but I couldn’t find it in the documentation. Constructions like @sync @distributed for j=1:N (body) end make static schedule of iterations by default. But my iterations …

---

## [How to distribute the definitions of preexisting functions to workers?](https://discourse.julialang.org/t/how-to-distribute-the-definitions-of-preexisting-functions-to-workers/22072)

<div class="topic-metadata">

**Author:** [@Babber](https://discourse.julialang.org/u/Babber)\
**Replies:** 12\
**Last updated:** [March 21, 2019, 3:43pm UTC](https://discourse.julialang.org/t/how-to-distribute-the-definitions-of-preexisting-functions-to-workers/22072 "2019-03-21T15:43:21Z")

</div>

I think, more or less, I defined my problem in the title. After extensive search, I couldn’t find a solution to it, maybe someone can help me here. So in a nutshell: I have some functions already defined well before I wo…

---

## [FFTW Plans for multiple threads](https://discourse.julialang.org/t/fftw-plans-for-multiple-threads/21861)

<div class="topic-metadata">

**Author:** [@M\_Kaj](https://discourse.julialang.org/u/M_Kaj)\
**Replies:** 1\
**Last updated:** [March 14, 2019, 1:13pm UTC](https://discourse.julialang.org/t/fftw-plans-for-multiple-threads/21861 "2019-03-14T13:13:49Z")

</div>

I need to compute many convolutions of arrays with the same length and I want to use multiple threads for that. So i want each thread to compute a convolution with fft’s. As I understand it right now the creating of the…

---

## [Set a function on master and send it to all workers](https://discourse.julialang.org/t/set-a-function-on-master-and-send-it-to-all-workers/21586)

<div class="topic-metadata">

**Author:** [@mleprovost](https://discourse.julialang.org/u/mleprovost)\
**Replies:** 1\
**Last updated:** [March 7, 2019, 6:21pm UTC](https://discourse.julialang.org/t/set-a-function-on-master-and-send-it-to-all-workers/21586 "2019-03-07T18:21:30Z")

</div>

Hello, I would like to construct an interpolation function from a very large dataset in a serialized way(master worker) and then send it to all workers for parallel computing. I tried sendto(workers(),interp\_function…

---

## [Parallel calls to build-in functions](https://discourse.julialang.org/t/parallel-calls-to-build-in-functions/21300)

<div class="topic-metadata">

**Author:** [@stakaz](https://discourse.julialang.org/u/stakaz)\
**Replies:** 4\
**Last updated:** [March 2, 2019, 5:23pm UTC](https://discourse.julialang.org/t/parallel-calls-to-build-in-functions/21300 "2019-03-02T17:23:23Z")

</div>

Hello, I am not new to julia but until now never was forced to do any parallel computations with it. Now, I have a simple case where I want to use Thread or Processes to speed up the computations but I don’t really see …

---

## [Parallel Computations](https://discourse.julialang.org/t/parallel-computations/21070)

<div class="topic-metadata">

**Author:** [@ennvvy](https://discourse.julialang.org/u/ennvvy)\
**Replies:** 6\
**Last updated:** [February 22, 2019, 2:07pm UTC](https://discourse.julialang.org/t/parallel-computations/21070 "2019-02-22T14:07:54Z")

</div>

I have migrated to Julia very recently. To be able to lower computation time, I have been trying to implement parallel programming concepts in my serial code. @time begin @everywhere N = 100 ; @everywhere agents = co…

---

## [\[solved\] Can't install MPI.jl in julia 1.0.1/0.7.0 on a CENTOS 7.4](https://discourse.julialang.org/t/solved-cant-install-mpi-jl-in-julia-1-0-1-0-7-0-on-a-centos-7-4/16532)

<div class="topic-metadata">

**Author:** [@GM\_44](https://discourse.julialang.org/u/GM_44)\
**Replies:** 13\
**Last updated:** [February 28, 2019, 12:47am UTC](https://discourse.julialang.org/t/solved-cant-install-mpi-jl-in-julia-1-0-1-0-7-0-on-a-centos-7-4/16532 "2019-02-28T00:47:19Z")

</div>

Hello, I’m new to Julia. I hope this post is at the right place. So, I have to install MPI.jl module for one of our users. It doesn’t work. I have installed Linux 64 bits binary versions of julia 0.7.0 and 1.0.1 julia…

---

## [Distributed nested in Pmap](https://discourse.julialang.org/t/distributed-nested-in-pmap/21081)

<div class="topic-metadata">

**Author:** [@Alessio\_Bellisomi](https://discourse.julialang.org/u/Alessio_Bellisomi)\
**Replies:** 2\
**Last updated:** [February 22, 2019, 5:54pm UTC](https://discourse.julialang.org/t/distributed-nested-in-pmap/21081 "2019-02-22T17:54:59Z")

</div>

Hello, I am fairly new to Julia, I am trying to understand a little better how the workload is distributed, so I prepared this little script to help me out. An external pmap splits the work between different processes,…

---

## [Recommended way to do work stealing in Julia 1.0?](https://discourse.julialang.org/t/recommended-way-to-do-work-stealing-in-julia-1-0/15573)

<div class="topic-metadata">

**Author:** [@orenbenkiki](https://discourse.julialang.org/u/orenbenkiki)\
**Replies:** 6\
**Last updated:** [February 20, 2019, 1:48pm UTC](https://discourse.julialang.org/t/recommended-way-to-do-work-stealing-in-julia-1-0/15573 "2019-02-20T13:48:02Z")

</div>

I have a Python system which is a mixture of C++ code, pandas/numpy code running on top of BLAS/MKL, and plain Python code binding the whole thing together. “It ain’t a pretty sight”. It can be viewed as a nested tree (…

---

## [Functions in @distributed for do not work](https://discourse.julialang.org/t/functions-in-distributed-for-do-not-work/20737)

<div class="topic-metadata">

**Author:** [@MMalakhov](https://discourse.julialang.org/u/MMalakhov)\
**Replies:** 3\
**Last updated:** [February 13, 2019, 10:04am UTC](https://discourse.julialang.org/t/functions-in-distributed-for-do-not-work/20737 "2019-02-13T10:04:21Z")

</div>

Hello! I use Julia 1.0.2 on Atom. I’m trying to parallel my loop and there is a problem which I don’t understand. Parallel loops work great without functions inside. Also program work great on 1 worker with functions. …

---

## [Using RemoteChannels in a Publisher/Subscriber Model](https://discourse.julialang.org/t/using-remotechannels-in-a-publisher-subscriber-model/20523)

<div class="topic-metadata">

**Author:** [@colinxs](https://discourse.julialang.org/u/colinxs)\
**Replies:** 2\
**Last updated:** [February 12, 2019, 10:52am UTC](https://discourse.julialang.org/t/using-remotechannels-in-a-publisher-subscriber-model/20523 "2019-02-12T10:52:31Z")

</div>

Hello all! I’m interested in leveraging Julia’s built-in RemoteChannels to replicate some of the functionality that ROS (http://www.ros.org/) provides. Unfortunately I’m running into some issues with calling put! on a R…

---

## [Run external commands in parallel](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 10\
**Last updated:** [February 7, 2019, 6:59pm UTC](https://discourse.julialang.org/t/run-external-commands-in-parallel/20268 "2019-02-07T18:59:22Z")

</div>

I have a computation for which the time-consuming part boils down to calling an external program, in particular with read(cmd, String) in parallel for 1-5 values of cmd. Should I just construct the commands, and use pma…

---

## [How to design distributed (CPU/GPU) reinforcement learning methods?](https://discourse.julialang.org/t/how-to-design-distributed-cpu-gpu-reinforcement-learning-methods/20539)

<div class="topic-metadata">

**Author:** [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Replies:** 0\
**Last updated:** [February 7, 2019, 10:17am UTC](https://discourse.julialang.org/t/how-to-design-distributed-cpu-gpu-reinforcement-learning-methods/20539 "2019-02-07T10:17:55Z")

</div>

Question to the experts on distributed computing: How should we design distributed reinforcement learning methods in julia? Ray has a nice approach with different optimizers for methods that move around gradients or ba…

---

## [Waiting on a Remote Condition](https://discourse.julialang.org/t/waiting-on-a-remote-condition/20170)

<div class="topic-metadata">

**Author:** [@rohanmclure](https://discourse.julialang.org/u/rohanmclure)\
**Replies:** 0\
**Last updated:** [January 28, 2019, 5:33am UTC](https://discourse.julialang.org/t/waiting-on-a-remote-condition/20170 "2019-01-28T05:33:52Z")

</div>

I’m hoping to use Conditions to synchronise communication between a master and worker process. The notify function will wake up only the tasks that are already waiting on a condition. If we do not know which of the foll…

---

## [Multi-dimensional SharedArrays.jl don't work in @distributed loops](https://discourse.julialang.org/t/multi-dimensional-sharedarrays-jl-dont-work-in-distributed-loops/19944)

<div class="topic-metadata">

**Author:** [@michaelkyu](https://discourse.julialang.org/u/michaelkyu)\
**Replies:** 2\
**Last updated:** [January 23, 2019, 4:04pm UTC](https://discourse.julialang.org/t/multi-dimensional-sharedarrays-jl-dont-work-in-distributed-loops/19944 "2019-01-23T16:04:12Z")

</div>

I’m following the documentation for parallel code in Julia v1.0. The simple example in the docs for using a 1D SharedArrays and @distributed does execute correctly for me. However, if I modify the example to make the Sha…

---

## [Julia 0.6 parallelization of modules and functions](https://discourse.julialang.org/t/julia-0-6-parallelization-of-modules-and-functions/19952)

<div class="topic-metadata">

**Author:** [@wonderboy3489](https://discourse.julialang.org/u/wonderboy3489)\
**Replies:** 2\
**Last updated:** [January 23, 2019, 4:00pm UTC](https://discourse.julialang.org/t/julia-0-6-parallelization-of-modules-and-functions/19952 "2019-01-23T16:00:13Z")

</div>

Hi all, I’m trying to modify some example code from the documentation and am running into an error. Here’s my module that I’m trying to create: @everywhere module parallel\_test using Base.Distributed @everywhere using…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=10)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=12)