# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=9

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 10

---

## [Enumerate not suported for @threads](https://discourse.julialang.org/t/enumerate-not-suported-for-threads/37670)

<div class="topic-metadata">

**Author:** [@martincornejo](https://discourse.julialang.org/u/martincornejo)\
**Replies:** 2\
**Last updated:** [April 16, 2020, 3:04am UTC](https://discourse.julialang.org/t/enumerate-not-suported-for-threads/37670 "2020-04-16T03:04:28Z")

</div>

When trying to use @threads on a for loop with enumerate, an error is thrown Threads.@threads for (idx, a) in enumerate(A) ... # Process end ERROR: LoadError: TaskFailedException: MethodError: no method matching f…

---

## [Minimal examples of how to use MPI in JupyterLab](https://discourse.julialang.org/t/minimal-examples-of-how-to-use-mpi-in-jupyterlab/36852)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 3\
**Last updated:** [April 7, 2020, 5:00pm UTC](https://discourse.julialang.org/t/minimal-examples-of-how-to-use-mpi-in-jupyterlab/36852 "2020-04-07T17:00:05Z")

</div>

I am looking for minimal examples of how to use MPI in JupyterLab. In particular, I would be most interested in an approach that comes as close as possible to an all-equal approach instead of a master-slave approach. But…

---

## [Got good results for some tasks and got an error for other tasks in a job array?](https://discourse.julialang.org/t/got-good-results-for-some-tasks-and-got-an-error-for-other-tasks-in-a-job-array/37049)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 2\
**Last updated:** [April 5, 2020, 4:22pm UTC](https://discourse.julialang.org/t/got-good-results-for-some-tasks-and-got-an-error-for-other-tasks-in-a-job-array/37049 "2020-04-05T16:22:42Z")

</div>

I am using my university HPC cluster and I have a job array to do some similar optimization problems using the package Optim.jl. Tasks should use the same packages and etc. They should only differ in the starting values.…

---

## [Multiple Computer Example](https://discourse.julialang.org/t/multiple-computer-example/10518)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 11\
**Last updated:** [April 3, 2020, 11:07am UTC](https://discourse.julialang.org/t/multiple-computer-example/10518 "2020-04-03T11:07:23Z")

</div>

Does anybody have a really simple example of a local host farming out a task to another ssh-connected computer (with julia on it)? think @everywhere function abc(n::Int) sum=0; for i=1:n; sum+=i; end;#for r…

---

## [Pointer error](https://discourse.julialang.org/t/pointer-error/36341)

<div class="topic-metadata">

**Author:** [@marouane1994](https://discourse.julialang.org/u/marouane1994)\
**Replies:** 1\
**Last updated:** [March 25, 2020, 11:57pm UTC](https://discourse.julialang.org/t/pointer-error/36341 "2020-03-25T23:57:24Z")

</div>

it’s my first time using Threads for parallel computing as i’m new to it, and i wanted to work with the parallel for loop using @threads, there are 4 threads and i put the macro before the for loop then i called the code…

---

## ["ERROR: LoadError: TaskFailedException: IOError: stream is closed or unusable" with @distributed on multinode cluster](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768)

<div class="topic-metadata">

**Author:** [@teored90](https://discourse.julialang.org/u/teored90)\
**Replies:** 3\
**Last updated:** [March 19, 2020, 4:22pm UTC](https://discourse.julialang.org/t/error-loaderror-taskfailedexception-ioerror-stream-is-closed-or-unusable-with-distributed-on-multinode-cluster/34768 "2020-03-19T16:22:57Z")

</div>

I have some code that evaluates trajectories of a stochastic master equation in parallel, using a @distributed for with reduction for evaluating average quantities. The code is available here. Some jobs fail with the fo…

---

## [How to avoid large movement of data when using remotecall\_fetch?](https://discourse.julialang.org/t/how-to-avoid-large-movement-of-data-when-using-remotecall-fetch/35746)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 0\
**Last updated:** [March 9, 2020, 6:41am UTC](https://discourse.julialang.org/t/how-to-avoid-large-movement-of-data-when-using-remotecall-fetch/35746 "2020-03-09T06:41:53Z")

</div>

I wrote a code to solve 10k linear programming problems in parallel. I timed this code on a computer with nprocs()=4. Ideally, it should be three times faster than non-parallel code, however, it is only two times faster.…

---

## [Get an error when keep certain variables only on the main process](https://discourse.julialang.org/t/get-an-error-when-keep-certain-variables-only-on-the-main-process/35738)

<div class="topic-metadata">

**Author:** [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Replies:** 0\
**Last updated:** [March 8, 2020, 11:58pm UTC](https://discourse.julialang.org/t/get-an-error-when-keep-certain-variables-only-on-the-main-process/35738 "2020-03-08T23:58:08Z")

</div>

I am trying to learn parallel computing in Julia. I want to send something to every processes, but keep other things only at the main process. I had the following weird error. My sample code is (for simplicity): using …

---

## [Combining distributed computing / multithreading](https://discourse.julialang.org/t/combining-distributed-computing-multithreading/24320)

<div class="topic-metadata">

**Author:** [@moudbis](https://discourse.julialang.org/u/moudbis)\
**Replies:** 7\
**Last updated:** [March 7, 2020, 10:42am UTC](https://discourse.julialang.org/t/combining-distributed-computing-multithreading/24320 "2020-03-07T10:42:06Z")

</div>

Hello everyone, I’m trying to use multithreading in Julia 1.0 over an HPC cluster. This HPC cluster own differents nodes (separate CPU units) and each node has several cores. I want to run Julia parallel code over thi…

---

## [Help setting up Julia on a cluster](https://discourse.julialang.org/t/help-setting-up-julia-on-a-cluster/5519)

<div class="topic-metadata">

**Author:** [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Replies:** 28\
**Last updated:** [March 4, 2020, 12:10am UTC](https://discourse.julialang.org/t/help-setting-up-julia-on-a-cluster/5519 "2020-03-04T00:10:12Z")

</div>

I am trying Julia for the first time on a HPC cluster at my university. Julia v0.6 is already installed, I have a few questions: How to install packages on a custom folder in the cluster other than .julia/v0.6, and wit…

---

## [ClusterManagers.jl hangs on pbs](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056)

<div class="topic-metadata">

**Author:** [@Snir\_Gazit](https://discourse.julialang.org/u/Snir_Gazit)\
**Replies:** 9\
**Last updated:** [March 3, 2020, 11:37pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056 "2020-03-03T23:37:57Z")

</div>

I am trying to run several jobs on a PBS cluster using ClusterManagers.jl. using ClusterManagers addprocs\_pbs(4,queue="batch") The job seems to hang: job id is 24, waiting for job to start ...........................…

---

## [Passing constructed closures to child processes](https://discourse.julialang.org/t/passing-constructed-closures-to-child-processes/34723)

<div class="topic-metadata">

**Author:** [@platawiec](https://discourse.julialang.org/u/platawiec)\
**Replies:** 3\
**Last updated:** [February 20, 2020, 9:02pm UTC](https://discourse.julialang.org/t/passing-constructed-closures-to-child-processes/34723 "2020-02-20T21:02:04Z")

</div>

I am performing a distributed calculation via pmap. For context, I have some expensive model which I need to differentiate, and for this I’ve chosen Zygote. Part of the process is defining an “adjoint” for pmap. I have f…

---

## [Are SharedArrays bypassing cache?](https://discourse.julialang.org/t/are-sharedarrays-bypassing-cache/34269)

<div class="topic-metadata">

**Author:** [@j-fu](https://discourse.julialang.org/u/j-fu)\
**Replies:** 4\
**Last updated:** [February 7, 2020, 6:51pm UTC](https://discourse.julialang.org/t/are-sharedarrays-bypassing-cache/34269 "2020-02-07T18:51:58Z")

</div>

Hi, I am testing parallel computations based on multithreading/multiprocessing. Aimed at iterative solvers for PDEs I am interested in large loops. An interesting benchmark in this respect is the “Schönauer vector triad…

---

## [Add Package on every processor?](https://discourse.julialang.org/t/add-package-on-every-processor/33853)

<div class="topic-metadata">

**Author:** [@squirrel](https://discourse.julialang.org/u/squirrel)\
**Replies:** 6\
**Last updated:** [February 5, 2020, 7:30pm UTC](https://discourse.julialang.org/t/add-package-on-every-processor/33853 "2020-02-05T19:30:25Z")

</div>

I’m trying to use DistributedArrays (per the example), but it seems like the package needs to be added to every processor. Is this right? When I first start Julia, and run using Distributed addprocs() @everywhere using…

---

## [Computer freezes with parallel](https://discourse.julialang.org/t/computer-freezes-with-parallel/33903)

<div class="topic-metadata">

**Author:** [@squirrel](https://discourse.julialang.org/u/squirrel)\
**Replies:** 0\
**Last updated:** [January 28, 2020, 8:13pm UTC](https://discourse.julialang.org/t/computer-freezes-with-parallel/33903 "2020-01-28T20:13:18Z")

</div>

Hi. I need to use parallel programming and I’ve been trying to play around with some basic parts. When I try to benchmark a simple threaded function, my laptop freezes for over five minutes and I resort to doing a hard s…

---

## [Struggling with parallel code in MCMC simulation](https://discourse.julialang.org/t/struggling-with-parallel-code-in-mcmc-simulation/33448)

<div class="topic-metadata">

**Author:** [@johnheyer](https://discourse.julialang.org/u/johnheyer)\
**Replies:** 7\
**Last updated:** [January 17, 2020, 7:52pm UTC](https://discourse.julialang.org/t/struggling-with-parallel-code-in-mcmc-simulation/33448 "2020-01-17T19:52:50Z")

</div>

Hi! I’m really struggling with parallel code execution in Julia. I have implemented an MCMC sampler that depends on packages such as Distributions, Random, etc., as well as two other julia files which I have written. T…

---

## [Pmap but with control over which workers do the tasks?](https://discourse.julialang.org/t/pmap-but-with-control-over-which-workers-do-the-tasks/33270)

<div class="topic-metadata">

**Author:** [@Alan\_Crawford](https://discourse.julialang.org/u/Alan_Crawford)\
**Replies:** 0\
**Last updated:** [January 12, 2020, 5:51pm UTC](https://discourse.julialang.org/t/pmap-but-with-control-over-which-workers-do-the-tasks/33270 "2020-01-12T17:51:08Z")

</div>

I am wondering how to parallelize an expensive computation but where I specify which workers do the work? By way of background, in my use case each worker does a large computation on worker-specific objects (i.e. store…

---

## [Losing some output from redirect\_stdout() on remote workers](https://discourse.julialang.org/t/losing-some-output-from-redirect-stdout-on-remote-workers/33153)

<div class="topic-metadata">

**Author:** [@healyp](https://discourse.julialang.org/u/healyp)\
**Replies:** 2\
**Last updated:** [January 9, 2020, 4:52pm UTC](https://discourse.julialang.org/t/losing-some-output-from-redirect-stdout-on-remote-workers/33153 "2020-01-09T16:52:01Z")

</div>

I’m running some CPLEX (a Math. Programming solver) jobs as remote jobs and trapping each job’s output with the following (more or less): local res open(traceFile, "w") do out # https://discourse.julialang.org/t…

---

## [Addprocs() on remote machines failing](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999)

<div class="topic-metadata">

**Author:** [@healyp](https://discourse.julialang.org/u/healyp)\
**Replies:** 6\
**Last updated:** [December 9, 2019, 4:01pm UTC](https://discourse.julialang.org/t/addprocs-on-remote-machines-failing/29999 "2019-12-09T16:01:39Z")

</div>

I would like to spawn some jobs on a cluster of linux machines. The machines are configured a little differently to my own so I need to specify where the julia executable is and the username when ssh’ing into them. …

---

## [Memory Usage and SharedArrays](https://discourse.julialang.org/t/memory-usage-and-sharedarrays/31694)

<div class="topic-metadata">

**Author:** [@jmcastro2109](https://discourse.julialang.org/u/jmcastro2109)\
**Replies:** 8\
**Last updated:** [December 6, 2019, 8:02am UTC](https://discourse.julialang.org/t/memory-usage-and-sharedarrays/31694 "2019-12-06T08:02:57Z")

</div>

Hi all, I am new to parallel computing and I had a question regarding memory usage with SharedArrays. I am doing parallel computing to do one operation in each worker and write on a particular slice of an Array. If I a…

---

## [Edging towards distributed HPC](https://discourse.julialang.org/t/edging-towards-distributed-hpc/31809)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 0\
**Last updated:** [December 3, 2019, 4:01pm UTC](https://discourse.julialang.org/t/edging-towards-distributed-hpc/31809 "2019-12-03T16:01:25Z")

</div>

No paticular mention of Julia, but a good article

---

## [Distributed Julia computing failed in Rstudio](https://discourse.julialang.org/t/distributed-julia-computing-failed-in-rstudio/23359)

<div class="topic-metadata">

**Author:** [@Ziyi\_Chen](https://discourse.julialang.org/u/Ziyi_Chen)\
**Replies:** 10\
**Last updated:** [November 29, 2019, 5:27pm UTC](https://discourse.julialang.org/t/distributed-julia-computing-failed-in-rstudio/23359 "2019-11-29T17:27:54Z")

</div>

I successfully setup JuliaCall in R: install.packages(“JuliaCall”) devtools::install\_github(“Non-Contradiction/JuliaCall”) # get the development version of JuliaCall library(JuliaCall) julia=julia\_setup(JULIA\_HOME …

---

## [Do Channels work with multithreaded tasks in 1.3?](https://discourse.julialang.org/t/do-channels-work-with-multithreaded-tasks-in-1-3/30931)

<div class="topic-metadata">

**Author:** [@GordStephen](https://discourse.julialang.org/u/GordStephen)\
**Replies:** 1\
**Last updated:** [November 12, 2019, 3:45am UTC](https://discourse.julialang.org/t/do-channels-work-with-multithreaded-tasks-in-1-3/30931 "2019-11-12T03:45:44Z")

</div>

This is perhaps an overly simple question, but I haven’t been able to find a straightfoward answer for it. Now that Julia 1.3 supports spawning parallel tasks across multiple threads, can we do work stealing via those pa…

---

## ["Module Parsers is missing from the cache"](https://discourse.julialang.org/t/module-parsers-is-missing-from-the-cache/30765)

<div class="topic-metadata">

**Author:** [@BioTurboNick](https://discourse.julialang.org/u/BioTurboNick)\
**Replies:** 1\
**Last updated:** [November 5, 2019, 7:08pm UTC](https://discourse.julialang.org/t/module-parsers-is-missing-from-the-cache/30765 "2019-11-05T19:08:53Z")

</div>

I made some changes to my package and then tried running it with workers. Now I get these errors on the workers: Warning: Module Parsers with build ID 337459641262600 is missing from the cache. Cannot write cache file …

---

## [Julia vs Python's Dask: Known speed comparisons?](https://discourse.julialang.org/t/julia-vs-pythons-dask-known-speed-comparisons/29459)

<div class="topic-metadata">

**Author:** [@joshhjacobson](https://discourse.julialang.org/u/joshhjacobson)\
**Replies:** 13\
**Last updated:** [October 15, 2019, 3:58pm UTC](https://discourse.julialang.org/t/julia-vs-pythons-dask-known-speed-comparisons/29459 "2019-10-15T15:58:16Z")

</div>

Hi all, I’m new to the Julia community and have arrived in hopes of improving the speed of parallelization tasks I’ve been running with Dask (distributed) in Python. I know there are some Julia packages that are inspire…

---

## [Parallel Postprocessing](https://discourse.julialang.org/t/parallel-postprocessing/29487)

<div class="topic-metadata">

**Author:** [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Replies:** 10\
**Last updated:** [October 5, 2019, 7:00pm UTC](https://discourse.julialang.org/t/parallel-postprocessing/29487 "2019-10-05T19:00:21Z")

</div>

Hi everyone, I am using Julia PyPlot to do postprocessing for simulation data. Here is the sample script for my parallel work: using Distributed @everywhere using PyCall @everywhere matplotlib = pyimport("matplotlib") …

---

## [Distributed parallelism within packages/applications](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320)

<div class="topic-metadata">

**Author:** [@SebastianM-C](https://discourse.julialang.org/u/SebastianM-C)\
**Replies:** 15\
**Last updated:** [October 4, 2019, 2:30pm UTC](https://discourse.julialang.org/t/distributed-parallelism-within-packages-applications/14320 "2019-10-04T14:30:59Z")

</div>

How can I use parallelism (on multiple processes) in a julia application/package? I have an application (in the Pkg3 sense that is not a package) in which I use pmap. I tried to load the package with using ParallelTest …

---

## [DARPA looks to propel parallelism with PAPPA](https://discourse.julialang.org/t/darpa-looks-to-propel-parallelism-with-pappa/28421)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 2\
**Last updated:** [September 6, 2019, 5:38pm UTC](https://discourse.julialang.org/t/darpa-looks-to-propel-parallelism-with-pappa/28421 "2019-09-06T17:38:27Z")

</div>

https://t.co/iybdsybN0l PAPPA specifically seeks to fill the gap between programing frameworks that automate parallelism but lack the ability to scale and popular data science programming languages like Python and R th…

---

## [Memory allocation and SharedArrays](https://discourse.julialang.org/t/memory-allocation-and-sharedarrays/27272)

<div class="topic-metadata">

**Author:** [@HugoLhuillier](https://discourse.julialang.org/u/HugoLhuillier)\
**Replies:** 3\
**Last updated:** [August 7, 2019, 10:43pm UTC](https://discourse.julialang.org/t/memory-allocation-and-sharedarrays/27272 "2019-08-07T22:43:10Z")

</div>

Hi all, I am new to parallel computing and I had a question regarding memory allocation. Is it possible to update an array in place while doing parallel computing? I tried using SharedArrays and DistributedArrays witho…

---

## [Start up costs when using remotecall\_fetch](https://discourse.julialang.org/t/start-up-costs-when-using-remotecall-fetch/27072)

<div class="topic-metadata">

**Author:** [@DanielAvilaGirardot](https://discourse.julialang.org/u/DanielAvilaGirardot)\
**Replies:** 2\
**Last updated:** [August 4, 2019, 2:21pm UTC](https://discourse.julialang.org/t/start-up-costs-when-using-remotecall-fetch/27072 "2019-08-04T14:21:51Z")

</div>

Hello, Is there a measure of the start up costs when using functions such as pmap or remotecall\_fetch?. By start up cost I mean the time elapsed from the instant in which the remotecall\_fetch command is called and the i…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=8)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=10)