# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=10

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 11

---

## [Reusing variable in multiple parallel calls](https://discourse.julialang.org/t/reusing-variable-in-multiple-parallel-calls/23187)

<div class="topic-metadata">

**Author:** [@dsweber2](https://discourse.julialang.org/u/dsweber2)\
**Replies:** 5\
**Last updated:** [August 4, 2019, 11:48am UTC](https://discourse.julialang.org/t/reusing-variable-in-multiple-parallel-calls/23187 "2019-08-04T11:48:35Z")

</div>

I am going to be calling the same fft on several remote workers many times each. To get an additional speed improvement, I am aiming to use plan\_fft. But plan\_fft objects aren’t thread safe, so I can’t reuse the same pla…

---

## [Error of using Dagger, according to the manual](https://discourse.julialang.org/t/error-of-using-dagger-according-to-the-manual/27011)

<div class="topic-metadata">

**Author:** [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Replies:** 1\
**Last updated:** [August 1, 2019, 12:24am UTC](https://discourse.julialang.org/t/error-of-using-dagger-according-to-the-manual/27011 "2019-08-01T00:24:39Z")

</div>

I tried Dagger according to the manual and got some errors. using Dagger using Distributed addprocs(3) @everywhere function f1(v1::Float64, v2::Float64) r = 0. for i in 1:10^7 r += sin(v1) + sin(v2) …

---

## [How to reuse object in multiple parallel calls instead of making copies in each call](https://discourse.julialang.org/t/how-to-reuse-object-in-multiple-parallel-calls-instead-of-making-copies-in-each-call/26891)

<div class="topic-metadata">

**Author:** [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Replies:** 0\
**Last updated:** [July 27, 2019, 6:47pm UTC](https://discourse.julialang.org/t/how-to-reuse-object-in-multiple-parallel-calls-instead-of-making-copies-in-each-call/26891 "2019-07-27T18:47:44Z")

</div>

I’m using pmap() to make a large number (~12000) of parallel calls to perform power flow computations based on the pseudocode below: ''' PD, QD are K x N matrices where N ~= 12000 and K = number of loads in current case…

---

## ["Textbook" use of pmap but strange execution times](https://discourse.julialang.org/t/textbook-use-of-pmap-but-strange-execution-times/26852)

<div class="topic-metadata">

**Author:** [@mgiugliano](https://discourse.julialang.org/u/mgiugliano)\
**Replies:** 0\
**Last updated:** [July 26, 2019, 4:15pm UTC](https://discourse.julialang.org/t/textbook-use-of-pmap-but-strange-execution-times/26852 "2019-07-26T16:15:28Z")

</div>

I have 28 cores and 56 threads and I call julia with “-p auto”. When starting from a (non-parallel) example, @time for x in \[10;ones(Int, 10)\] sleep(x/10); println(x) end its execution time (~2s) makes perfect se…

---

## [Reduce allocations in @distributed for](https://discourse.julialang.org/t/reduce-allocations-in-distributed-for/26819)

<div class="topic-metadata">

**Author:** [@xzackli](https://discourse.julialang.org/u/xzackli)\
**Replies:** 1\
**Last updated:** [July 26, 2019, 3:29am UTC](https://discourse.julialang.org/t/reduce-allocations-in-distributed-for/26819 "2019-07-26T03:29:44Z")

</div>

I’m doing some cheap operations on a set of huge SharedArrays, i.e. computing the distance from x,y,z. using Distributed addprocs(40); @everywhere using SharedArrays function compute\_distance(result::SharedArray, pos::…

---

## [Decrease in performance using Threads.@threads in Linux](https://discourse.julialang.org/t/decrease-in-performance-using-threads-threads-in-linux/26522)

<div class="topic-metadata">

**Author:** [@amcostant](https://discourse.julialang.org/u/amcostant)\
**Replies:** 16\
**Last updated:** [July 23, 2019, 5:19am UTC](https://discourse.julialang.org/t/decrease-in-performance-using-threads-threads-in-linux/26522 "2019-07-23T05:19:09Z")

</div>

I am attempting to parallelize this code function pdiff(b1::Array{Basic,1}, b2::Basic) dfdx = Array{Basic}(undef,length(b1)) Threads.@threads for i = 1:length(b1) a = Basic() ret = ccall((:basic\_…

---

## [Julia multiprocessing is not using fork in Linux?](https://discourse.julialang.org/t/julia-multiprocessing-is-not-using-fork-in-linux/26513)

<div class="topic-metadata">

**Author:** [@jwu](https://discourse.julialang.org/u/jwu)\
**Replies:** 2\
**Last updated:** [July 19, 2019, 1:39am UTC](https://discourse.julialang.org/t/julia-multiprocessing-is-not-using-fork-in-linux/26513 "2019-07-19T01:39:45Z")

</div>

I am not sure I understand it correctly or not. In python multiprocessing, it is using fork in Linux, so the children processes shares memory directly. The memory sharing will not change if you are not changing it since …

---

## [CSP examples in Julia?](https://discourse.julialang.org/t/csp-examples-in-julia/20011)

<div class="topic-metadata">

**Author:** [@NHDaly](https://discourse.julialang.org/u/NHDaly)\
**Replies:** 2\
**Last updated:** [July 18, 2019, 12:13am UTC](https://discourse.julialang.org/t/csp-examples-in-julia/20011 "2019-07-18T00:13:57Z")

</div>

Golang’s concurrency primitives were originally inspired by C. A. R. Hoare’s famous 1978 paper “Communicating sequential processes”, (paper, book). Here’s a blog post where they talk about it: This git repo re-implem…

---

## [@everywhere using FFTW produces "Package FFTW not found in current path"](https://discourse.julialang.org/t/everywhere-using-fftw-produces-package-fftw-not-found-in-current-path/26342)

<div class="topic-metadata">

**Author:** [@egcjm](https://discourse.julialang.org/u/egcjm)\
**Replies:** 3\
**Last updated:** [July 15, 2019, 1:35pm UTC](https://discourse.julialang.org/t/everywhere-using-fftw-produces-package-fftw-not-found-in-current-path/26342 "2019-07-15T13:35:20Z")

</div>

\[This issue has been discussed in various places; unfortunately none of the proposed fixes seems to work for me\]. When executing @everywhere CertainPackages I get an error. As suggested by several I start with: using…

---

## [CPU utilization in parallel computing](https://discourse.julialang.org/t/cpu-utilization-in-parallel-computing/26320)

<div class="topic-metadata">

**Author:** [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)\
**Replies:** 1\
**Last updated:** [July 13, 2019, 2:32pm UTC](https://discourse.julialang.org/t/cpu-utilization-in-parallel-computing/26320 "2019-07-13T14:32:07Z")

</div>

When using @distributed parallel code, if the amount of data needed to be manipulated is large, will CPU utilization reach 100%, which can be used as a basis for successful parallel？ (of course, direct test runtime is mo…

---

## [DistributedArray operation fails](https://discourse.julialang.org/t/distributedarray-operation-fails/26177)

<div class="topic-metadata">

**Author:** [@monty](https://discourse.julialang.org/u/monty)\
**Replies:** 3\
**Last updated:** [July 11, 2019, 8:24am UTC](https://discourse.julialang.org/t/distributedarray-operation-fails/26177 "2019-07-11T08:24:27Z")

</div>

da = DistributedArrays.distribute(rand(10,10)) da' \* da works. However, da \* da' fails. Any suggestions why and how it can be fixed? Thank you very much!

---

## [@threads is really working in parallel?](https://discourse.julialang.org/t/threads-is-really-working-in-parallel/25788)

<div class="topic-metadata">

**Author:** [@kwild](https://discourse.julialang.org/u/kwild)\
**Replies:** 4\
**Last updated:** [June 28, 2019, 12:06pm UTC](https://discourse.julialang.org/t/threads-is-really-working-in-parallel/25788 "2019-06-28T12:06:59Z")

</div>

Hi, on my system: julia\> versioninfo() Julia Version 1.1.1 Commit 55e36cc308 (2019-05-16 04:10 UTC) Platform Info: OS: Windows (x86\_64-w64-mingw32) CPU: Intel(R) Core(TM) i5-4210M CPU @ 2.60GHz WORD\_SIZE: 64 LI…

---

## [Behavior of threaded loop with BoundsError](https://discourse.julialang.org/t/behavior-of-threaded-loop-with-boundserror/25733)

<div class="topic-metadata">

**Author:** [@MaximilianJHuber](https://discourse.julialang.org/u/MaximilianJHuber)\
**Replies:** 1\
**Last updated:** [June 27, 2019, 11:09am UTC](https://discourse.julialang.org/t/behavior-of-threaded-loop-with-boundserror/25733 "2019-06-27T11:09:26Z")

</div>

function f1() x = zeros(10000) Threads.@threads for i in 1:100000 #typo causes BoundsError x\[i\] = 2.0 \* i end return x end f1() returns x with values from 2 up to 20000 and prints Error thrown i…

---

## [First steps with Dagger](https://discourse.julialang.org/t/first-steps-with-dagger/25739)

<div class="topic-metadata">

**Author:** [@abx](https://discourse.julialang.org/u/abx)\
**Replies:** 2\
**Last updated:** [June 27, 2019, 1:15am UTC](https://discourse.julialang.org/t/first-steps-with-dagger/25739 "2019-06-27T01:15:12Z")

</div>

I am moving my first steps with Dagger.jl and I need some help. The following code works fine using Dagger using Distributed addprocs(4) @everywhere using Dagger @everywhere function f(x) println(x) x end a = delay…

---

## [Asynchronous variable change](https://discourse.julialang.org/t/asynchronous-variable-change/25441)

<div class="topic-metadata">

**Author:** [@KestutisMa](https://discourse.julialang.org/u/KestutisMa)\
**Replies:** 2\
**Last updated:** [June 25, 2019, 9:35am UTC](https://discourse.julialang.org/t/asynchronous-variable-change/25441 "2019-06-25T09:35:07Z")

</div>

Hi, I am implementing biophysical simulation software with GUI using Blink.jl and js that controls Julia part of code. I want to be able to start at stop Julia algorithm at any iteration from GUI. I considered idea about…

---

## [Parallel computing: calling module failed](https://discourse.julialang.org/t/parallel-computing-calling-module-failed/25571)

<div class="topic-metadata">

**Author:** [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)\
**Replies:** 1\
**Last updated:** [June 24, 2019, 5:22pm UTC](https://discourse.julialang.org/t/parallel-computing-calling-module-failed/25571 "2019-06-24T17:22:59Z")

</div>

Background: I wrote a module myself, where the function requires parallel computing.Here is my code: module sds using Distributed @everywhere using SharedArrays export test function test(N) S …

---

## [Using Distributed: computational efficiency](https://discourse.julialang.org/t/using-distributed-computational-efficiency/25570)

<div class="topic-metadata">

**Author:** [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)\
**Replies:** 5\
**Last updated:** [June 24, 2019, 12:50pm UTC](https://discourse.julialang.org/t/using-distributed-computational-efficiency/25570 "2019-06-24T12:50:12Z")

</div>

Background: I want to add elements to the matrix and want to use @distributed for acceleration,here is my code: function fill(N) S = zeros(N,N) for i in 1:length(S) S\[i\] = i end end @btime fill(10…

---

## [Questions about getting started with parallel computing](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341)

<div class="topic-metadata">

**Author:** [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)\
**Replies:** 18\
**Last updated:** [June 22, 2019, 2:01pm UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341 "2019-06-22T14:01:03Z")

</div>

Background: I have already written a program with julia. In order to pursue speed, I now want to solve this problem with parallel computing. OpenMP C++: I know that when I used C++ before, I used OpenMP to speed up my p…

---

## [Pmap over data frame](https://discourse.julialang.org/t/pmap-over-data-frame/25432)

<div class="topic-metadata">

**Author:** [@abx](https://discourse.julialang.org/u/abx)\
**Replies:** 1\
**Last updated:** [June 20, 2019, 6:18pm UTC](https://discourse.julialang.org/t/pmap-over-data-frame/25432 "2019-06-20T18:18:13Z")

</div>

I am trying to pmap a function over a DataFrame, probably I am missing something really obvious. As per the example below, I would each DataFrameRow to be an element of the pmap, and each column value of the DataFrameRo…

---

## [How to make parallel subprocesses?](https://discourse.julialang.org/t/how-to-make-parallel-subprocesses/25005)

<div class="topic-metadata">

**Author:** [@HenriDeh](https://discourse.julialang.org/u/HenriDeh)\
**Replies:** 6\
**Last updated:** [June 19, 2019, 10:56am UTC](https://discourse.julialang.org/t/how-to-make-parallel-subprocesses/25005 "2019-06-19T10:56:46Z")

</div>

I am trying to create small subprocesses that would operate in parallel to my main process. In the example below I have 10 different agents that are callable and that generate some numbers. I am interested in saving thes…

---

## [UndefVarError when initializing variable inside function with @everywhere](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183)

<div class="topic-metadata">

**Author:** [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Replies:** 10\
**Last updated:** [June 13, 2019, 2:01pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183 "2019-06-13T14:01:31Z")

</div>

I am trying to solve a problem in which I have to evaluate a function, f(x), for several different values of an argument x. I want to write a function that depending on the number of available processes it uses either m…

---

## [Pmap use of processor cores](https://discourse.julialang.org/t/pmap-use-of-processor-cores/25121)

<div class="topic-metadata">

**Author:** [@abx](https://discourse.julialang.org/u/abx)\
**Replies:** 13\
**Last updated:** [June 12, 2019, 3:06pm UTC](https://discourse.julialang.org/t/pmap-use-of-processor-cores/25121 "2019-06-12T15:06:26Z")

</div>

I have a simple question about pmap and I can’t find a clear answer anywhere. When I have a series of pmap’d processes and let’s say, few of them are very heavy and require most of the resources, are the heavy tasks abl…

---

## [Worker Proc Failing to Iterate JuliaDB Table](https://discourse.julialang.org/t/worker-proc-failing-to-iterate-juliadb-table/25046)

<div class="topic-metadata">

**Author:** [@dburch](https://discourse.julialang.org/u/dburch)\
**Replies:** 1\
**Last updated:** [June 11, 2019, 5:17pm UTC](https://discourse.julialang.org/t/worker-proc-failing-to-iterate-juliadb-table/25046 "2019-06-11T17:17:49Z")

</div>

I am getting an error when I try to iterate through a table on worker 2. I can open the table on worker 2 and println the table, but iteration which I can normally do in a serial process is failing. It errors on line 25:…

---

## [Issue with nested pmap](https://discourse.julialang.org/t/issue-with-nested-pmap/25129)

<div class="topic-metadata">

**Author:** [@abx](https://discourse.julialang.org/u/abx)\
**Replies:** 2\
**Last updated:** [June 11, 2019, 1:55pm UTC](https://discourse.julialang.org/t/issue-with-nested-pmap/25129 "2019-06-11T13:55:03Z")

</div>

Hello, I have this script running in Julia 1.1. If I run it withing a single process (no workers added) everything works fine. If I add process (e.g. addprocs(3)) nothing gets run and the execution hangs indefinitely u…

---

## [Float64 reduction of Float32 computation idiom using @distributed?](https://discourse.julialang.org/t/float64-reduction-of-float32-computation-idiom-using-distributed/25085)

<div class="topic-metadata">

**Author:** [@fghorow](https://discourse.julialang.org/u/fghorow)\
**Replies:** 8\
**Last updated:** [June 10, 2019, 10:45pm UTC](https://discourse.julialang.org/t/float64-reduction-of-float32-computation-idiom-using-distributed/25085 "2019-06-10T22:45:51Z")

</div>

I have a computation where hundreds of millions of Float32 values (for speed) need to be reduction summed in Float64 accumulators. Assuming the reduction is being driven by something like: accum = Array{Float64}(undef,…

---

## [Configuring MPI.jl to run on XC-40 / KNL with craype and cray-mpich](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197)

<div class="topic-metadata">

**Author:** [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Replies:** 10\
**Last updated:** [June 6, 2019, 2:56am UTC](https://discourse.julialang.org/t/configuring-mpi-jl-to-run-on-xc-40-knl-with-craype-and-cray-mpich/20197 "2019-06-06T02:56:12Z")

</div>

I am attempting to configure the MPI.jl package to run in Julia 1.0.3 on theta, a Cray XC-40/KNL. Cross-compilation has not been an issue, but interfacing to the CrayPE and Cray-MPICH are currently stumping me. I have be…

---

## [JULIA\_WORKER\_TIMEOUT error](https://discourse.julialang.org/t/julia-worker-timeout-error/24866)

<div class="topic-metadata">

**Author:** [@abx](https://discourse.julialang.org/u/abx)\
**Replies:** 7\
**Last updated:** [June 4, 2019, 6:22pm UTC](https://discourse.julialang.org/t/julia-worker-timeout-error/24866 "2019-06-04T18:22:48Z")

</div>

I am working on a fairly big code base written in Julia 1.1 and I am experiencing erratic issues with parallel processing. It is very annoying as most of the time everything works perfectly. The application is mainly co…

---

## [Pmap structure master process computes too](https://discourse.julialang.org/t/pmap-structure-master-process-computes-too/24886)

<div class="topic-metadata">

**Author:** [@Wikunia](https://discourse.julialang.org/u/Wikunia)\
**Replies:** 1\
**Last updated:** [June 3, 2019, 11:18am UTC](https://discourse.julialang.org/t/pmap-structure-master-process-computes-too/24886 "2019-06-03T11:18:09Z")

</div>

What does happen if the master process handles the part part after remotecall\_fetch but computes itself as well? Does everything work or are there some catches? I have two pmap like functions nested and the first calls …

---

## [Command line parameters in parallel](https://discourse.julialang.org/t/command-line-parameters-in-parallel/24863)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 0\
**Last updated:** [June 2, 2019, 8:26pm UTC](https://discourse.julialang.org/t/command-line-parameters-in-parallel/24863 "2019-06-02T20:26:38Z")

</div>

If I’ve got a julia script that’s using Distributed for shared memory parallel computing, and I want to read in a command line parameter (i.e., number of iterations per worker), what’s the right way to do this?

---

## [Should memory in worker be freed after fetching result?](https://discourse.julialang.org/t/should-memory-in-worker-be-freed-after-fetching-result/24780)

<div class="topic-metadata">

**Author:** [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Replies:** 1\
**Last updated:** [May 31, 2019, 10:22pm UTC](https://discourse.julialang.org/t/should-memory-in-worker-be-freed-after-fetching-result/24780 "2019-05-31T22:22:11Z")

</div>

Hi there! I was doing some processing using Julia 1.1.0 and ClusterManagers v0.3.2 I have two nodes and I tried launching a function in the second node using the worker 3 with fetch(@spawnat 3 function()) This is bec…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=9)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=11)