# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=14

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 15

---

## [Example of application level packages with multiple processes?](https://discourse.julialang.org/t/example-of-application-level-packages-with-multiple-processes/10839)

<div class="topic-metadata">

**Author:** [@jwu](https://discourse.julialang.org/u/jwu)\
**Replies:** 0\
**Last updated:** [May 11, 2018, 10:41am UTC](https://discourse.julialang.org/t/example-of-application-level-packages-with-multiple-processes/10839 "2018-05-11T10:41:34Z")

</div>

I would like to make my package multiprocessing, which should be a strong feature of Julia. However, I have not found any application level package with multiple processing. Could any one give a few example packages?

---

## [Parallel mc recommendation](https://discourse.julialang.org/t/parallel-mc-recommendation/10711)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 8\
**Last updated:** [May 7, 2018, 12:07pm UTC](https://discourse.julialang.org/t/parallel-mc-recommendation/10711 "2018-05-07T12:07:26Z")

</div>

For the following code, which runs independent chains of Random Walk Metropolis, there is the obvious potential for parallelization. I am wondering what strategies people would recommend. This is related to a prior que…

---

## [Docker API for Julia](https://discourse.julialang.org/t/docker-api-for-julia/10705)

<div class="topic-metadata">

**Author:** [@lage](https://discourse.julialang.org/u/lage)\
**Replies:** 0\
**Last updated:** [May 4, 2018, 4:08pm UTC](https://discourse.julialang.org/t/docker-api-for-julia/10705 "2018-05-04T16:08:01Z")

</div>

Hello, Does anybody know if there is a Docker API for managing containers in Julia? The only project I’m aware of is Docker.jl \* which is not compatible to Julia 0.6.2. Thanks, André Lage. (\*) I opened this issue: h…

---

## [Pmap with external command](https://discourse.julialang.org/t/pmap-with-external-command/10676)

<div class="topic-metadata">

**Author:** [@Olivier\_Merchiers](https://discourse.julialang.org/u/Olivier_Merchiers)\
**Replies:** 0\
**Last updated:** [May 3, 2018, 9:24am UTC](https://discourse.julialang.org/t/pmap-with-external-command/10676 "2018-05-03T09:24:17Z")

</div>

Hello everyone, I’m not sure if this is the right category, so feel free to move it. I have the following issue. I would like to combine pmap with an external c++ code (scuff-em). In order to streamline the workflow I …

---

## [Modify a remote object](https://discourse.julialang.org/t/modify-a-remote-object/10612)

<div class="topic-metadata">

**Author:** [@bbrunaud](https://discourse.julialang.org/u/bbrunaud)\
**Replies:** 3\
**Last updated:** [April 30, 2018, 1:38am UTC](https://discourse.julialang.org/t/modify-a-remote-object/10612 "2018-04-30T01:38:47Z")

</div>

Hello, I would like to send an object to a worker and modify the remote object. The following code did not work as expected. julia\> addprocs(1) 1-element Array{Int64,1}: 2 julia\> n = 1 1 julia\> remotecall\_fetch(()-\>n…

---

## [Local Parallel Processing Benchmark Advice](https://discourse.julialang.org/t/local-parallel-processing-benchmark-advice/10519)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 2\
**Last updated:** [April 26, 2018, 2:32pm UTC](https://discourse.julialang.org/t/local-parallel-processing-benchmark-advice/10519 "2018-04-26T14:32:14Z")

</div>

ok, I spent a few days benchmarking various versions of local multiprocessing under julia 0.6.2, with 4-core and 8-core intel processors. The results are at the rear of http://julia.cookbook.tips/doku.php?id=parallel . …

---

## [Baffling addprocs() with @everywhere](https://discourse.julialang.org/t/baffling-addprocs-with-everywhere/10464)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 17\
**Last updated:** [April 24, 2018, 10:33pm UTC](https://discourse.julialang.org/t/baffling-addprocs-with-everywhere/10464 "2018-04-24T22:33:37Z")

</div>

Can addprocs() function only be used at the start of code? @everywhere function samplefun(x::Real) 2.0 end addprocs(2) ## if here, it does not work @everywhere M= 10 sa = SharedArray{Float64}(M) s= @sync @parallel…

---

## [Open Community Runtime](https://discourse.julialang.org/t/open-community-runtime/10500)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 0\
**Last updated:** [April 24, 2018, 5:30am UTC](https://discourse.julialang.org/t/open-community-runtime/10500 "2018-04-24T05:30:15Z")

</div>

Over on the singularity list, I came across mention of OCR https://wiki.modelado.org/Open\_Community\_Runtime Described as a task-based paradigm along the lines of Stanford’s Legion or Cray’s Chapel. It strikes me as it…

---

## [Suggestion of workflow to modify composite object in parallel](https://discourse.julialang.org/t/suggestion-of-workflow-to-modify-composite-object-in-parallel/10495)

<div class="topic-metadata">

**Author:** [@bbrunaud](https://discourse.julialang.org/u/bbrunaud)\
**Replies:** 3\
**Last updated:** [April 24, 2018, 4:47am UTC](https://discourse.julialang.org/t/suggestion-of-workflow-to-modify-composite-object-in-parallel/10495 "2018-04-24T04:47:25Z")

</div>

Hello, I have an object composed of other objects inside, (Graph, Nodes). I want to run an operation on the nodes that modifies the node, adjacent nodes, and the graph. What is the best workflow to do it in parallel. I …

---

## [Error when using "too many" workers](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413)

<div class="topic-metadata">

**Author:** [@bisraelsen](https://discourse.julialang.org/u/bisraelsen)\
**Replies:** 10\
**Last updated:** [April 18, 2018, 3:14pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413 "2018-04-18T15:14:59Z")

</div>

Hi, I have been running code in parallel using julia -p200 for a while. Now I am using a cloud server that has: CPU(s) 96 Cores Per Socket 24 Sockets 2 By my calculat…

---

## [Best practice for data-parallel computation using DistributedArrays](https://discourse.julialang.org/t/best-practice-for-data-parallel-computation-using-distributedarrays/10078)

<div class="topic-metadata">

**Author:** [@jinliangwei](https://discourse.julialang.org/u/jinliangwei)\
**Replies:** 1\
**Last updated:** [April 3, 2018, 7:46pm UTC](https://discourse.julialang.org/t/best-practice-for-data-parallel-computation-using-distributedarrays/10078 "2018-04-03T19:46:44Z")

</div>

There is a popular programming paradigm in distributed ML training called data parallelism, which I will briefly describe below. What is the best way to program it using DistributedArray (or anything else that might be a…

---

## [Decide and queue when core is available?](https://discourse.julialang.org/t/decide-and-queue-when-core-is-available/9949)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 5\
**Last updated:** [March 25, 2018, 2:43am UTC](https://discourse.julialang.org/t/decide-and-queue-when-core-is-available/9949 "2018-03-25T02:43:14Z")

</div>

dear julia experts—I would like to queue tasks onto open CPU cores until one of my tasks has happened to find the correct solution. then I want to be done. In particular, I need to know how my master waits for many sla…

---

## [Using Julia with @parallel pmap or blank makes no difference in speed.](https://discourse.julialang.org/t/using-julia-with-parallel-pmap-or-blank-makes-no-difference-in-speed/9885)

<div class="topic-metadata">

**Author:** [@Tillie](https://discourse.julialang.org/u/Tillie)\
**Replies:** 3\
**Last updated:** [March 22, 2018, 1:08pm UTC](https://discourse.julialang.org/t/using-julia-with-parallel-pmap-or-blank-makes-no-difference-in-speed/9885 "2018-03-22T13:08:23Z")

</div>

Hi, the follwoing three test examples are minmal examples of my code. I do not understand why the speed is always the same. Moreover, Julia seems to use only one core. The whole CPU usage is only around 30%. If I us…

---

## [How can I "fetch" without caching the struct?](https://discourse.julialang.org/t/how-can-i-fetch-without-caching-the-struct/9688)

<div class="topic-metadata">

**Author:** [@Christian\_Dengler](https://discourse.julialang.org/u/Christian_Dengler)\
**Replies:** 0\
**Last updated:** [March 13, 2018, 4:27pm UTC](https://discourse.julialang.org/t/how-can-i-fetch-without-caching-the-struct/9688 "2018-03-13T16:27:50Z")

</div>

Greetings, I ran into trouble, or maybe my approach is just wrong . What I want is to create an object on remote workers, which perform some kind of work. Then I need to grab some data from this object. The problem is …

---

## [External processes called by Julia locked to 100% cpu use, even when run in parallel?](https://discourse.julialang.org/t/external-processes-called-by-julia-locked-to-100-cpu-use-even-when-run-in-parallel/8762)

<div class="topic-metadata">

**Author:** [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Replies:** 1\
**Last updated:** [February 23, 2018, 7:42pm UTC](https://discourse.julialang.org/t/external-processes-called-by-julia-locked-to-100-cpu-use-even-when-run-in-parallel/8762 "2018-02-23T19:42:50Z")

</div>

Am I don’t something wrong / is this a known issue / what are the workaround? Below I added some sample C++ code. All it is a bunch of calls to sin to burn enough time so I can watch system resources in top. $ g++ -Of…

---

## [Arguments passing behaviour for @spawnat and fetch](https://discourse.julialang.org/t/arguments-passing-behaviour-for-spawnat-and-fetch/9018)

<div class="topic-metadata">

**Author:** [@Gregstrq](https://discourse.julialang.org/u/Gregstrq)\
**Replies:** 0\
**Last updated:** [February 12, 2018, 5:06pm UTC](https://discourse.julialang.org/t/arguments-passing-behaviour-for-spawnat-and-fetch/9018 "2018-02-12T17:06:17Z")

</div>

I am trying to figure out how data communication between procs is organized, but the docs are a bit vague about this. Do I understand correctly, that any time you do @spawnat worker\_id somefunc(args...), the arguments …

---

## [How to change code on workers - Revise.jl not able](https://discourse.julialang.org/t/how-to-change-code-on-workers-revise-jl-not-able/8847)

<div class="topic-metadata">

**Author:** [@pasha](https://discourse.julialang.org/u/pasha)\
**Replies:** 4\
**Last updated:** [February 6, 2018, 3:19pm UTC](https://discourse.julialang.org/t/how-to-change-code-on-workers-revise-jl-not-able/8847 "2018-02-06T15:19:27Z")

</div>

I have a custom package that runs on multiple processes. If I addprocs(n); using Mypkg it will load the code to each worker (I can tell because of the annoying Nullable warnings). Now when I edit the source code, my Re…

---

## [Parallel Map/Reduce with RemoteChannels - comments on MWE?](https://discourse.julialang.org/t/parallel-map-reduce-with-remotechannels-comments-on-mwe/8761)

<div class="topic-metadata">

**Author:** [@pasha](https://discourse.julialang.org/u/pasha)\
**Replies:** 3\
**Last updated:** [February 3, 2018, 11:40pm UTC](https://discourse.julialang.org/t/parallel-map-reduce-with-remotechannels-comments-on-mwe/8761 "2018-02-03T23:40:18Z")

</div>

Greetings, I’m working on a project that is a candidate for parallel map/reduce. So, I wrote a MWE of a framework for doing that below, using RemoteChannels and remote\_do(), and am interested in feedback. I’m just com…

---

## [Thread lock with async channel?](https://discourse.julialang.org/t/thread-lock-with-async-channel/8674)

<div class="topic-metadata">

**Author:** [@jwu](https://discourse.julialang.org/u/jwu)\
**Replies:** 2\
**Last updated:** [January 31, 2018, 3:37am UTC](https://discourse.julialang.org/t/thread-lock-with-async-channel/8674 "2018-01-31T03:37:25Z")

</div>

The @threads seems not working well with async channel. the following code blocks forever, is there something wrong? using Base.Threads c = Channel{Int}(1) mutex = SpinLock() @sync begin @async begin # …

---

## [When the function is parallaxed, the speed drops](https://discourse.julialang.org/t/when-the-function-is-parallaxed-the-speed-drops/8282)

<div class="topic-metadata">

**Author:** [@1112](https://discourse.julialang.org/u/1112)\
**Replies:** 2\
**Last updated:** [January 11, 2018, 10:51am UTC](https://discourse.julialang.org/t/when-the-function-is-parallaxed-the-speed-drops/8282 "2018-01-11T10:51:40Z")

</div>

How can I speed this code up? function scalar\_mult(k::BigInt) .... end function twofor(n) @sync @parallel for i in 1000:n for j in 10:115 scalar\_mult(BigInt(j \* ^(10,3)+i)) end …

---

## [Is there any way to describe worker number in each machine?](https://discourse.julialang.org/t/is-there-any-way-to-describe-worker-number-in-each-machine/8249)

<div class="topic-metadata">

**Author:** [@jwu](https://discourse.julialang.org/u/jwu)\
**Replies:** 2\
**Last updated:** [January 9, 2018, 8:23pm UTC](https://discourse.julialang.org/t/is-there-any-way-to-describe-worker-number-in-each-machine/8249 "2018-01-09T20:23:20Z")

</div>

I know that one way is repeating the hostname N times to have N workers. Is there a way to explicitly say hostname N ?

---

## [Tset error while trying loading machinefile using multiple machines](https://discourse.julialang.org/t/tset-error-while-trying-loading-machinefile-using-multiple-machines/8238)

<div class="topic-metadata">

**Author:** [@jwu](https://discourse.julialang.org/u/jwu)\
**Replies:** 2\
**Last updated:** [January 8, 2018, 10:41pm UTC](https://discourse.julialang.org/t/tset-error-while-trying-loading-machinefile-using-multiple-machines/8238 "2018-01-08T22:41:26Z")

</div>

I can ssh login without password, but still get some errors. My machinefile format is “host”, one host for each line. I started julia using julia --machinefile machinefile but get some tset errors. Any ideas? tset:…

---

## [How to launch the julia on cluster under Torque scheduler](https://discourse.julialang.org/t/how-to-launch-the-julia-on-cluster-under-torque-scheduler/8155)

<div class="topic-metadata">

**Author:** [@Anna\_W](https://discourse.julialang.org/u/Anna_W)\
**Replies:** 2\
**Last updated:** [January 5, 2018, 9:36pm UTC](https://discourse.julialang.org/t/how-to-launch-the-julia-on-cluster-under-torque-scheduler/8155 "2018-01-05T21:36:00Z")

</div>

Hello, I would be grateful for any help. I was successful in using the ClusterManager.jl package udder slurm scheduler. But I had to switch to another cluster with the torque scheduler and the addprocs() and addprocs\_…

---

## [Addprocs() error, add remote machine](https://discourse.julialang.org/t/addprocs-error-add-remote-machine/8119)

<div class="topic-metadata">

**Author:** [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Replies:** 1\
**Last updated:** [January 3, 2018, 6:14am UTC](https://discourse.julialang.org/t/addprocs-error-add-remote-machine/8119 "2018-01-03T06:14:14Z")

</div>

I have several machines in the LAN, and run the addprocs() on one machine to add cores from another remote machine. I do not know how to set the user password in addprocs(). The error is as follows. Can anyone give me s…

---

## [Why is the result of the operation between \`SharedMatrix\`es not a \`SharedMatrix\`?](https://discourse.julialang.org/t/why-is-the-result-of-the-operation-between-sharedmatrix-es-not-a-sharedmatrix/8013)

<div class="topic-metadata">

**Author:** [@usefulhyun](https://discourse.julialang.org/u/usefulhyun)\
**Replies:** 1\
**Last updated:** [December 27, 2017, 5:57am UTC](https://discourse.julialang.org/t/why-is-the-result-of-the-operation-between-sharedmatrix-es-not-a-sharedmatrix/8013 "2017-12-27T05:57:30Z")

</div>

I created two sharedmatrix as the below sm1 = SharedMatrix{Float64}( rand(1000, 1000) ) sm2 = SharedMatrix{Float64}( rand(1000, 1000) ) sm3 = sm1 \* sm2 ---- (1) However, sm3 is of type Matrix{Float64} not of type Sha…

---

## [Questions on parallel programming and Threads.@threads vs @parallel with SharedArray](https://discourse.julialang.org/t/questions-on-parallel-programming-and-threads-threads-vs-parallel-with-sharedarray/7985)

<div class="topic-metadata">

**Author:** [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Replies:** 7\
**Last updated:** [December 26, 2017, 4:16am UTC](https://discourse.julialang.org/t/questions-on-parallel-programming-and-threads-threads-vs-parallel-with-sharedarray/7985 "2017-12-26T04:16:50Z")

</div>

Hi! I have a few questions on parallel programming and multi-threading. If I understand correctly, the Threads.@threads feature is modeled after OpenMP’s and Cilk’s multi-threading schemes, where one and only one thread…

---

## [Issues running on a PBS cluster](https://discourse.julialang.org/t/issues-running-on-a-pbs-cluster/7900)

<div class="topic-metadata">

**Author:** [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Replies:** 0\
**Last updated:** [December 21, 2017, 6:14pm UTC](https://discourse.julialang.org/t/issues-running-on-a-pbs-cluster/7900 "2017-12-21T18:14:52Z")

</div>

I think this is probably related to the issue I am having with a PBS cluster at my university. We can’t use SSH and I still didn’t figure out how to make addprocs\_pbs(100) to work. Do you have recommendations on how to …

---

## [Issues with machinefile and SLURM](https://discourse.julialang.org/t/issues-with-machinefile-and-slurm/7882)

<div class="topic-metadata">

**Author:** [@nickeubank](https://discourse.julialang.org/u/nickeubank)\
**Replies:** 6\
**Last updated:** [December 21, 2017, 6:08pm UTC](https://discourse.julialang.org/t/issues-with-machinefile-and-slurm/7882 "2017-12-21T18:08:37Z")

</div>

Spent last week fighting with issues on a slurm cluster, and having finally figured it out, I wanted to share the result (and a warning): I had been parallelizing through a slurm batch script with this call: julia --ma…

---

## [Debugging possible issue with \`machinefile\` option on SLURM system](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857)

<div class="topic-metadata">

**Author:** [@nickeubank](https://discourse.julialang.org/u/nickeubank)\
**Replies:** 9\
**Last updated:** [December 19, 2017, 7:10pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857 "2017-12-19T19:10:32Z")

</div>

Trying to debug a problem that may be due to julia not using a provided nodefile on a SLURM system. It appears that jobs are getting spun-off on hosts where it shouldn’t be. The SLURM batch script is: #!/bin/bash #SBA…

---

## [Clarification on pmap master-worker model](https://discourse.julialang.org/t/clarification-on-pmap-master-worker-model/7824)

<div class="topic-metadata">

**Author:** [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Replies:** 3\
**Last updated:** [December 18, 2017, 3:17am UTC](https://discourse.julialang.org/t/clarification-on-pmap-master-worker-model/7824 "2017-12-18T03:17:14Z")

</div>

In the master-worker model with pmap, what is the correct way of checking for parallel vs. serial execution? if nprocs() \> 1 # do task in parallel else # fallback to serial execution end or if nprocs() \> 2 # do …

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=13)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=15)