# Julia at Scale

**URL:** https://discourse.julialang.org/c/domain/parallel/34.md?page=12

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 13

---

## [Are certain operations just not good candidates for threading?](https://discourse.julialang.org/t/are-certain-operations-just-not-good-candidates-for-threading/19909)

<div class="topic-metadata">

**Author:** [@colintbowers](https://discourse.julialang.org/u/colintbowers)\
**Replies:** 4\
**Last updated:** [January 22, 2019, 11:51pm UTC](https://discourse.julialang.org/t/are-certain-operations-just-not-good-candidates-for-threading/19909 "2019-01-22T23:51:43Z")

</div>

Hi all, I was experimenting with using threading to speed up a loop over vcat that swaps the first and second half of a sequence of vectors. I fire up Julia and verify: julia\> Threads.nthreads() 4 I wrote the followin…

---

## [how to make julia install a curl that respects the proxy? ](https://discourse.julialang.org/t/how-to-make-julia-install-a-curl-that-respects-the-proxy/19891)

<div class="topic-metadata">

**Author:** [@frankwillmore](https://discourse.julialang.org/u/frankwillmore)\
**Replies:** 2\
**Last updated:** [January 21, 2019, 11:46pm UTC](https://discourse.julialang.org/t/how-to-make-julia-install-a-curl-that-respects-the-proxy/19891 "2019-01-21T23:46:26Z")

</div>

I’m building Julia 1.0.3 (and probably 0.7.0) for deployment on a KNL cluster which sits behind a proxy. Cross-compilation succeeds and julia repl runs, as long as there is no need to load dependencies (e.g. MPI) at run-…

---

## [MPI.jl: Functions for cartesian process topology](https://discourse.julialang.org/t/mpi-jl-functions-for-cartesian-process-topology/19437)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 3\
**Last updated:** [January 21, 2019, 8:20am UTC](https://discourse.julialang.org/t/mpi-jl-functions-for-cartesian-process-topology/19437 "2019-01-21T08:20:21Z")

</div>

Hi all, I cannot find any functions to create and use a cartesian process topology on the list of supported functions on the github page of MPI.jl. In particular, I am missing the functions: MPI\_Dims\_create MPI\_Cart\_c…

---

## [MPI.jl - Different behaviors using MPIManager and mpirun](https://discourse.julialang.org/t/mpi-jl-different-behaviors-using-mpimanager-and-mpirun/19758)

<div class="topic-metadata">

**Author:** [@jalving](https://discourse.julialang.org/u/jalving)\
**Replies:** 0\
**Last updated:** [January 17, 2019, 8:01pm UTC](https://discourse.julialang.org/t/mpi-jl-different-behaviors-using-mpimanager-and-mpirun/19758 "2019-01-17T20:01:39Z")

</div>

I’m getting some interesting behavior using the MPIManager with a parallel optimization solver and I’m trying to rule out whether the MPIManager is doing something different than just using MPI.Init() and running with my…

---

## [Multithreading and pmap](https://discourse.julialang.org/t/multithreading-and-pmap/19284)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 8\
**Last updated:** [January 5, 2019, 1:50am UTC](https://discourse.julialang.org/t/multithreading-and-pmap/19284 "2019-01-05T01:50:59Z")

</div>

I have access to a new machine that has 2 x 20 core Xeon Gold CPUs which support multithreading. I was playing around with some simple pmap problems, and I was not seeing a performance jump when I tried going from 40 to…

---

## [MPI.jl: Error building: recompile with -fPIC?](https://discourse.julialang.org/t/mpi-jl-error-building-recompile-with-fpic/19267)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 11\
**Last updated:** [January 4, 2019, 5:35pm UTC](https://discourse.julialang.org/t/mpi-jl-error-building-recompile-with-fpic/19267 "2019-01-04T17:35:24Z")

</div>

Hi all, Before a complete re-installation of Julia in early December, MPI.jl worked for me. Now I cannot install the MPI.jl package anymore - it fails building: julia\> import Pkg; Pkg.add("MPI") Updating registry at …

---

## [Segmentation fault using ArrayFire and @spawn](https://discourse.julialang.org/t/segmentation-fault-using-arrayfire-and-spawn/19266)

<div class="topic-metadata">

**Author:** [@Noel\_Araujo](https://discourse.julialang.org/u/Noel_Araujo)\
**Replies:** 0\
**Last updated:** [January 4, 2019, 10:58am UTC](https://discourse.julialang.org/t/segmentation-fault-using-arrayfire-and-spawn/19266 "2019-01-04T10:58:47Z")

</div>

Hello beautiful people, I have some process that can fit at my GPU memory, so to speed up, I was wondering if I could run parallel process (using Distributed). My hope was that Julia would do its magic, but I just got a…

---

## [Issues with shared memory parallel k-means implementation](https://discourse.julialang.org/t/issues-with-shared-memory-parallel-k-means-implementation/18894)

<div class="topic-metadata">

**Author:** [@novoselrok](https://discourse.julialang.org/u/novoselrok)\
**Replies:** 1\
**Last updated:** [December 28, 2018, 4:34am UTC](https://discourse.julialang.org/t/issues-with-shared-memory-parallel-k-means-implementation/18894 "2018-12-28T04:34:25Z")

</div>

Hello. I’m trying to implement the k-means algorithm using the @threads macro. I’ve also implemented a sequential program for comparison, but it always run faster than the parallel version. The changes to the parallel a…

---

## [SharedArray example from documentation not working](https://discourse.julialang.org/t/sharedarray-example-from-documentation-not-working/18837)

<div class="topic-metadata">

**Author:** [@mikkling](https://discourse.julialang.org/u/mikkling)\
**Replies:** 1\
**Last updated:** [December 19, 2018, 9:42pm UTC](https://discourse.julialang.org/t/sharedarray-example-from-documentation-not-working/18837 "2018-12-19T21:42:29Z")

</div>

I’m trying to learn the intricacies of distributed computing, and was having troubles with SharedArrays not being available on workers. I went back to the documentation and tried the example. It threw the error below! I…

---

## [DistributedArrays: basic element-wise vector operation is slow / fails](https://discourse.julialang.org/t/distributedarrays-basic-element-wise-vector-operation-is-slow-fails/18604)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 1\
**Last updated:** [December 18, 2018, 9:32am UTC](https://discourse.julialang.org/t/distributedarrays-basic-element-wise-vector-operation-is-slow-fails/18604 "2018-12-18T09:32:48Z")

</div>

Hi all, I am new to Julia and did the following setup with DistributedArrays: julia -p 2 julia\> @everywhere using DistributedArrays julia\> nx = 6; julia\> A = zeros(nx-1); julia\> B = (1:nx).^2.0; julia\> A = distrib…

---

## [DistributedArrays: basic example from doc fails](https://discourse.julialang.org/t/distributedarrays-basic-example-from-doc-fails/18551)

<div class="topic-metadata">

**Author:** [@samo](https://discourse.julialang.org/u/samo)\
**Replies:** 5\
**Last updated:** [December 11, 2018, 7:46pm UTC](https://discourse.julialang.org/t/distributedarrays-basic-example-from-doc-fails/18551 "2018-12-11T19:46:23Z")

</div>

Hi all, I am new to Julia and would like to try out DistributedArrays. However, the basic example from the Julia documentation fails for me. Trying to make sure that there is no issue with the installation, I completely…

---

## [How to use CachingPool?](https://discourse.julialang.org/t/how-to-use-cachingpool/18498)

<div class="topic-metadata">

**Author:** [@MaximilianJHuber](https://discourse.julialang.org/u/MaximilianJHuber)\
**Replies:** 4\
**Last updated:** [December 10, 2018, 11:55pm UTC](https://discourse.julialang.org/t/how-to-use-cachingpool/18498 "2018-12-10T23:55:19Z")

</div>

Do I used the CachingPool incorrectly, or is this a bug? On Julia 1.0.2: using Distributed addprocs(2) @everywhere f\_passall(a,x) = length(x) + a its = 1:20 bigarr = ones(10^8) pool = CachingPool(workers()) pmap(pool, …

---

## [How to use FFTW MPI routines with MPI.jl](https://discourse.julialang.org/t/how-to-use-fftw-mpi-routines-with-mpi-jl/18533)

<div class="topic-metadata">

**Author:** [@favba](https://discourse.julialang.org/u/favba)\
**Replies:** 2\
**Last updated:** [December 10, 2018, 7:33pm UTC](https://discourse.julialang.org/t/how-to-use-fftw-mpi-routines-with-mpi-jl/18533 "2018-12-10T19:33:37Z")

</div>

I’d like to use FFTW’s distributed memory routines and I have several doubts on how to achieve this. I guess one should be able to build that on top of MPI.jl and FFTW.jl, right? I’m still stuck in very conceptual doubt…

---

## [RemoteChannel with priority](https://discourse.julialang.org/t/remotechannel-with-priority/18442)

<div class="topic-metadata">

**Author:** [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Replies:** 0\
**Last updated:** [December 7, 2018, 10:49pm UTC](https://discourse.julialang.org/t/remotechannel-with-priority/18442 "2018-12-07T22:49:28Z")

</div>

Is there a preferred way to incorporate priorities into RemoteChannels so that take! will deliver the most prioritized item first?

---

## [Best practice for executing many instances parallel](https://discourse.julialang.org/t/best-practice-for-executing-many-instances-parallel/18107)

<div class="topic-metadata">

**Author:** [@mike\_k](https://discourse.julialang.org/u/mike_k)\
**Replies:** 8\
**Last updated:** [December 3, 2018, 1:31pm UTC](https://discourse.julialang.org/t/best-practice-for-executing-many-instances-parallel/18107 "2018-12-03T13:31:05Z")

</div>

Dear community, as indicated in the title, I would like to run many (thousands) of computational experiments on a hpc-cluster using julia code. Notice that I’m new to julia, so hopefully there are better options than th…

---

## [Parallel Good Practice](https://discourse.julialang.org/t/parallel-good-practice/17486)

<div class="topic-metadata">

**Author:** [@Julien](https://discourse.julialang.org/u/Julien)\
**Replies:** 22\
**Last updated:** [November 30, 2018, 2:10pm UTC](https://discourse.julialang.org/t/parallel-good-practice/17486 "2018-11-30T14:10:24Z")

</div>

Hi, I have quite a naive question regarding parallelism in Julia. Let’s say I have a machine with 4 CPUs. I want to fully use my machine and I can easily use 4 workers in parallel for my problem. The manual says: " Le…

---

## [\[Ann\] julia in parallel batch mode: job schedulers, etc](https://discourse.julialang.org/t/ann-julia-in-parallel-batch-mode-job-schedulers-etc/17855)

<div class="topic-metadata">

**Author:** [@floswald](https://discourse.julialang.org/u/floswald)\
**Replies:** 2\
**Last updated:** [November 26, 2018, 6:20am UTC](https://discourse.julialang.org/t/ann-julia-in-parallel-batch-mode-job-schedulers-etc/17855 "2018-11-26T06:20:03Z")

</div>

This may be completely obvious and trivial to most people, but I constantly encounter problems with ClusterManagers.jl package on hpc systems. This is related to this discussion. The main reason is that the model of that…

---

## [Using SharedArray with structs](https://discourse.julialang.org/t/using-sharedarray-with-structs/17983)

<div class="topic-metadata">

**Author:** [@HildingElmqvist](https://discourse.julialang.org/u/HildingElmqvist)\
**Replies:** 1\
**Last updated:** [November 25, 2018, 6:30pm UTC](https://discourse.julialang.org/t/using-sharedarray-with-structs/17983 "2018-11-25T18:30:25Z")

</div>

I wanted to use @distributed for and SharedArray{MyStruct,1}. However, I get the error: ArgumentError("type of SharedArray elements must be bits types, got Modia.Instantiation.Instance") Is there any other way of retur…

---

## [Failure-resilient parallel computing](https://discourse.julialang.org/t/failure-resilient-parallel-computing/17929)

<div class="topic-metadata">

**Author:** [@belab](https://discourse.julialang.org/u/belab)\
**Replies:** 6\
**Last updated:** [November 25, 2018, 5:09am UTC](https://discourse.julialang.org/t/failure-resilient-parallel-computing/17929 "2018-11-25T05:09:56Z")

</div>

Hi, I’ve been using Julia’s parallel computing capabilities for some embarrassingly parallel computations (i.e., synchronization between workers is rare) with some success. However, given the HPC environment that I’m wo…

---

## [Global Inner functions used with remote call julia v0.6 vs v1.0](https://discourse.julialang.org/t/global-inner-functions-used-with-remote-call-julia-v0-6-vs-v1-0/17822)

<div class="topic-metadata">

**Author:** [@johanter](https://discourse.julialang.org/u/johanter)\
**Replies:** 0\
**Last updated:** [November 21, 2018, 3:33pm UTC](https://discourse.julialang.org/t/global-inner-functions-used-with-remote-call-julia-v0-6-vs-v1-0/17822 "2018-11-21T15:33:20Z")

</div>

Hello, I’m trying to update some code that was working on julia v0.6 with a global inner function. Below is the basic idea of what worked in julia 0.6. In my case (I read image data and process it in the remote) its a …

---

## [Pmap vs RemoteChannel performance?](https://discourse.julialang.org/t/pmap-vs-remotechannel-performance/17524)

<div class="topic-metadata">

**Author:** [@aaowens](https://discourse.julialang.org/u/aaowens)\
**Replies:** 2\
**Last updated:** [November 22, 2018, 1:28am UTC](https://discourse.julialang.org/t/pmap-vs-remotechannel-performance/17524 "2018-11-22T01:28:35Z")

</div>

I’m working on a notebook here https://gist.github.com/aaowens/0f4ddc74a180bc726d5d9d2290c29b72 to demonstrate parallel value function iteration. I’ve tried it as serial, with threads, pmap, and remote channels. With th…

---

## [Identifying wokers and CPU sockets](https://discourse.julialang.org/t/identifying-wokers-and-cpu-sockets/17776)

<div class="topic-metadata">

**Author:** [@Julien](https://discourse.julialang.org/u/Julien)\
**Replies:** 1\
**Last updated:** [November 20, 2018, 3:18pm UTC](https://discourse.julialang.org/t/identifying-wokers-and-cpu-sockets/17776 "2018-11-20T15:18:08Z")

</div>

Hi, When working on a cluster, is there a way to know which worker belongs to which socket? I am trying to send tasks to a group of workers belonging to the same socket. Thanks a lot.

---

## [Making a struct definition available on the cluster](https://discourse.julialang.org/t/making-a-struct-definition-available-on-the-cluster/17460)

<div class="topic-metadata">

**Author:** [@rabarar](https://discourse.julialang.org/u/rabarar)\
**Replies:** 2\
**Last updated:** [November 13, 2018, 6:25pm UTC](https://discourse.julialang.org/t/making-a-struct-definition-available-on-the-cluster/17460 "2018-11-13T18:25:13Z")

</div>

So I’m new to Julia - and I’ve been experimenting with the Distributed package - I wrote an example program to illustrate some of the concepts. Using remote channels, I can create primitive remote channel type, but how c…

---

## [SharedDict for multiple processes?](https://discourse.julialang.org/t/shareddict-for-multiple-processes/17447)

<div class="topic-metadata">

**Author:** [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Replies:** 1\
**Last updated:** [November 12, 2018, 8:08pm UTC](https://discourse.julialang.org/t/shareddict-for-multiple-processes/17447 "2018-11-12T20:08:56Z")

</div>

I am doing a bunch of distributed long calculations and I want to write results to a Dict. Is there a way for multiple Julia processes to share a Dict (with write access)? I don’t care if writing to the Dict in this wa…

---

## [How to use channel to communicate with a process](https://discourse.julialang.org/t/how-to-use-channel-to-communicate-with-a-process/17074)

<div class="topic-metadata">

**Author:** [@Alexander-Barth](https://discourse.julialang.org/u/Alexander-Barth)\
**Replies:** 0\
**Last updated:** [November 1, 2018, 9:35pm UTC](https://discourse.julialang.org/t/how-to-use-channel-to-communicate-with-a-process/17074 "2018-11-01T21:35:50Z")

</div>

I try to run a function with @span and get some status back on the computation using e.g. a Channel before I get the end-result. The following code already works, but I am new to channels and I am wondering if there is a…

---

## [Avoiding JIT Compilation of Remotecall Functions](https://discourse.julialang.org/t/avoiding-jit-compilation-of-remotecall-functions/15904)

<div class="topic-metadata">

**Author:** [@rohanmclure](https://discourse.julialang.org/u/rohanmclure)\
**Replies:** 2\
**Last updated:** [October 31, 2018, 5:29am UTC](https://discourse.julialang.org/t/avoiding-jit-compilation-of-remotecall-functions/15904 "2018-10-31T05:29:49Z")

</div>

I am implementing a distributed transpose kernel which depends on distributing an array by means of generating equal sized ‘subarrays’ of the whole, distributing them to my worker processes and then performing a remoteca…

---

## [Error when I run for the first time my parallel code](https://discourse.julialang.org/t/error-when-i-run-for-the-first-time-my-parallel-code/16435)

<div class="topic-metadata">

**Author:** [@dsHitman](https://discourse.julialang.org/u/dsHitman)\
**Replies:** 1\
**Last updated:** [October 25, 2018, 7:32pm UTC](https://discourse.julialang.org/t/error-when-i-run-for-the-first-time-my-parallel-code/16435 "2018-10-25T19:32:35Z")

</div>

Hi, when I run for the first time my parallel code the REPL give me this error \`ERROR: LoadError: On worker 2: UndefVarError: SharedArray not defined top-level scope at none:0 eval at .\\boot.jl:319 #116 at C:\\cygwin…

---

## [ccall, C pointers and Multi-Core ](https://discourse.julialang.org/t/ccall-c-pointers-and-multi-core/16733)

<div class="topic-metadata">

**Author:** [@mischmi](https://discourse.julialang.org/u/mischmi)\
**Replies:** 1\
**Last updated:** [October 24, 2018, 4:32pm UTC](https://discourse.julialang.org/t/ccall-c-pointers-and-multi-core/16733 "2018-10-24T16:32:55Z")

</div>

Hello, I’m having a bit of trouble with parallelization and a C library. I’ll try to describe the issue as best as possible: I have a certain mutable struct that stores pointers to memory allocated by C through a ccall…

---

## [Producer/Consumer Tasks Do Not End Properly](https://discourse.julialang.org/t/producer-consumer-tasks-do-not-end-properly/16645)

<div class="topic-metadata">

**Author:** [@zekeriya.sari](https://discourse.julialang.org/u/zekeriya.sari)\
**Replies:** 2\
**Last updated:** [October 23, 2018, 7:15pm UTC](https://discourse.julialang.org/t/producer-consumer-tasks-do-not-end-properly/16645 "2018-10-23T19:15:28Z")

</div>

I want to construct an unbuffered channel and bind a producer task feeding the channel for some certain amount of data and a consumer task consuming the produced data in in infinite loop until a poission pill is taken. T…

---

## [How to achieve atomic global variables with multiple processes (not threads)?](https://discourse.julialang.org/t/how-to-achieve-atomic-global-variables-with-multiple-processes-not-threads/16626)

<div class="topic-metadata">

**Author:** [@novoselrok](https://discourse.julialang.org/u/novoselrok)\
**Replies:** 2\
**Last updated:** [October 22, 2018, 2:57pm UTC](https://discourse.julialang.org/t/how-to-achieve-atomic-global-variables-with-multiple-processes-not-threads/16626 "2018-10-22T14:57:04Z")

</div>

Hello. I’m writing a dynamically scheduled parallel algorithm, which spawns new processes when other processes run out of work. I’m keeping a counter of currently running processes so I don’t spawn more than max availab…

[Previous page](https://discourse.julialang.org/c/domain/parallel/34.md?page=11)

[Next page](https://discourse.julialang.org/c/domain/parallel/34.md?page=13)