@everywhere everywhere

General Usage
1

I’m using Distributed and pmap together for some fairly standard problems, but I realized that, to get my code to run, I am putting @everywhere on just about every line of code, and I’m trying to see if I can simplify this or otherwise clean up what I’m doing.

As an example, suppose I have a function f(x,y), and I want to apply

x1vals = pmap(x->f(x,y0), x0vals)

for some fixed value of y0 and some array x0vals. In my case, y0 is the solution of an optimization problem, minimizing another function, g(y). Currently, I have to do:

@everywhere using Optim
@everywhere function g(y)
# code for g goes here
end 
@everywhere y_guess  = #starting guess for y
@everywhere min = optimize(g, y_guess) 
@everywhere y0 = min.minimizer

Again, this code works fine, but:

  • It would appear I am running the minimization problem on each worker, which seems a bit wasteful
  • IMO, the code is getting a bit cluttered with the @everywhere lines everywhere.
4 Likes
2

You can put all initialization code in a separate file and include the file everywhere

1 Like
3

I guess that’s probably the simplest solution, at least with regard to simplifying code.

4

I wrote a handbook for my colleagues a while back, you might find some tricks there if you’re new to distributed computing in Julia
https://lup.lub.lu.se/search/publication/873af4d5-6229-4ad2-b907-c0ae0f667822

5 Likes
5

You can also just do

@everywhere begin
    using Optim
    function g(y)
        # code for g goes here
    end 
    y_guess  = #starting guess for y
    min = optimize(g, y_guess) 
    y0 = min.minimizer
end
9 Likes
6

Thanks @Mason. I applied your ans to mine, so:
I ran my program with include(main.jl), of which contained an include("functions.jl) in the preamble. In functions.jl I then top-and-tailed the code with:

@everywhere begin # first line of file
[rest of functions.jl]
end # last line of file