@everywhere and Dictionaries

General Usage
,
1

I am running some code in parallel.

I have some parameters that I save in a dictionary:

mod_pars = Dict("par1"=>par1,"par2"=>par2);

and for different values of par1 I want to run a particular function. So given a list of values p_values, I do the following:

@distributed for i=1:length(p_values)
      mod_pars["par1"] = p_values[i];
      res = solve_model(mod_pars);
     #plus some line to save res 
end

While I write
@everywhere function solve_model
I try to do:

@everywhere mod_pars = Dict("par1"=>par1,"par2"=>par2);

but it gives me an error LoadError: On worker 2:.... Must I @everywhere each of the parameters individually as well?

Also, am I allowed to change mod_pars in each loop iteration as I do?

Thank you!

2

You deleted important parts of the error message, so I can only guess, that you encounter this error or similar:

julia> using Distributed

julia> addprocs(2)
2-element Array{Int64,1}:
 26
 27

julia> par1=1
1

julia> par2=2
2

julia> @everywhere mod_pars = Dict("par1"=>par1,"par2"=>par2);
ERROR: On worker 2:
UndefVarError: par1 not defined
top-level scope at none:1

If I am correct, than you are correct, you need @everywhere on par1 and par2 as those are unknown at the workers. Or you can interpolate, like:

@everywhere mod_pars = Dict("par1"=>$par1,"par2"=>$par2);

Both ways may be not appropriate for your real problem, so it would be best to provide a MWE.

1 Like
3

Thank you! Yes you are correct.

But then is there no problem with changing mod_pars within the loop even though I have applied the @everywhere operator on it?

4

The changes would be local only, not at the workers. Thats what I meant with

maybe not appropriate

If you provide a MWE (minimal working example) we (or others) can find a solution.

5

I see. here is a quick example:

using Distributed, PyPlot;

addprocs(2);

#Parameters:
par1 = 5;
par2 = 2;

@everywhere mod_pars = Dict("par1"=>$par1,"par2"=>$par2);
@everywhere par_values = collect(1:10);
 
@distributed for iv = 1:10
    mod_pars["par1"] = par_values[iv];
    plot(1:10,ones(10)*mod_pars["par1"]);
    savefig(joinpath("figures","plot"*"_iv"*".png"));
    close();
end

This runs, doesn’t give me an error but also does not save any figures.

Thank you!

6

This is another example (also doesn’t seem to work).

using Distributed, CSV, DataFrames;

addprocs(2);

#Parameters:
par1 = 5;
par2 = 2;

@everywhere mod_pars = Dict("par1"=>$par1,"par2"=>$par2);
@everywhere par_values = collect(1:10);

@distributed for iv = 1:10
    mod_pars["par1"] = par_values[iv];
    df_tmp = DataFrame();
    df_tmp[!,:values1] = ones(10)*mod_pars["par1"];
    df_tmp[!,:values2] = ones(10)*mod_pars["par2"];
    CSV.write(joinpath("figures","data_"*string(iv)*".csv"),df_tmp);
end

It ends with no error, just by saying:

Task (runnable) @0x00007f00fde7dae0
7

I think I figured it out! I need to declare I am using a particular package on all the workers, that is:

using Distributed;

@everywhere begin
     using CSV, DataFrames
end

addprocs(2);

#Parameters:
par1 = 5;
par2 = 2;

@everywhere mod_pars = Dict("par1"=>$par1,"par2"=>$par2);
@everywhere par_values = collect(1:10);

@distributed for iv = 1:10
    mod_pars["par1"] = par_values[iv];
    df_tmp = DataFrame();
    df_tmp[!,:values1] = ones(10)*mod_pars["par1"];
    df_tmp[!,:values2] = ones(10)*mod_pars["par2"];
    CSV.write(joinpath("figures","data_"*string(iv)*".csv"),df_tmp);
end