Distributed loop and "include"

General Usage
,
1

Hello. My question is how I include an external file within a distributed loop.

For example, the following works:

using Distributed

addprocs(2);

#Parameters:
par1 = 5;
par2 = 2;
@everywhere begin
    using CSV, DataFrames;
end
@everywhere mod_pars = Dict("par1"=>$par1,"par2"=>$par2);
@everywhere par_values = collect(1:10);

@distributed for iv = 1:10
    mod_pars["par1"] = par_values[iv];
    df_tmp = DataFrame();
    df_tmp[!,:values1] = ones(10)*mod_pars["par1"];
    df_tmp[!,:values2] = ones(10)*mod_pars["par2"];
    CSV.write(joinpath("figures","data_"*string(iv)*".csv"),df_tmp);    
end

rmprocs(workers())

but including what’s inside the distributed loop into a separate file, called new_file.jl, will not work:

@distributed for iv = 1:10
    mod_pars["par1"] = par_values[iv];
    include("new_file.jl")
end

Is there some syntax I am getting wrong?

2

You have to write @everywhere include("new_file.jl") so that the file’s content is available on all workers.

3

what does the error say, this use of include is an anti-pattern and I don’t know if it should work. Maybe your pathing for workers are different?

4

That doesn’t seem to work. What I found I can do is wrap it a function and then call that function (after including that function @everywhere).

1 Like
5

https://stackoverflow.com/questions/41288626/what-exactly-does-include-do

the real question is why do you want to put some script code (parts of a for loop) into a file. I suggest the following pattern:

@everhwer par1, par2 ...
@distributed for iv = par_values
   mod_pars = Dict("par1"=>par1,"par2"=>par2);
   do_and_write_CSV(mod_pars, iv)    
end
3 Likes