Problems using pmap(), and doubt about the number of workers/processes to use

General Usage
1

Hello everyone,

I am using Julia 0.6.4.
I am trying to use parallel computing in a Monte Carlo simulation program, by utilizing the pmap() function, so to apply the same function my_function to different inputs (which I will express in the following as inputs ) by means of different cores. So, the basic structure of my code will be like:

  1. initialization
  2. while … a certain condition
    outputs = pmap(my_function, inputs);
    … additional operations…
    end
  3. final calculations

Now, since my computer has only two physical cores, it should be recommended to use:
addprocs(1), so to have a total number of processes equal to 2. So my question is:
will pmap() manage to use both process 1 and process 2, or maybe will it manage to use only process 2 consequently not taking advantage of process 1? I guess in the second case there would be no point to use pmap().

I am wondering this because according to https://docs.julialang.org/en/v0.6/manual/parallel-computing/:

The processes used by default for parallel operations are referred to as “workers”. When there is only one process, process 1 is considered a worker. Otherwise, workers are considered to be all processes other than process 1.

Thank you in advance for your help!

2

If total processes is 2, then there is only 1 worker.
If you want to use both cores as workers, then addprocs(2)
You can confirm number of workers used by pmap with nworkers()

3

@greg_plowman thank you very much for your feedback. However, I am still having problems implementing pmap, so I am not sure anymore the problem is only the number of processes/workers.
In the following, I will provide more detailed information and rephrase my questions, hoping they are easier to answer.

So, as I said, I am using Julia 0.6.4, and my Windows 10 computer has 2 physical cores.
I have a main script, where I call functions defined in other .jl files, for example:
include("monte_carlo_simulation.jl")

The file “monte_carlo_simulation.jl” has is structured like this:

include("scenario_generator.jl")
include("optimization_function.jl")
function monte_carlo_simulation(input)
…body (in which, among the other things, I use the 2 just included functions)…
return output
end

That is, monte_carlo_simulation() uses other functions (all defined in the same project folder).
Now, I am trying to implement parallel computing to speed up the simulations, by using pmap().
I have applied it in the following way. In the main script, I wrote:

addprocs(2)
@everywhere include("monte_carlo_simulation.jl")
…
ouput = pmap( monte_carlo_simulation, [input_1, input_2]);
output_1 = output[1];
output_2 = output[2];
…
  1. First question:
    If I simply run the first two lines:
addprocs(2)
@everywhere include("monte_carlo_simulation.jl")

I get warnings saying:

WARNING: replacing module JuMP.
WARNING: replacing module JuMP.
WARNING: replacing module Gurobi.
WARNING: replacing module Gurobi.

What does it mean exactly? What could be the reason? And how to fix this?
Notice that JuMP and Gurobi are used in optimization_function(). In such a function, I wrote: using JuMP, Gurobi at the very beginning of the file.
By the way, if I use addprocs(1) instead of addprocs(2), the warning messages halves:

WARNING: replacing module JuMP.
WARNING: replacing module Gurobi.

  1. Second question:

The code runs with no further warnings or errors until just before running pmap(). But when I run also:
ouput = pmap(monte_carlo_simulation, [input_1, input_2]);

I get the following error:

PyError (ccall(@pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw)) <class ‘TypeError’>
TypeError(“cannot serialize ‘_io.TextIOWrapper’ object”,)

in pmap at [base\distributed\pmap.jl:156](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in pmap#213 at [base\distributed\pmap.jl:156](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in pmap at [base\distributed\pmap.jl:101](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in pmap#203 at [base\distributed\pmap.jl:126](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in at [base&lt;missing>](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in at [base&lt;missing>](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in #async_usemap#560 at [base\asyncmap.jl:103](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in wrap_n_exec_twice at [base\asyncmap.jl:154](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in maptwice at [base\asyncmap.jl:178](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in foreach at [base\abstractarray.jl:1733](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

in #575 at [base\asyncmap.jl:178](file:///C:/Users/username/AppData/Local/atom/app-1.34.0/resources/app.asar/static/index.html)

Does anyone have any idea about what this is due to?
By the way, also with addprocs(1) instead of addprocs(2) the situation is the same.
Thank you very much in advance!

4

Why not just use Threads.@threads?