I am trying to parallelize solve_alpha from Xfoil.jl. The problem is: Xfoil.jl has global variables inside the package, so I cannot use @threads to run multiple solve_alphas at the same time. I have to use Distributed.jl with @everywhere to make multiple instances of Xfoil.jl. This works in a script, but it doesn’t work when I try to use @everywhere inside a module. And I want to be able to run the code inside a module.
The script that works:
using Distributed
using Xfoil
const procs = addprocs()
@everywhere begin
using Xfoil
function solve_alpha(x, y)
Xfoil.set_coordinates(x, y)
Xfoil.solve_alpha(0.0; mach=0.1)
return nothing
end
end
try
x = collect(0:0.1:1)
y = x .^ 2
@show x y
@sync @distributed for j in 1:10
solve_alpha(x, y)
end
catch e
println(e)
finally
println("removing processes")
rmprocs(procs)
end
The module that doesn’t work:
module DistributedXfoil
using Distributed
using Xfoil
const procs = addprocs()
@everywhere begin
using Xfoil
function solve_alpha(x, y)
Xfoil.set_coordinates(x, y)
Xfoil.solve_alpha(0.0; mach=0.1)
return nothing
end
end
try
x = collect(0:0.1:1)
y = x .^ 2
@show x y
@sync @distributed for j in 1:10
solve_alpha(x, y)
end
catch e
println(e)
finally
println("removing processes")
rmprocs(procs)
end
end
I found a solution. You can run a script from a module using @eval Main include("polar_exec.jl")
"src/DistributedXfoil.jl"
module DistributedXfoil
using Distributed
using Xfoil
export init
function init()
@eval Main include("polar_exec.jl")
end
end
using .DistributedXfoil
init()
"src/polar_exec.jl"
using Distributed
using Xfoil
const procs = addprocs()
@everywhere begin
using Xfoil
function solve_alpha(x, y)
Xfoil.set_coordinates(x, y)
Xfoil.solve_alpha(0.0; mach=0.1)
return nothing
end
end
try
x = collect(0:0.1:1)
y = x .^ 2
@show x y
@sync @distributed for j in 1:10
solve_alpha(x, y)
end
catch e
println(e)
finally
println("removing processes")
rmprocs(procs)
end
I mean, that might work in some situations, but it’s really not the solution you want.
You generally want modules to only define the machinery that they need. And then you create functions that can be called to run that machinery.
Detangling these things is a very good thing. Modules are generally setup such that they can be defined at any time. And then they have functions that do the interesting work.
Or you can have scripts — external to the module — that run the functions defined by the module.
But in this case, it is a way of automatically running the script from the Module. In the real module, I add checks to see if the script should run, and only run it if it needs to.
The script is still external to the model. And the user can run it themselves. But now the running of the script is automated. Maybe it would be better to do this automation in a bash script. But then the user has to run the bash script on startup instead of just running Julia.
No, because @everywhere has to be called in the global scope. Wrapping in a function causes an error. Feel free to experiment with the solution I provided, but I wasn’t able to wrap it in a function.
It doesn’t help. Using @eval @everywhere I get errors with undefined variables without names or undefined modules. The solution I provided is the only one that I got working.
module DistributedXfoil
using Distributed
using Xfoil
export init
function init()
# @eval Main include("polar_exec.jl")
procs = addprocs()
@eval @everywhere begin
using .DistributedXfoil, Xfoil
function solve_alpha(x, y)
Xfoil.set_coordinates(x, y)
Xfoil.solve_alpha(0.0; mach=0.1)
return nothing
end
end
try
x = collect(0:0.1:1)
y = x .^ 2
@show x y
@sync @distributed for j in 1:10
solve_alpha(x, y)
end
catch e
println(e)
finally
println("removing processes")
rmprocs(procs)
end
end
end
using .DistributedXfoil
init()
Not sure why you put the .DistributedXfoil there. Just remove that and it should fix the error you are seeing. Unfortunately, I cannot test right now, so I don’t know whether it truly works then.
If you wanted to include the “local” package into each worker, you need a different setup to ensure that the workers can find it.
I guess the question is why do you want to put the @everywhere inside a module? The whole point of @everywhere is to mess with Main. Modules generally shouldn’t be messing with other modules like that.
I would suspect that the pain point you’re trying to solve is that you want users to just call X.run_in_parallel() and not need to worry about addprocs or worker state. But the control of workers and worker state is very much a global sort of thing.
Good summary. I don’t want the user to think about this parallel stuff, the code needs to “just work” and “be fast”. @threads is not an option, as Xfoil.jl doesn’t support it (there are global variables in the Xfoil module, and each thread would need its own instance of the Xfoil module).
