Addprocs: specify both multithreading and active environment

Specific Domains Julia at Scale
1

Let’s say I want to add a process with 8 threads. This is easy:

using Distributed
addprocs(1, exeflags="-t 8")

One can check that @fetchfrom 2 Threads.nthreads() returns 8 as expected.

But, if I try to specify the environment as well, only one thread appears.

using Distributed
addprocs(1, exeflags="--project=$(Base.active_project()) -t 8")
@fetchfrom 2 Threads.nthreads()

Is it a bug, or am I doing something wrong?

2

I found out this can work:

julia> addprocs(1, exeflags=["--threads=4", "--project=$(Base.active_project())"])

It’s maybe not strictly a bug, or then a doc bug, I would still file an issue.

What I wrote before figuring out the above (and you may want this to work, or the docs explicit this will not work):

The s in exeflags implies to me more than one allowed, and I tried switching around:

julia> addprocs(1, exeflags="--threads=4 --project=$(Base.active_project())")
ERROR: julia: -t,--threads=<n> must be an integer >= 1
ERROR: TaskFailedException

    nested task error: Unable to read host:port string from worker. Launch command exited with error?
[stacktrace skipped]

It seems nonsensical, as in the parser trying to split up -t and the --project part not working. I guess in your case -t was just ignored. Is there a different way to invoke this?

1 Like
3

I think the problem is that exeflags is supposed to be a Cmd (this was documented incorrectly for a while, and either way isn’t explained in the docstrings). The code does happen to work for a String if this String only contains a single option, otherwise it’s problematic.

In other words, the following works:

using Distributed
addprocs(1, exeflags=`--project=$(Base.active_project()) -t 8`)
@fetchfrom 2 Threads.nthreads()