Macro to turn on/off parallelisation

General Usage
, ,
1

I want to define a macro, @parallel, to turn on/off parallelisation in for loops, depending on a command-line argument. Here, parallelisation is embarassing and uses Threads.@threads.

Why would I want to do this? Because I want to fully turn off multi-threading when checking for unwanted allocations. And Threads.@threads has spurious allocations.

Here is a MWE:

# Reading the command-line arguments
using ArgParse
tabargs = ArgParseSettings()
@add_arg_table! tabargs begin
    "--parallel"
    help = "Parallelisation: true/false"
    arg_type = Bool
    default = false
end
parsed_args = parse_args(tabargs)
const PARALLEL = parsed_args["parallel"]
#####
if PARALLEL
    macro parallel(ex::Expr) # We want to use multiple threads
        return :( Threads.@threads $(ex) )
    end
else
    macro parallel(ex::Expr) # No multi-threading
        return :( $(ex) )
    end
end
#####
# Using the macro in a function
function run!()
    @parallel for i=1:2
        sleep(1)
    end
end
#####
@time run!()

The code can then be tested in various regimes via
julia code.jl --parallel false (Time: 2s)
julia code.jl --parallel true (Time: 2s)
julia -t 1 code.jl --parallel false (Time: 2s)
julia -t 1 code.jl --parallel true (Time: 2s)
julia -t 2 code.jl --parallel false (Time: 2s)
julia -t 2 code.jl --parallel true (Time: 1s)

Hence, the definition of the macro seems (reasonably) correct. Yet, when used in my real test case, it leads to unexpected errors. I do not get these errors when using directly Threads.@threads. So, @parallel seems badly defined.

What would be the most idiomatic (and correct) way of implementing this macro in julia?

2

I’m not by my computer so forgive me if this code isn’t quite right. But you could do something like


Threads.foreach(iter; nworkers=PARALLEL ? Threads.nthreads() : 1) do x
# iter code here
end

No macro needed!

3

Unfortunately, the syntax of Threads.foreach is indeed a bit different:

Threads.foreach(f, channel::Channel;
                  schedule::Threads.AbstractSchedule=Threads.FairSchedule(),
                  ntasks=Threads.threadpoolsize())

I’m sure you can use this with a proper Channel and switch between sequential and parallel processing via ntasks, but Threads.foreach seems a bit complicated.

1 Like
4

I forgot it was for channels only!

5

It’s hard to find the problem when the code you post runs fine. :slight_smile:

My guess is that the issue you’re encountering is related to macro hygiene. For example,

macro parallel(ex::Expr)
    return :(Threads.@threads $ex)
end

function run!()
    sleep_time = 1.
    @parallel for i = 1:2
        sleep(sleep_time)
     end
end

run!()

throws an error:

ERROR: TaskFailedException

    nested task error: UndefVarError: `sleep_time` not defined
    ...

Changing to macro to

macro parallel(ex::Expr)
    return esc(:(Threads.@threads $ex))
end

and keeping the rest of the code the same, everything now runs just fine.

2 Likes
6

A somewhat similar approach is suggested here by oxinabox: use Base.foreach for sequential and ThreadsX.foreach for parallel execution (and automatically switch based on the value of PARALLEL).

@jibe Check out the linked topic: it deals with the same question and has some more answers.

3 Likes
7

One more thought: if you want to use the function proposed in that link, you could still compile away the if statement without the macro.

if PARALLEL
    _foreach =ThreadsX.foreach
else
    _foreach = Base.foreach
end

Now, you can use _foreach everywhere, and it’ll reference the correct function.

That said, You probably shouldn’t be making a threading decision in a very hot part of the code, so compiling away that if doesn’t really matter.

2 Likes
8

Fantastic! This does get the job done, no errors are left, and I got numerous links/references to mull over :wink:
In practice, for the case PARALLEL == false, should I also make the change to

macro parallel(ex::Expr) # No multi-threading
    return esc(:( $(ex) ))
end

?

1 Like
9

Thank you very much for your suggestion. Unfortunately, in the case PARALLEL == false, although the code would use a single thread, it would use Threads.foreach and not Base.foreach, hence making unwanted allocations?

10

One last newbie question :slight_smile:
In that piece of code, is it a good practice to add a const [in particular if in global scope] to read

if PARALLEL
    const _foreach = ThreadsX.foreach
else
    const _foreach = Base.foreach
end

This may prevent useless allocations when calling _foreach?

11

Not a newb question – you’re absolutely right that you should do that!

Even if it didn’t affect allocations, always good to const a global if you don’t intend to change it.

I originally wrote this as defining a new function which calls either base or ThreadsX. Then change my mind because that would be an unnecessary pollution of the stack trace, But when I switched to just aliasing the functions, I forgot to add const.

1 Like
12

I don’t understand your question here. In the case where PARALLEL is false, it will call Base.foreach, which is serial. Could you clarify what you mean?

1 Like
13

You’re right and my wording was confusing – my apologies. Here, I was referering to your initial attempt reading

Threads.foreach(iter; nworkers=PARALLEL ? Threads.nthreads() : 1) ...

There, one always uses Threads.foreach, even for PARALLEL == false. I believe that this would not have been ideal. Indeed, a threaded loop with a single thread is not the same as a “base” loop. Would you agree?

14

Indeed. You can again check that

macro parallel(ex)
    return ex
end

would throw an ERROR: UndefVarError: `sleep_time` not defined, because all variables in ex, in casu sleep_time, have essentially been automatically made local, so that Julia does not recognise them as coming from the surrounding scope.

macro parallel(ex)
    return esc(ex)
end

(which is the same as your esc(:($(ex))), but a bit shorter) will again run fine.

1 Like
15

Yeah, I think you should forget my first message for two reasons:

  1. As you pointed out, it will still spawn tasks. On single threaded.
  2. More importantly, it only works with Channels.
1 Like
16

FWIW, OhMyThreads.jl has a SerialScheduler that you can pass to tforeach and co that turns off any multihreading.

2 Likes