How to use Dagger in remote machines?

Specific Domains Julia at Scale
1

I began using Dagger.jl and I want to run some tests it with my local host machines, but those tests only runs in multithreading in my notebook.

I know that my local machines work, because I can run them with Distributed.jl and everything goes according to expected. Then I wish to know an equivalent syntax with Dagger.jl

# (...) load packages and add process (...)

# WORKS: this piece of code runs into my machines
map(1:10) do i
    Distributed.@spawn println("inside machine: $(pwd())")
end;

# HERE is the part that I need help
map(1:10) do i
    Dagger.@spawn println("inside machine: $(pwd())")
end;

As far I am aware, I should use Dagger.tochunk in some way and Scopes, but I got a lot of error messages, and I don’t even know what I’m doing.

To avoid more confusing, I’m here asking for some basic guidance, and maybe help people in the future with the same problem will benefit from reading the answer.

Thank you

2

Can you please provide the error and stacktrace that you receive? Also, Dagger.@spawn expects just a single function call right now (will change once I address `@spawn` has confusing move semantics · Issue #357 · JuliaParallel/Dagger.jl · GitHub), but you have println, string interpolation (which is a call to Base.string), and pwd. You should instead write this as:

f() = println("inside machine $(pwd())")
map(1:10) do i
    Dagger.@spawn f()
end;
3

Thank you, your example works once I add an everywhere in front of the function name.

What I was trying to do, was to modify the example of Union Scopes, but the command Dagger.thunk_processor() does not work. The error message is:

using Distributed, Dagger
@everywhere using Distributed, Dagger

> Dagger.thunk_processor()
ERROR: KeyError: key :_dagger_processor not found
Stacktrace:
 [1] getindex
   @ ./iddict.jl:108 [inlined]
 [2] task_local_storage
   @ ./task.jl:270 [inlined]
 [3] thunk_processor()
   @ Dagger ~/.julia/packages/Dagger/y0NVO/src/processor.jl:304
4

Dagger.thunk_processor() is intended to be used within a thunk, such as when defining f: f() = println("running on processor $(Dagger.thunk_processor())"). Using it outside of a Dagger thunk doesn’t work because Dagger doesn’t have any control of which processor is being used.