Hi. I have an application to create an Array of tasks and then launch them asynchronously (preferably in parallel). The time of launching is not necessarily the time of creation hence @spawn is not ideal.
Is there a macro like @task that creates non sticky tasks? Right now, I create tasks kind of this way.
Fill the array of tasks. Currently I am letting them be sticky if the original task is sticky. Is there a map like function that could in place modify attributes of elements of an array, so I could make them all non sticky in one line?
Then I do
Schedule.(TaskArray)
Also if I have a multidimensional array of Tasks, what is the scheduling order.
Say TaskArray = fill(@task myfunction(argument),dim1,dim2). Would it traverse dim1_idx = 1:dim1 first for dim2 = 1, then do the same for dim2 = 2…, and so on till n dims?
The documentation is a little hard to follow. Are there other ways to create tasks instead of the @ task macro where we specify these attributes?
Tasks have no reliable scheduling order. However there is a “depth-first” kind of scheduling, i.e. if a Task spawns further Tasks then these are prioritized. See
you can use foreach
tasks = [...]
foreach(t->t.sticky = false, tasks)
No there is no built-in for that. You’ll need to set the stick flag manually or create new marcro that does that for you, e.g.
macro mytask(args)
return quote
let t = @task($args)
t.sticky = false
t
end
end
end
t = @mytask println(5)
@assert !t.sticky
It feels like this might a XY problem kind of question because your questions seem a bit odd and oddly specific to me.
You seem to care about performance. However having lots of (small?) tasks is not very efficient due to overheads. If you create tasks in a multi-dimensional array, then it would probably be more efficient to instead split the indices and create only a limited amount of tasks that each cover some range of indices.
If you describe what problem you try to solve, then perhaps you can get better guidance. So feel invited to share more about the origin of these questions.
The tasks are not particularly small. I have a 2 D task array of size about n x y (where n is a number between 1 and 5 typically, and y is typically 6).
The tasks are assigned in this manner, all dim2 elements for a dim1 element is populated ie Tasks[idx,:] is populated one at a time along with a validity checker. Once all tasks are populated and all validity checks are true, all tasks could be run in parallel across any dimension.
The reason to prefer scheduling in Tasks[:,idx] order first is that these tasks are pre processing steps for another set of operations that occur in sequence after each Tasks[:,idx] is complete. And then there is another set of post processing tasks that are to be scheduled (can be in parallel like Tasks, but provided the required operations Tasks[:,idx] and the operations after that is complete). The flow can be summarized as
For your setting, a pretty standard construction is to use backpressure. That is: You emplace a limit that PreProcess(i) cannot start before Process(i-n1) is done; and you limit that Process(i) cannot start until PostProcess(i-n2) is done, with generous values of n1 and n2; otherwise a job is scheduled once its other prerequisites are done (PreProcess(i) and Process(i-1) for Process(i), Process(i) for PostProcess(i)).
This is especially if any of your steps expand memory usage. E.g. if PreProcess(i) creates a large array that only gets free’d once Process(i) or PostProcess(i) is done with it.
You don’t really need to create most of the tasks before scheduling them: Process(i) can create the tasks for Process(i+1), PostProcess(i) and PreProcess(i+n1), and can manage the vectors of tasks (since it’s effectively under a mutex anyway, no extra locks required).
Regardless of how you end up coding the thing, you need to make sure that the back-pressure exists, in order to prevent a situation where you first spend lots of like doing all the PreProcess, and then linearly do the Process / PostProcess in order, using only a single core.