# Dynamic distributed execution for data parallel tasks in Julia

**URL:** <https://discourse.julialang.org/t/dynamic-distributed-execution-for-data-parallel-tasks-in-julia/23596>\
**Category:** General Usage\
**Tags:** question\
**Created:** [April 27, 2019, 12:47pm UTC](https://discourse.julialang.org/t/dynamic-distributed-execution-for-data-parallel-tasks-in-julia/23596 "2019-04-27T12:47:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zecx](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zecx](https://discourse.julialang.org/u/Zecx)\
**Post date:** [April 27, 2019, 12:47pm UTC](https://discourse.julialang.org/t/dynamic-distributed-execution-for-data-parallel-tasks-in-julia/23596/1 "2019-04-27T12:47:21Z")

</div>

I was assigned a homework.  
**Task** : Dynamic distributed execution for data parallel tasks in Julia.

This project proposal is to implement a native Julia framework for distributed execution for general purpose data parallelism, using dynamic, runtime-generated general task graphs which are flexible enough to describe multiple classes of parallel algorithms. You will be expected to weave together native Julia parallelism constructs such as the `ClusterManager` for massively parallel execution, and automate the handling of data dependencies using native Julia `RemoteRefs` as remote data futures and handles. You will also be encouraged to experiment with novel scheduling algorithms.

This project is from http://www\_old.julialang.org/soc/projects/hpc.html  
But I have no idea.  
Welcome everyone’s advice.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [April 27, 2019, 11:17pm UTC](https://discourse.julialang.org/t/dynamic-distributed-execution-for-data-parallel-tasks-in-julia/23596/2 "2019-04-27T23:17:05Z")

</div>

Implemented already:

> **[GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel...](https://github.com/JuliaParallel/Dagger.jl)**
>
> A framework for out-of-core and parallel execution - GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel execution

although there’s plenty of room for improvements, possibly around adding more kinds of scheduling algorithms.

And if I may ask, who is assigning you Julia GSoC ideas as homework??? 😛

---

<div class="post-metadata">

**Author:** ![Zecx](https://avatars.discourse-cdn.com/v4/letter/z/b782af/32.png) [@Zecx](https://discourse.julialang.org/u/Zecx)\
**Post date:** [April 28, 2019, 3:03am UTC](https://discourse.julialang.org/t/dynamic-distributed-execution-for-data-parallel-tasks-in-julia/23596/3 "2019-04-28T03:03:48Z")

</div>

thx