Discussion continuing from here: https://github.com/JuliaLang/julia/issues/22341#issuecomment-308134991
Invenia is working on DaskDistributedDispatcher.jl. We are using DeferredFutures to store the results of computations and allow other workers to get the results of those computations. In Dispatcher, we create all the DeferredFutures at the beginning and pass them to the workers. The idea is to have the results available for fetching by any worker and only fetch them once it becomes apparent that they are needed on that worker.
With dask.distributed, we are expected to be able to transfer the data through MsgPack from the client (Julia, ours) to the scheduler (Python, theirs), to the workers (Julia, ours). The serialized inner RemoteChannels only live for (at maximum) the duration of the cluster’s lifetime, and are never actually used by the scheduler. We would like to be able to serialize DeferredFutures/RemoteChannels for that purpose.