Help with data movement and commuication

Specific Domains Julia at Scale
1

Hello,

I’ve been struggling for a long time with an apparently simple problem. Given a graph data structure with nodes inside I want to apply a function to each node that shares information across nodes and modifies the node in place.

Here is an example of what I want to achieve

Begin GraphModule.jl

module GraphModule

export MyGraph, MyNode, nodefunction

mutable struct MyGraph
    nodes
end

mutable struct MyNode
    label
    inbox
    child
    value
end

function nodefunction(n::MyNode)
    val = n.inbox + 1
    push!(n.value, val)
    n.child.inbox = val
    println("I am node $(n.label), from $(myid()), my values are $(n.value)")
end

end

Begin simpletest.jl

push!(LOAD_PATH,".")
addprocs(2)

using GraphModule

g = MyGraph([MyNode("parent", 0, nothing, Float64[]),
             MyNode("child", 0, nothing, Float64[])])

g.nodes[1].child = g.nodes[2]
g.nodes[2].child = g.nodes[1]

for i in 1:5
    for node in g.nodes
        nodefunction(node)
    end
end

Here is the list of things I want to achieve

  1. Move each node to a different worker
  2. Apply nodefunction to each node in the worker
  3. nodefunction does three things:
    3.1. Adds 1 to the value received in the inbox
    3.2. Puts that value in its child inbox
    3.3. Stores the value inside the node in the value property

I also want that the operation is performed in the same worker for a given node, because the data inside the node can be extensive and I want to minimize data movement.

Could someone be so kind to help me with this?

Thank you

2

I guess your question involves an efficient algorithm, which I’m not qualified to answer. The practical side of the question can probably be solved with ParallelDataTransfer.jl