Help with setting up Distributed computing

General Usage Performance
,
5

Depending on if you go with threads or distributed it might look a little different, but I think the main idea for both could be to have the main loop just spawn jobs as long as there are jobs in the queue, and in case there are jobs running but the queue is empty you wait for a condition. The block with job processing would notify to the same condition when adding new nodes to the queue. The queue would then have to be either thread safe, using locks as you did in the last post, or if distributed using something like remote channel.

I would probably go with threads if you don’t need the distributed.
I haven’t put proper locks on any of the lines I added here, so you will have to think through that. Just a quick sketch of how I think it could be done.

queue = Vector{Node}()
push!(queue, rootNode)
bestKnownSolution = 100
queueLock = ReentrantLock()
running_jobs = 0
new_job = Condition()

while true
    if length(queue) + running_jobs == 0
        break;
    end
    if  isempty(queue)
        wait(new_job)
    end
    @async begin
        nextNode = pop!(queue) #also locked like below
        running_jobs += 1
        newNode1, newNode2, bestFoundSolution = processNode!(nextNode)
        running_jobs -= 1 
        if !isnothing(newNode1)
            begin
                lock(queueLock)
                try
                    if !isempty(queue)
                        addToSortedQueue!(newNode1)
                        notify(new_job)
                    end
                finally
                    unlock(queueLock)
                end
            end
        end
        if !isnothing(newNode2)
            #same deal with the lock
        end
        if isBetter(bestFoundSolution, bestKnownSolution)
            #same deal with the lock
        end
    end
end

return bestKnownSolution
2 Likes