Hi,
I am receiving a tuple from a channel, (a remote channel to be precise) and I would like to add those data to a particular data structure I have.
This data structure is composed of arrays, and I would like to append the incoming vectors to these arrays, since they are not equal in length I cannot do a matrix.
Also, and most important, I would like to not store that data in a temporary variable! I have not managed to understand if the cycle for i in data
actually allocate the i
element, but I want to think that it does not.
So this solution would be optimal in terms of data allocated, but very bad for the who read it.
Can you suggest me something more julian?
Thanks,
Alessio
using Distributed
using BenchmarkTools
const jobs = RemoteChannel(() -> Channel{Tuple}(32))
const results = RemoteChannel(() -> Channel{Tuple}(32))
const save_output = "save_output.h5"
if isfile(save_output)
rm(save_output)
end
@everywhere function do_work(jobs, results, save_output="save_output.h5")
local1 = zeros(10)
local2 = zeros(20)
locals = (local1,local2)
jobs_id = 1
while true
index =0
### this is the critical cycle, solved in the ugliest way possible
### (please notice I have to subtract the element to keep tuple bounds)
for data in take!(jobs)
index+=1
if index ==1
jobs_id = data
else
append!(locals[index-1],data)
end
end
# end
# h5write(save_output, "time_step"*string(jobs_id), jobs_id)
put!(results, (jobs_id, myid(),length(local1), length(local2)))
end
end
# @everywhere do_work(jobs, results)= do_work(jobs, results, save_output)
function make_jobs(n, jobs)
for i in 1:n
put!(jobs, (i,ones(10)*i,ones(20)))
end
end;
n = 10;
channels = (jobs, results)
addprocs(2)
@async make_jobs(n, jobs); # feed the jobs channel with "n" jobs
for p in workers()
remote_do(do_work, p,channels...)
# @spawnat p do_work(jobs, results)
end
@elapsed while n > 0 # print out results
job_id, proc, array1, array2 = take!(results)
println("$job_id finished on worker $proc with array length: $array1, $array2")
global n = n - 1
end