Worker Proc Failing to Iterate JuliaDB Table

Specific Domains Julia at Scale
, ,
1

I am getting an error when I try to iterate through a table on worker 2. I can open the table on worker 2 and println the table, but iteration which I can normally do in a serial process is failing. It errors on line 25: for x in tblORD.

Code:

using Distributed, JuliaDB

#add worker processes
addprocs(1) 

@everywhere using JuliaDB

#set constants
const jobs = RemoteChannel(()->Channel{Int}(32))
const results = RemoteChannel(()->Channel{Tuple}(32))

#set local variables
n = 1

@everywhere function do_work(jobs, results) # define work function everywhere
        job_id = take!(jobs)

        #load files
        tblORD = loadtable("C:\\JL\\New_Test\\ORD.csv",indexcols=[1])

        #set local variables
        storeList = []

        #loop through the order file to create an array of the stores on order
        for x in tblORD
            push!(storeList, x.STORE)
        end

        exec_time = rand()
        put!(results, (job_id, exec_time, myid()))
end

function make_jobs(n)
    for i in 1:n
        put!(jobs, i)
    end
end

@async make_jobs(n) # feed the jobs channel with "n" jobs

for p in workers() # start tasks on the workers to process requests in parallel
    remote_do(do_work, p, jobs, results)
end

@elapsed while n > 0 # print out results
    job_id, exec_time, where = take!(results)
    println("$job_id finished in $(round(exec_time; digits=2)) seconds on worker $where")
    global n = n - 1
end

rmprocs(workers())

Error:

From worker 2:    MethodError: no method matching iterate(::JuliaDB.DIndex
edTable{NamedTuple{(:STORE, :QTY, :UL, :W_START, :W_END),NTuple{5,Int64}},NamedT
uple{(:STORE,),Tuple{Int64}}})
From worker 2:    Closest candidates are:
From worker 2:      iterate(!Matched::Core.SimpleVector) at essentials.jl:
589
From worker 2:      iterate(!Matched::Core.SimpleVector, !Matched::Any) at
essentials.jl:589
From worker 2:      iterate(!Matched::ExponentialBackOff) at error.jl:171
From worker 2:      ...
From worker 2:    do_work(::RemoteChannel{Channel{Int64}}, ::RemoteChannel
{Channel{Tuple}}) at C:\JL\testing123.jl:25
From worker 2:    (::getfield(Distributed, Symbol("##120#122")){Distribute
d.RemoteDoMsg})() at C:\cygwin\home\Administrator\buildbot\worker\package_win64\
build\usr\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:282
From worker 2:    run_work_thunk(::getfield(Distributed, Symbol("##120#122
")){Distributed.RemoteDoMsg}, ::Bool) at C:\cygwin\home\Administrator\buildbot\w
orker\package_win64\build\usr\share\julia\stdlib\v1.0\Distributed\src\process_me
ssages.jl:56
2

I noticed that the table is a distributed table ( JuliaDB.DIndexedTable).

I found a discussion which seems to indicate that iteration is not possible on a distributed table.
https://github.com/JuliaComputing/JuliaDB.jl/issues/231

In my case though, I am not should why the table has to be distributed. I want each process to run on a single core with separate tables running in each process. I don’t need to distribute the tables across multiple processes. Is there a way to work with multiple tables concurrently without making them “Distributed”?