JuliaDB with Parallel Procs Not Working in JuliaPro

I’m using Julia 1.0.3. How do I get JuliaDB with work with parallel procs? When I run the below code I get an error that the package is not available on worker 2.

Code:

using Distributed

while length(procs()) < 2
    addprocs(1) # add worker processes
end

@everywhere using JuliaDB

const jobs = RemoteChannel(()->Channel{Int}(32))
const results = RemoteChannel(()->Channel{Tuple}(32))

#set local variables
n = 1

@everywhere function do_work(jobs, results) # define work function everywhere
    while true
        job_id = take!(jobs)
        exec_time = rand()
        sleep(exec_time) # simulates elapsed time doing actual work
        put!(results, (job_id, exec_time, myid()))

        #########################################################
        #########################################################

        #load file
        tblDT = loadtable("C:/JL/New_Test/DT_INT.csv",indexcols=[1,2])

        #########################################################
        #########################################################
    end
end

function make_jobs(n)
    for i in 1:n
        put!(jobs, i)
        println("make jobs ",i)
    end
end

@async make_jobs(n) # feed the jobs channel with "n" jobs

for p in workers() # start tasks on the workers to process requests in parallel
    remote_do(do_work, p, jobs, results)
    println("p workers ",p)
end

@elapsed while n > 0 # print out results
    job_id, exec_time, where = take!(results)
    println("$job_id finished in $(round(exec_time; digits=2)) seconds on worker $where")
    global n = n - 1
end

Error Message:

ERROR: LoadError: On worker 2:
ArgumentError: Package JuliaDB not found in current path:
- Run `import Pkg; Pkg.add("JuliaDB")` to install the JuliaDB package.

require at .\loading.jl:823
eval at .\boot.jl:319
#116 at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Varar{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Disributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} wher N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\emotecall.jl:433
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at .\task.jl:259

...and 1 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at .\task.jl:245 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:206
 [4] top-level scope at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:190
in expression starting at C:\JL\MultiThreads0001.jl:7

Is the error thrown in this line @everywhere using JuliaDB?

Try restarting Julia, run import Pkg; Pkg.add("JuliaDB"); using JuliaDB and let it precompile. Restart and run your code.

I restarted and ran import Pkg; Pkg.add("JuliaDB"); using JuliaDB. I still get the same error. I am able to use JuliaDB just fine, but I want to use it on multiple processes simultaneously. The error is happening on worker 2. It is like the second process cannot find the path to the JuliaDB package.

I have trouble reproducing the error. What JuliaDB version are you using?

If you just run:

using Distributed, JuliaDB

while length(procs()) < 2
    addprocs(1) # add worker processes
end

@everywhere using JuliaDB

Is the error occurring?

Yes. I do get an error when I run that. Same error.

Please, report version numbers from Pkg.status().

Maybe upgrade to Julia 1.1.1 and JuliaDB 0.12.0 and try again?

I want to stay on the stable, long-term supported version on Julia. I am on JuliaDB v12.

julia> julia> Pkg.status()
    Status `C:\Users\[REDACTED]\.juliapro\JuliaPro_v1.0.3.2\environments\v1.0\Project.toml`
  [c52e3926] Atom v0.8.5
  [9961bab8] Cbc v0.6.0
  [e2554f3b] Clp v0.6.2
  [60bf3e95] GLPK v0.9.1
  [3c7084bd] GLPKMathProgInterface v0.4.4
  [7073ff75] IJulia v1.18.1
  [4076af6c] JuMP v0.19.0
  [a93385a2] JuliaDB v0.12.0
  [e5e0dc1b] Juno v0.7.0
  [429524aa] Optim v0.18.1
  [91a5bcdd] Plots v0.24.0
  [8ba89e20] Distributed
  [37e2e46d] LinearAlgebra

I also get the same error with other packages. For instance, I tried Plots and also got the error on worker 2.

This thread suggests I need to push stuff onto the LOAD_PATH:

1 Like

I actually think this should be DEPOT_PATH instead.

Do you know what path I am supposed to push? DEPOT_PATH did not work, but I am having some kind of progress with LOAD_PATH. The error message changed, and I am getting errors from worker 2 now.

Code:

using Distributed, JuliaDB

while length(procs()) < 2
    addprocs(1) # add worker processes
end

#@everywhere push!(DEPOT_PATH, "C:\\Users\\[REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\packages\\JuliaDB\\jDAlJ\\src")
@everywhere push!(LOAD_PATH, "C:\\Users\\[REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\packages\\JuliaDB\\jDAlJ\\src")

@everywhere using JuliaDB

Error:

      From worker 2:    ERROR: LoadError: ArgumentError: Package IndexedTables not found in current path:
      From worker 2:    - Run `import Pkg; Pkg.add("IndexedTables")` to install the IndexedTables package.
      From worker 2:
      From worker 2:    Stacktrace:
      From worker 2:     [1] require(::Module, ::Symbol) at .\loading.jl:823
      From worker 2:     [2] include at .\boot.jl:317 [inlined]
      From worker 2:     [3] include_relative(::Module, ::String) at .\loading.jl:1044
      From worker 2:     [4] include(::Module, ::String) at .\sysimg.jl:29
      From worker 2:     [5] top-level scope at none:2
      From worker 2:     [6] eval at .\boot.jl:319 [inlined]
      From worker 2:     [7] eval(::Expr) at .\client.jl:393
      From worker 2:     [8] top-level scope at .\none:3
      From worker 2:    in expression starting at C:\Users\[REDACTED]\.juliapro\JuliaPro_v1.0.3.2\packages\JuliaDB\jDAlJ\src\JuliaDB.jl:7
ERROR: LoadError: On worker 2:
Failed to precompile JuliaDB [top-level] to C:\Users\[REDACTED]\.julia\compiled\v1.0\JuliaDB.ji.
error at .\error.jl:33
compilecache at .\loading.jl:1203
_require at .\loading.jl:960
require at .\loading.jl:858
require at .\loading.jl:853
eval at .\boot.jl:319
#116 at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Varar{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Disributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} wher N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\emotecall.jl:433
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at .\task.jl:259
Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at .\task.jl:245 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:206
 [4] top-level scope at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:190
in expression starting at untitled-d3683fe8e6936bb5b606f0e41275b73b:10

well it’s the same error. worker 2 cannot find your packages. what is your setup? multiple workers on one machine or several machines (like, 2?) with a shared file system?
i have no experience with juliapro at all. if you can try out with basic julia, that works for me out of the box.

@floswald and @MaximilianJHuber thank you for the help. I downloaded Julia 1.1.1 and the abbreviated code worked.

I ran the full code at the top of the post. It is almost working. It says the homePath is not defined. It almost works now. I think this is why I need to push to LOAD_PATH or DEPOT_PATH.

Error:

      From worker 2:     UndefVarError: homePath not defined
      From worker 2:    do_work(::RemoteChannel{Channel{Int64}}, ::RemoteChannel {Channel{Tuple}}) at .\REPL[9]:12
      From worker 2:    (::getfield(Distributed, Symbol("##120#122")){Distributed.RemoteDoMsg})() at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:282
      From worker 2:    run_work_thunk(::getfield(Distributed, Symbol("##120#122")){Distributed.RemoteDoMsg}, ::Bool) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:56

I think I am going to close this as answered. I don’t know why Julia 1.0.3 is not working for me. Julia 1.1.1 appears to be working the best.

julia> using Distributed

julia> using JuliaDB

julia> addprocs(1)
1-element Array{Int64,1}:
2

julia> @everywhere using JuliaDB

julia> r = remotecall(loadtable, 2, β€œC:/JL/New_Test/ORD.csv”)
Future(2, 1, 7, nothing)

julia> fetch( r )
Distributed Table with 77 rows in 1 chunks:
STORE QTY UL W_START W_END
──────────────────────────────
44514 2 29 3360 3480
44513 2 38 3300 3420
44507 2 41 3360 3480
44489 2 36 3285 3330
21630 2 41 3360 3600
21627 2 41 3330 3570
44509 3 37 3360 3480
44495 3 36 3285 3330
44490 3 50 3285 3345
44479 3 42 3255 3330
21704 3 48 3330 3570
44487 4 47 3195 3270
44121 4 37 3330 3570
44109 4 58 3330 3570
44104 4 38 3360 3600
44098 4 43 3360 3600
43953 4 38 3330 3570
43817 4 38 3360 3600
43802 4 41 3360 3600
?

julia>

You might find it useful to compare the output of Base.load_path() on master vs worker:

Base.load_path()
remotecall_fetch(Base.load_path, 2)

Your responses below might suggest that pushing the location of a package on to LOAD_PATH does indeed work, but you were too specific. The worker can now find JuliaDB, but not surprisingly can’t find IndexedTables within the JuliaDB sub-directory.

Perhaps try pushing C:\\Users\\REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\packages instead.

I did try pushing the less specific C:\\Users\\[REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\packages and, unfortunately, the error message reverts to not being able to find the JuliaDB package at all.

I checked the Base.load_path() and compared to the worker. There is one path that is different between the two.

Master:
"C:\\Users\\[REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\environments\\v1.0\\Project.toml"

Worker 2:
"C:\\Users\\[REDACTED]\\.julia\\environments\\v1.0\\Project.toml"

I am not sure why worker 2 would be using a path to the .julia environment instead of .juliapro. Maybe this was my first installation, so it is some kind of default path. Not sure.

I pushed "C:\\Users\\[REDACTED]\\.juliapro\\JuliaPro_v1.0.3.2\\environments\\v1.0\\Project.toml" to worker 2’s LOAD_PATH (keeping the less specific "C:\\Users\\dburch.CSWG\\.juliapro\\JuliaPro_v1.0.3.2\\packages" as well. This sort of worked. It got me back to the point where I’m getting messages from worker 2 and JuliaDB is failing to precompile.

Current Test Code:

using Distributed, JuliaDB

while length(procs()) < 2
    addprocs(1) # add worker processes
end

@everywhere push!(LOAD_PATH, "C:\\Users\\dburch.CSWG\\.juliapro\\JuliaPro_v1.0.3.2\\packages")
@everywhere push!(LOAD_PATH, "C:\\Users\\dburch.CSWG\\.juliapro\\JuliaPro_v1.0.3.2\\environments\\v1.0\\Project.toml")

println(Base.load_path())
println(remotecall_fetch(Base.load_path, 2))

@everywhere using JuliaDB

Error:

      From worker 2:    ERROR: LoadError: ArgumentError: Package Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c] is required but does not seem to be installed:
      From worker 2:     - Run `Pkg.instantiate()` to install all recorded dependencies.
      From worker 2:
      From worker 2:    Stacktrace:
      From worker 2:     [1] _require(::Base.PkgId) at .\loading.jl:929
      From worker 2:     [2] require(::Base.PkgId) at .\loading.jl:858
      From worker 2:     [3] require(::Module, ::Symbol) at .\loading.jl:853
      From worker 2:     [4] include at .\boot.jl:317 [inlined]
      From worker 2:     [5] include_relative(::Module, ::String) at .\loading.jl:1044
      From worker 2:     [6] include(::Module, ::String) at .\sysimg.jl:29
      From worker 2:     [7] top-level scope at none:2
      From worker 2:     [8] eval at .\boot.jl:319 [inlined]
      From worker 2:     [9] eval(::Expr) at .\client.jl:393
      From worker 2:     [10] top-level scope at .\none:3
      From worker 2:    in expression starting at C:\Users\[REDACTED]\.julia\packages\IndexedTables\DgX0A\src\IndexedTables.jl:12
      From worker 2:    ERROR: LoadError: Failed to precompile IndexedTables [6deec6e2-d858-57c5-ab9b-e6ca5bd20e43] to C:\Users\[REDACTED]\.julia\compiled\v1.0\IndexedTables\YIs33.i.
      From worker 2:    Stacktrace:
      From worker 2:     [1] error(::String) at .\error.jl:33
      From worker 2:     [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1203
      From worker 2:     [3] _require(::Base.PkgId) at .\loading.jl:960
      From worker 2:     [4] require(::Base.PkgId) at .\loading.jl:858
      From worker 2:     [5] require(::Module, ::Symbol) at .\loading.jl:853
      From worker 2:     [6] include at .\boot.jl:317 [inlined]
      From worker 2:     [7] include_relative(::Module, ::String) at .\loading.jl:1044
      From worker 2:     [8] include(::Module, ::String) at .\sysimg.jl:29
      From worker 2:     [9] top-level scope at none:2
      From worker 2:     [10] eval at .\boot.jl:319 [inlined]
      From worker 2:     [11] eval(::Expr) at .\client.jl:393
      From worker 2:     [12] top-level scope at .\none:3
      From worker 2:    in expression starting at C:\Users\[REDACTED]\.julia\packages\JuliaDB\jDAlJ\src\JuliaDB.jl:7
ERROR: LoadError: On worker 2:
Failed to precompile JuliaDB [a93385a2-3734-596a-9a66-3cfbb77141e6] to C:\Users\[REDACTED]\.julia\compiled\v1.0\JuliaDB\4FA8g.ji.
error at .\error.jl:33
compilecache at .\loading.jl:1203
_require at .\loading.jl:960
require at .\loading.jl:858
require at .\loading.jl:853
eval at .\boot.jl:319
#116 at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Uers\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.j:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\ApData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##163#165")){Module,Expr})() at .\task.jl:259
Stacktrace:
 [1] sync_end(::Array{Any,1}) at .\task.jl:226
 [2] macro expansion at .\task.jl:245 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:206
 [4] top-level scope at C:\Users\julia\AppData\Local\Julia-1.0.3\share\julia\stdlib\v1.0\Distributed\src\macros.jl:190
in expression starting at untitled-d3683fe8e6936bb5b606f0e41275b73b:15

JuliaPro_v1.0.4.1 is now the long-term support release. I am going to download and install and see if I still experience the same error.

Is Worker 2 on a separate server from the one you are using? The C: drive on Worker 2 will not be a shared drive with the C: drive on your first server.
I am also having a bit of a lightbulb moment here. From the documentation:
Add processes on remote machines via SSH. Requires julia to be installed in the same location on each node, or to be available via a shared file system.

Perhaps on Windows when the login is via ssh the environment is set up differently. That is what @greg_plowman is asking you to test.

No, I am just trying to get it to work locally at this point. It wasn’t working for me with Julia 1.0.3. I am installing 1.0.4 to see if the issue goes away. I would like to only use the latest long-term support release, so I would prefer not to use 1.1.1.

The following code does not work with JuliaPro 1.0.3 or 1.0.4, but it works on Julia 1.1.1.

using Distributed, JuliaDB
addprocs(1)
@everywhere using JuliaDB

I now suspect that the issue has to do with the JuliaPro folder path. I am going to install Julia 1.0.4 and see if the result is different from the JuliaPro issue.