Using Distributed.jl from within Weave.jl?

General Usage
,
1

I’m using Distributed from within my Weave.jl document because I have a machine which has a big store of data, and I’m asking it to process that data and return me some summary/filtered version of it.

When I select bits of the code and run it in vscode it works just fine. When I run it as weave(“myfile.jmd”) I get an error like:

ERROR: On worker 2:
UndefVarError: ##WeaveSandBox#312 not defined
Stacktrace:
  [1] deserialize_module
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:996
  [2] handle_deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:895
  [3] deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
  [4] deserialize_datatype
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1363
  [5] handle_deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:866
...

Which suggests that something is happening on the remote machine that doesn’t make sense because of Weave.

Is there an approved way to use Distributed.jl from within Weave?

2

Do you have a MWE? So that we can test. It looks like the only place where WeaveSandBox is defined is here. The comment above reads

    # New sandbox for each document with args exposed
    isnothing(mod) && (mod = sandbox = Core.eval(Main, :(module $(gensym(:WeaveSandBox)) end))::Module)

Which makes me think that the error is because the environment that your document is tried to run (##WeaveSandBox#312) does not exist on the other machine.

3

I can try to put together a MWE tomorrow but I’m guessing just the slightest use of @spawnat to do anything much is going to cause this. Like tell it to spawnat another machine and generate a random vector or read a csv file or something… Am not at my machine so will have to get you something tomorrow.

4

Ok, @suavesito here’s my MWE, put this in test.jmd edit the machines and paths to work with your machine/setup, and try to weave it

# Testing

```julia

using Distributed 
rmprocs.(workers())
censproc = addprocs(["mydevice.lan"]; dir = "/var/tmp/test/", exename="/home/dlakelan/julia/bin/julia")

@everywhere using Pkg
@everywhere Pkg.activate(".")

@everywhere using StatsPlots, DataFrames, DataFramesMeta, FredData, Dates, GLM, StatsBase,
    FredData, CSV, Serialization, Downloads, Interpolations, Optim, Turing, LinearAlgebra, CategoricalArrays, ReverseDiff, Memoization

Distributed.remotecall_eval(Main, workers(), :(include("./utilities.jl")))
include("./utilities.jl")


f = @spawnat censproc[1] rand(200) 
fetch(f)

```

What did we get?

This results in:

ERROR: On worker 4:
UndefVarError: ##WeaveSandBox#488 not defined
Stacktrace:
  [1] deserialize_module
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:996
  [2] handle_deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:895
  [3] deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
  [4] deserialize_datatype
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1363
  [5] handle_deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:866
  [6] deserialize
    @ /var/local/dlakelan/julia/julia-1.8.0/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
  [7] handle_deserialize
...