Using @everywhere include() from Distributed.jl?

dlakelan · September 13, 2022, 9:01pm

I’ve got a Weave document (.jmd) and I’m using

@everywhere include("./utilities.jl")

There’s an error that says the remote machine is trying to load a file not in the current directory, but the same directory as on my local machine, which doesn’t exist on the remote machine.

documentation for @everywhere macro says:

  Similar to calling remotecall_eval(Main, procs, expr), but with two extra
  features:

  - `using` and `import` statements run on the calling process first, to ensure
    packages are precompiled.
  - The current source file path used by `include` is propagated to other processes.

I think the second feature is a misfeature. If I wanted it to read from the same path as on my local computer, I’d send it an absolute path… instead it takes the relative path, converts it to an absolute path, and sends that to the remote machine, which fails since my home directory is not mounted on the remote machine…

What’s the workaround? and should this be a bug?

At the moment, I’m trying to work around with Distributed.remotecall_eval(…) we’ll see how it goes.

suavesito · September 13, 2022, 9:14pm

From the same documentation, maybe you can try the next.

using Distributed
Distributed.remotecall_eval(Main, procs(), :(include("./utilities.jl")))

My guess is that in HPC environments, i.e. super computers, you would run @everywhere in the cluster as a script and probably all the machines would be configured similarly, including in terms of directories and filesystem structure.

dlakelan · September 13, 2022, 9:17pm

Yes, the remotecall_eval trick sort of worked. It did get the file included, but then things borked because it’s a Weave.jl document… I’ve got a separate post on that topic. Note that remotecall_eval is not exported by Distributed and seems kind of less friendly.

I think the assumption that all machines are configured exactly the same is flawed. At the least it should be possible to override this behavior for include().

suavesito · September 13, 2022, 9:25pm

You can fill an issue in the JuliaLang repository, I think it would be a nice option.

dlakelan · September 13, 2022, 9:28pm

Ok, I did that

github.com/JuliaLang/julia

Allow @everywhere include(...) to override default path behavior

opened 09:28PM - 13 Sep 22 UTC

dlakelan

parallel

According to tests and the documentation string: ``` Similar to calling re…motecall_eval(Main, procs, expr), but with two extra features: - `using` and `import` statements run on the calling process first, to ensure packages are precompiled. - The current source file path used by `include` is propagated to other processes. ``` This last "feature" overrides the path even if that isn't desired behavior. I discovered this when doing ``` @everywhere include("./utilities.jl") ``` and I expected it to read from the "current directory" that the remote julia process started in, but in fact it tried to read from my home directory (same directory as on my machine where the whole process was initiated) however that path doesn't exist remotely and isn't the desired behavior. It would be nice to have an option to not have `@everywhere include(...)` override the path.