# Include files on (remote) worker processes

**URL:** <https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807>\
**Category:** General Usage\
**Tags:** parallel, distributed\
**Created:** [June 15, 2022, 12:24pm UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807 "2022-06-15T12:24:42Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![daharn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daharn/32/27088_2.png) [@daharn](https://discourse.julialang.org/u/daharn)\
**Post date:** [June 15, 2022, 12:24pm UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/1 "2022-06-15T12:24:42Z")

</div>

I have pretty much the same setup as described [here](https://discourse.julialang.org/t/sending-modules-from-a-local-julia-file-to-remote-machines/8088), that is, I have a local run file that should start parallel computations on remote machines. These computations require functions defined in several files. Simply `include` these files would require them (and their folder structure) to be physically present on the remote machines, which I would like to avoid.

The suggested solution in the above post (using [ParallelDataTransfer.jl](https://github.com/ChrisRackauckas/ParallelDataTransfer.jl) does not work for me when I test it with local workers:

```julia
using Distributed
using ParallelDataTransfer: include_remote

file = touch("tmp.jl")
open(file, "w") do f
    write(f,"fun() = 42")
end

addprocs(1,dir=pwd())

#%% does not work, "fun not defined"

include_remote(file, workers())

@fetchfrom workers()[1] fun()

#%% works
@spawnat workers()[1] include(file)

@fetchfrom workers()[1] fun()

```

Is ParallelDataTransfer simply not working any more? I see that there hasn’t been much happening over the last couple of years. I am running Julia 1.7.2.

Or is there another way

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [June 15, 2022, 2:39pm UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/2 "2022-06-15T14:39:03Z")

</div>

May I ask what type of remote machines these are?  
On an HPC setup you would normally have a filesystem which is mounted on all workers.  
If you are using cloud machines you should be able to set up a shared filesystem quite easily.

I am not trying to second guess you.

---

<div class="post-metadata">

**Author:** ![daharn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daharn/32/27088_2.png) [@daharn](https://discourse.julialang.org/u/daharn)\
**Post date:** [June 15, 2022, 3:50pm UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/3 "2022-06-15T15:50:09Z")

</div>

These remote machines are local cluster nodes running Linux and administrated by our research group. You are right in that they all share a common filesystem.

However, I would typically initiate the overall process from my personal computer and just ship out all the calculating to the workers (Master - slave like). That means that I currently have to ensure that my local scripts are always synchronized with their counterparts on the shared cluster filesystem, so all in all two locations. That’s certainly manageable, but if this redundancy could be avoided, even better.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [June 16, 2022, 1:32am UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/4 "2022-06-16T01:32:53Z")

</div>

OK, so you do not mount the shared filesystem on your laptop - this makes sense as it travels around with you.  
There are many ways of syncing filesystems - a search comes up with

> **[Linux Fu: Keep In Sync](https://hackaday.com/2020/07/23/linux-fu-keep-in-sync/)**
>
> Once upon a time, computers were very expensive and you were lucky to have shared access to one computer. While that might seem to be a problem, it did have one big advantage: all of your files wer…

Another thought - how about keeping your code in a Git repository?  
At the start of your job the Julia code runs a git checkout.  
Somebody much smarter than me will come along and show that the Julia Package Manager can do this job much better.

If your remote machines are using a batch processing system this will create a unique directory on each server per job, called something like $TMPDIR  
you can use a git checkout to get your code into that directory

> **[GitHub - JuliaVersionControl/Git.jl: Use command-line Git in your Julia packages](https://github.com/JuliaVersionControl/Git.jl)**
>
> Use command-line Git in your Julia packages. Contribute to JuliaVersionControl/Git.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 16, 2022, 7:11am UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/5 "2022-06-16T07:11:31Z")

</div>

I have a similar usecase and run distributed code on machines that are not a part of any cluster at all - just a few random workstations in our group. Julia doesn’t really have built-in features to make it easy, and I’ve written a small helper package: [[ANN] DistributedAdhoc.jl: for machines that don't share a common filesystem](https://discourse.julialang.org/t/ann-distributedadhoc-jl-for-machines-that-dont-share-a-common-filesystem/75853). It can transfer individual files and directories without extra setup, and share whole julia environments from the local machine.

---

<div class="post-metadata">

**Author:** ![daharn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daharn/32/27088_2.png) [@daharn](https://discourse.julialang.org/u/daharn)\
**Post date:** [June 16, 2022, 10:22am UTC](https://discourse.julialang.org/t/include-files-on-remote-worker-processes/82807/6 "2022-06-16T10:22:35Z")

</div>

> [@johnh](#):
>
> Another thought - how about keeping your code in a Git repository?

Yes, that’s my current solution. Since I use `DrWatson` to manage the project, the syncing is no problem. You just need to remember doing it, and at some point I spent quite some time trying to figure out why stuff worked locally but not on the remote machines until I remembered that I forgot to sync the two repos… Arguably, this won’t happen to me again but who knows…

> [@aplavin](#):
>
> I have a similar usecase and run distributed code on machines that are not a part of any cluster at all - just a few random workstations in our group.

That’s a good point. If you just want to combine some random workstations, your approach seems very valuable. Thanks!