DistributedAdhoc.jl helps using Julia Distributed in an ad-hoc way, on computers not dedicated to distributed processing. The only setup required is to install Julia everywhere.
Each machine can have different paths, different usernames and home directories, no shared filesystem.
DistributedAdhoc sends files, directories, and whole environments from one process to other workers:
julia> using DistributedAdhoc # reexports Distributed, adds no new exports julia> addprocs(...) # send a local file and a directory to worker #3 julia> remote_file = DistributedAdhoc.send_file("local_path/to/my_file.txt", 3) "/tmp/jl_S1vUpx/my_file.txt" # path of the file on worker #3 julia> remote_dir = DistributedAdhoc.send_dir("local_path/to/my_dir", 3; include=["file.txt", "data/*.csv"]) "/tmp/jl_fKgj4Y" # path of the dir on worker #3 # send the current Julia environment to all workers, instantiate and activate it # this example includes source code in `src`, developed packages (in `./dev`), scripts (in `./scripts`), and CSV files (in `./data`) julia> DistributedAdhoc.send_env_activate_everywhere(include=[ "*.toml", "src/*.jl", "scripts/*.jl", "dev/*/*.toml", "dev/*/src/*.jl", "dev/*/src/*/*.jl", "data/*.csv", ])
More functions, options, and details: see src/DistributedAdhoc.jl · master · Alexander Plavin / DistributedAdhoc.jl · GitLab or docstrings.
After half a year of this package being publically available, I’ve just registered
Git repo available at Alexander Plavin / DistributedAdhoc.jl · GitLab.