[ANN] DistributedAdhoc.jl: for machines that don't share a common filesystem

Package Announcements
1

DistributedAdhoc.jl helps using Julia Distributed in an ad-hoc way, on computers not dedicated to distributed processing. The only setup required is to install Julia everywhere.

Each machine can have different paths, different usernames and home directories, no shared filesystem.

DistributedAdhoc sends files, directories, and whole environments from one process to other workers:

julia> using DistributedAdhoc  # reexports Distributed, adds no new exports

julia> addprocs(...)

# send a local file and a directory to worker #3
julia> remote_file = DistributedAdhoc.send_file("local_path/to/my_file.txt", 3)
"/tmp/jl_S1vUpx/my_file.txt"  # path of the file on worker #3

julia> remote_dir = DistributedAdhoc.send_dir("local_path/to/my_dir", 3; include=["file.txt", "data/*.csv"])
"/tmp/jl_fKgj4Y"  # path of the dir on worker #3

# send the current Julia environment to all workers, instantiate and activate it
# this example includes source code in `src`, developed packages (in `./dev`), scripts (in `./scripts`), and CSV files (in `./data`)
julia> DistributedAdhoc.send_env_activate_everywhere(include=[
    "*.toml", "src/*.jl", "scripts/*.jl",
    "dev/*/*.toml", "dev/*/src/*.jl", "dev/*/src/*/*.jl",
    "data/*.csv",
])

More functions, options, and details: see src/DistributedAdhoc.jl · master · Alexander Plavin / DistributedAdhoc.jl · GitLab or docstrings.

After half a year of this package being publically available, I’ve just registered DistributedAdhoc.jl in General.
Git repo available at Alexander Plavin / DistributedAdhoc.jl · GitLab.

5 Likes
2

Out of curiosity, why gitlab instead of github?

3

See this post of mine, from about a year ago:

Btw, looks like now 45 packages in General are hosted at gitlab. A 3-fold increase over a year (:

3 Likes