I posted this at Uni-directional communication in ElasticManager? · Issue #113 · JuliaParallel/ClusterManagers.jl · GitHub but that repo is pretty quiet plus I think there’s a wider audience here with Julia Distributed knowledge that could potentially answer my question, hence the x-post. Anyway, from the link:
I’m using ClusterManagers.jl on a cluster where the compute nodes (which run workers) can connect to the login nodes (which run the master process), but the other direction is blocked by firewalls. The current code which sets up an ElasticManager first has the workers connecting to master (which works), but then, via code in
Base.Distributed, the master establishes a second connection to the workers (which in my case is blocked). I’m wondering:
- What’s the reason for the extra connection here? (which very naively to me seems unnecessary since we already have one), and
- Is there any reason we can’t just use the original worker->master connection for all communication? I’ve been hacking at the code a bit and it kind of seems to work, but was wondering if there’s a showstopper I should know about before spending more energy getting it fully working.
Thanks.