# Spindle : Scalable Parallel Input Network for Dynamic Load Environments

**URL:** <https://discourse.julialang.org/t/spindle-scalable-parallel-input-network-for-dynamic-load-environments/47013>\
**Category:** Julia at Scale\
**Created:** [September 21, 2020, 2:37pm UTC](https://discourse.julialang.org/t/spindle-scalable-parallel-input-network-for-dynamic-load-environments/47013 "2020-09-21T14:37:08Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 21, 2020, 2:37pm UTC](https://discourse.julialang.org/t/spindle-scalable-parallel-input-network-for-dynamic-load-environments/47013/1 "2020-09-21T14:37:08Z")

</div>

From Todd Gamblin on Twitter: I keep reading papers saying containers are the only path to scalable python/shared lib loading. Large HPC jobs can DDOS filesystems by loading many libs/.py files. Spindle solves this _generally_: It does _not_ need containers or special staging.

[https://github.com/hpc/Spindle](https://github.com/hpc/Spindle)

I have no feeling if this would be useful for Julia on large HPC clusters.  
I can imagine if a sysimage is being used on thousands of nodes it will be useful.  
Aha: Spindle takes over the library loading mechanism. Hmmmm… might not be that useful for Julia then.