Run a julia application at large scale (on thousands of nodes)

Hello @samo I looked for this thread for you

The answer seems to be to create a local cache. You can do this even if you are running the compute nodes in a RAMdisk - you should still have a writeable /tmp

make a local temp directory for the cache

TMPDIR=mktemp -d
mkdir “$TMPDIR/compiled”

copy the cache to the temp directory

rsync -au “$JULIA_DEPOT_PATH/compiled/v1.0” “$TMPDIR/compiled/”

set the temp directory as the first depot so any (re)compilation will happen there and not interfere with other jobs

export JULIA_DEPOT_PATH=“$TMPDIR:$JULIA_DEPOT_PATH”

1 Like