I also ran into this problem and solved it by using a local cache for each job.
# make a local temp directory for the cache
TMPDIR=`mktemp -d`
mkdir "$TMPDIR/compiled"
# copy the cache to the temp directory
rsync -au "$JULIA_DEPOT_PATH/compiled/v1.0" "$TMPDIR/compiled/"
# set the temp directory as the first depot so any (re)compilation will happen there and not interfere with other jobs
export JULIA_DEPOT_PATH="$TMPDIR:$JULIA_DEPOT_PATH"
If you don’t have $JULIA_DEPOT_PATH set then you need to substitute the default which is $HOME/.julia.
However, I’m not sure if the extra complexity is worth it vs just running with --compilecache=no.