I would like to announce a new version of Elly.jl (Release Elly v0.4.0 · JuliaParallel/Elly.jl · GitHub) - a Julia HDFS and Yarn client. This version has some notable updates to Yarn cluster support.
It now allows Julia to run as a Yarn managed application, fully inside the Yarn cluster - what is called as “managed application master” in Yarn terminology.
Running as a managed app master would allow Julia Applications to be run independently and fully inside Yarn, unlike the case of unmanaged app master where the master process hangs outside the cluster. The submitting process can terminate safely, leaving the submited Julia process running inside the cluster.
The Julia process running inside the Yarn cluster can then start other worker processes as it desires - all running inside Yarn. Elly.jl also includes a ClusterManager implementation for Yarn that allows the familiar addprocs, rmprocs methods for starting and stopping worker processes with Yarn. The Yarn ClusterManager can now work in managed mode as well.
Status of the submitted application can be queried with provided APIs, or viewed on the Yarn status board.