This is half a question about Julia, half a question about Docker. Framing it in this way is what I think will be best for my comprehension though.
There are maintained Julia images on Dockerhub (See here). I get that you can use one of these images and just start up the container to use as a sort of self-contained Julia install.
What I’m trying to better understand is that, as far as I know, Docker requires root permissions to start images. This makes it hard in an environment where someone may not have root/sudo permissions, and if you do have root permissions, why not just install Julia directly? You also can’t (as far as I can tell) jump from one docker container in to a Julia container either (I.E - a SSH container that you can use to access Julia)
I feel like I must be misunderstanding something so - what are the use cases for a julia docker container? Can docker do more than I’ve managed to learn about searching the internet that makes it an appealing choice?
@fergu Docker is nbot the only game in town. Look at Singularity - it was invented for the shared use environments you refer to and does not need root access. Singuklarity can also happily read Docker containers and convert them.
I can help you install Singularity if you have problems.Regarding workloads, Singularity works with Slurm batch, Kubernetes and Hashicorp Nomad
I should say I have never done any of these three, but you can easily find writeups.
Is this something that I as a user of an HPC cluster can install on my own for my own usage? Or does it still require an admin to do an initial installation?
More generally, is there a solution which allows me to run containers on a Slurm cluster that I have access to without any admin intervention? (not sure I need this, but just curious).
Personally I have found docker really nice for deploying an application in a server environment. You can build your application stuff it in a docker container and pass that container off to other people to run. You don’t need to worry about any quirks of their OS install or permissions or anything really. You don’t even have to worry if they are running on Windows or Linux. Your application is self contained in a “blob” and you know that blob works.
So I wouldn’t use the Julia docker image for personal use. I would use it as the base docker image for my docker image that contains my Julia application for distribution. The docker model is to start with another docker image and add your files to it and release that modified image. And this would have to be a “server” application i.e. something people would access over the network. Yes docker can be used just to run an application locally but like you said it needs root, and that’s really frowned upon.
I believe docker’s market is really deploying server applications. (I have used docker to install pgadmin when my distro was having issues, but that is a web application anyway so it works well inside a docker image.) If you want something that deploys desktop applications that’s probably Canonical’s Snappy (which I’ve never actually used). It looks like a Julia was package in snappy as well, but it’s a little old:
Singularity is often available in HPC clusters, you should check the one you have access to. You can also install it in your userspace, but I found that not all features of version 3 may be available if installed in this way
Just to add that you can well install Julia without root permissions, just download/untar the binaries in your ~/bin (or wherever is in yr path) and you are ready.
I normally untar the Julia binaries in my ~/lib and symlink to ~/bin, never had problems…
So I wouldn’t use the Julia docker image for personal use. I would use it as the base docker image for my docker image that contains my Julia application for distribution. The docker model is to start with another docker image and add your files to it and release that modified image. And this would have to be a “server” application i.e. something people would access over the network. Yes docker can be used just to run an application locally but like you said it needs root, and that’s really frowned upon.
@pixel27 - what might one use Julia for in a web app sense? Like - I know you can (and I have done) install sshd on top of a Julia docker image and from there you can log in to it and access the REPL, etc.
My understanding was Docker is meant as a “one app one container” application - so a Julia docker image should contain Julia (+ whatever extra packages on top of default that I want), and a SSH docker image should contain SSH. The two should then launch separately and communicate via the network to do whatever it is you want - which, of course, is not something Julia can do easily. Am I misunderstanding and what is meant by “deploy my app” is exactly that I can wrap up SSH and Julia and say “here’s a ssh server with Julia so you can access it remotely but if the image restarts you’ll need to reinstall your stuff” and put that on github or wherever?
Continuing that, I could then set up a FTP server in a different container and configure my ssh container to be able to access/mount that FTP to get files while also permitting access to the FTP from, IE, my laptop instead of the docker image? (I’m aware of volumes - the remote access case is exactly why I mention FTP)
Basically I’m using Julia for the server side code of a web application. Although sometimes calling it a web application is a bit much. I like the web application model for desktop applications. I’m not sure if you’ve look at the various GUI libraries that are available for desktop application but my feeling is that all of them suck. Or at least I haven’t found one I’ve liked.
To me it feels like there has been a lot of work making HTML interfaces better, but desktop (native) UI libraries have stagnated. What I do like is the data driven model provided Angular and ReactJS. You create the HTML with ties back to the data objects. When the data objects change the HTML automatically changes. Using this model also segregates your code quite nicely. The code dealing with the UI is all HTML/JavaScript the “real” code is in Julia (or another language) and I have a nice well defined HTML API to communicate between them. When I’ve built GUI using a native library to create a desktop application, the UI codes tends to start infecting the the code that doing the “real” work.
So with Julia I can use the HTTP.jl package to handle the HTTP protocol and dispatch the requests for static and dynamic content to the appropriate handlers. When I’m done I have a “desktop” application that can be converted to a web application if I really want to.
As a side note I’m not really using Julia is the “standard” way. I’m not processing gigabytes of data doing weird matrix multiplications/reductions/whatever. I’m not creating neural nets, doing simulations and whatever. You guys start geeking on that and my eyes glaze over. I’m using Julia as a general programming language.
I wanted a modern language that was fast, i.e. compiled to native code. Speed is one of the goals of Julia so I decided to give it a whirl. C/C++ has headers and code files which have to keep kept in sync and are really just a legacy from running on 70s era computers. Rust…well the memory ownership model will drive you to drink. Java is pretty fast, the JIT will compile to native code, but the VM is kind of lowest common denominator which slows things down, and that’s what I deal with in my 9 to 5, so I want something that doesn’t remind me of work.
Yes, there is a slight misunderstanding there. Docker containers don’t have to contain only one “app” - instead think of them as serving one self-contained purpose. This means you can put any programs you want into a container and have them work together. So if you wanted a Julia instances, you can add Julia and the SSH server in the same container. The main purpose of the container is to provide isolation between different tools/applications/stacks so that you don’t run into library dependency problems, permissions issues, or data leaks. Another thing that the containers give you is the ability to run different OS versions easily - so I could compile using an Ubuntu 18.04 container running on my Fedora 32 install.
You also tend to find Docker containers a lot in the CI world - especially if you use GitLab CI. On their CI infrastructure you can supply a docker container to run your pipeline in - which allows you to build out a container that has any dependencies for your CI already included (for instance you could install any development libraries you need inside the base container so that you don’t have to always install them when your pipeline starts).