In my algorithms course, we plan to transition to using Julia this fall, and we are currently trying to make it work with INGInious, a system that automatically runs and tests code or code fragments submitted by students, using Docker containers. The main issue we keep facing is giving Julia access to the files it wants – primarily the ~/.julia directory. We’ve managed to solve this by setting HOME to a directory where the process has read and write access, which solves some problems. For example, we are now allowed to use the ordinary using mechanism (not possible before we adjusted HOME), as the relevant calls to stat seem to be permitted. And I guess this would also permit compilation caching of any modules the students wrote themselves and submitted. However, we’d also want to install external modules for the students to use – so I guess a pre-populated .julia directory would have to be copied into the relevant directory/container for each run?
We could place it in a shared directory with no write access, but that would prevent the students from precompiling their own modules. Possibly not a real issue (we’re mostly talking small code fragments here anyway), but it doesn’t seem ideal. It would be great if we could have multiple locations – one with precompiled globally installed/shared modules and one with the compiler cache for the local code of the students themselves. I haven’t seen any such functionality in Pkg3, but maybe I missed something? (I guess the reason this works for, e.g., Python is that it caches the compiled files alongside the source.)
Does anyone else have any experience with running Julia in a related context? Are there any other issue to consider (and any relevant solutions)?
2 Likes
Hm. I now see that there is something we might be able to use, i.e., JULIA_DEPOT_PATH. I guess I should have read the docs more carefully! (Not 100% sure this solves everything, but still…)
I think you can find some ideas on how to set it with Docker here.
I’m using this image myself to learn Julia and it’s proving really helpful.
1 Like
https://gist.github.com/quinnj/169a2d712ab8f780e56772ed88867d3a
Here’s the “base” Dockerfile I use w/ a few different custom projects. Basically it sets up a “default” user and uses shared Project.toml, Manifest.toml files to install global/shared external packages.
Custom applications can then just use this docker image as the base image, copy their own Project.toml/Manifest.toml in, instantiate their project and be on their way. julia is on the path (installed in /usr/local/bin), so that’s convenient.
Happy to answer any other questions about this setup.
8 Likes
It would be really nice if this were available somewhere that makes it easy to find. This even seems like something that should appear in the official Julia documentation somewhere as containers becoming increasingly more important these days.
4 Likes
I’ve experimented with this as well, see https://github.com/JuliaGPU/docker, where I specify a volume for installed packages and one for the current working directory (as well as a more complicated template scheme to provide preinstalled packages and build them upon first run). Might be a bit over-engineered, and I don’t plan to maintain it now that GPU packages are easier to install.
1 Like
I’ve used Julia in Docker for reproducibility and for running on MyBinder a few times. I find it works fairly well. You can pre-install and precompile packages in the Docker container by including something like this in your Dockerfile (assuming Julia 0.6):
COPY REQUIRE $HOME/REQUIRE
RUN julia -e "Pkg.init(); mv(\"$HOME/REQUIRE\",Pkg.dir(\"REQUIRE\"),remove_destination=true); Pkg.resolve(); for p in Pkg.available(); try @eval using \$(Symbol(p)); println(p); end; end"
where you have listed all your dependent packages in the REQUIRE file.
2 Likes
Thanks for lots of great answers! Something along these lines would indeed (as @ExpandingMan suggests) be a great tutorial or documentation section. But if nothing else, I guess people can at least find this thread, now 
Hm. We had things working with a 0.7 nightly, but now (with 1.0, and with 0.7 final, it seems), we again get access errors – presumably because precompilation is trying to access something it shouldn’t. Is there any easy way to avoid that? (It seems Pkg mainly uses DEPOT_PATH[1], but I also see activity in temp-directories, and – when installing things, which may not be that relevant, it also reads ~/.gitconfig, for example, presumably via libgit2.)
