Julia packages with Singularity


#1

How do we create Singularity containers which contains a reproducible set of packages which are installed at countainer build time?
Singularity https://www.sylabs.io/

This discussion should follow on from a thread on the Singularity email list, which is asking some good questions but I feel was getting away from strict discussions about that technology.


#2

Any chance you can link to the mailing list archives, and possibly provide snippets of the relevant parts of the discussion for those of us who aren’t subscribed?


#3

@jpsamaroo But of course. I shoudl have doen that. See here, it is quite a long thread.

https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/Vv5ccqWjO6E


#4

Thanks! So, let me see if I understand what’s going on. My Singularity knowledge is rusty and very basic, so please correct me as necessary:

  • You have a container that has julia installed in it (maybe from docker, maybe custom built in Singularity)
  • You then start it up as a non-root user, and when that user attempts to install packages, julia attempts to precompile into files that are under /root, so you get permission denied (aside: does this also occur when sync’ing the registry?)
  • So now instead, you need to workaround by either running everything as root (100% not a good idea, and not what one does in a Singularity container), or by bind mounting a directory on the outside that was built by the appropriate user and/or is mounted with the invoking user set as owner of all files

Note that, as you’ve said:

create a container with all the build tools which are needed

this is definitely correct in all circumstances, if you need to add packages which build binaries, like HDF5. Anything using BinaryProvider alone will probably not need these, but it would make the most sense to have them installed anyway.

Does this all sound accurate to you?


#5

You have it right!

  • You then start it up as a non-root user, and when that user attempts to install packages, julia attempts to precompile into files that are under /root, so you get permission denied (aside: does this also occur when sync’ing the registry?)
    This is not correct - the original poster there said that he installs the packages as root, and has permissions problems / crashes.

#6

I guess the discussion I would like to broaden is:
As you say, the best advice method would be to install the required build tools in the container.

My thoughts of using a bind mount of your home directory negate one advantage of containers - that the installed software is reproducible, as over time you will update your personal packages.
However I ask - if the container is immutable, Julia tries to update packages anyway when it starts ???


#7

There was a question asked yesterday on the Singularity list, regarding building Tensorflow.
the replay was that multi stage builds will probably come out with the 3.2.0 release.
Maybe that is what we need. https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/U2jSMcl32Mw


#8

I’m also interested in this topic. I help to maintain a bioinformatics tool called ‘MentaLiST’ that is written in Julia:

I’ve also struggled to build Docker and/or Singularity images for similar reasons.

This is where I’m at so far but I’m also getting some permissions errors when attempting to run the image.