What is the correct way to install packages (Julia 0.6.4) that can be accessed by any user?

New to Julia
1

I couldn’t see a self evident way of achieving this. Bare in mind I am working within a singularity container.

My first approach was to install as root but that didn’t work.

I then attempted to bundle the packages in a tar file and unpack upon running the container in the $PWD (setting $JULIA_PKGDIR to match). However references are still being made to the old directory.

Something to note, when working with singularity is that the the libraries installed as root are available on a write-only filesystem. I have observed that Julia needs to write in order to reconcile cache, etc (is there a way to avoid this writing altogether?).

I am open to alternate solutions. In the end, I want non-root users to be able to use the pre-defined containers. Is there a way to make a copy of all the packages in a new directory (that is read/write) and owned by the user?

Please let me know if I can add more detail. I am working on a workaround atm and will share it here.

1 Like
2

Here is my workaround. It is awfully hacky but until I learn what I am missing (both singularity and julia), I see no other way at present. It works by making use of /tmp for the read/write filesystem which Julia seems to need for its package directory.

Please keep in mind, the solution I am looking for on this forum is one which removes the need for there to be a writable filesystem for $JULIA_PKGDIR and also one where library files owned by root can still be read, write and executed by any other user (one of many user/group ids, etc)

STEP1: Build an image, which when executed produces an archive of packages/libraries within a tar file in the current working directory:

Singularity def for ‘dump tar’ image (save as nextgensequtils_julia-0-6-4_dumptar.def):

Bootstrap: docker
From: julia:0.6.4

%post
    apt-get -y update
    apt-get -y install bzip2

%environment
    export PATH=/usr/local/julia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH
    export JULIA_PATH=/usr/local/julia
    export JULIA_GPG=3673DF529D9049477F76B37566E3C7DC03D6E495
    export JULIA_VERSION=0.6.4
    export JULIA_PKGDIR=/tmp/.julia #This needs to be somewhere read/writable and location must be consistent between both images described here. 

%runscript
    rm -rf ${JULIA_PKGDIR}
    julia -e 'Pkg.init(); Pkg.update(); Pkg.clone("https://github.com/MurrellGroup/NextGenSeqUtils.jl"); using NextGenSeqUtils'
    export NEXTGEN_TARDIR=${PWD}/nextgensequtils_julia-0-6-4.tar
    cd ${JULIA_PKGDIR}
    rm -f ${NEXTGEN_TARDIR}
    tar cf ${NEXTGEN_TARDIR} . # Note the dot here

Singularity build command:
sudo singularity build nextgensequtils_julia-0-6-4_dumptar.sif nextgensequtils_julia-0-6-4_dumptar.def

STEP2: Run the image from step 1. It should produce a tar file in your $PWD when done

STEP3: Build an image based off this one, just a different runscript.
I failed to see how to do that, so I reluctently copied and pasted the top.

Singularity def for main image:

Bootstrap: docker
From: julia:0.6.4

%post
    apt-get -y update
    apt-get -y install bzip2

%environment
    export PATH=/usr/local/julia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH
    export JULIA_PATH=/usr/local/julia
    export JULIA_GPG=3673DF529D9049477F76B37566E3C7DC03D6E495
    export JULIA_VERSION=0.6.4
    export JULIA_PKGDIR=/tmp/.julia

%runscript
    rm -rf ${JULIA_PKGDIR}
    mkdir -p ${JULIA_PKGDIR}
    tar -xf ${PWD}/nextgensequtils_julia-0-6-4.tar -C ${JULIA_PKGDIR}/
    julia

Singularity build command:
sudo singularity build nextgensequtils_julia-0-6-4.sif nextgensequtils_julia-0-6-4.def

STEP4: Run the image from step 3. It should drop you into a Julia shell

STEP5: Attempt to make the import
using NextGenSeqUtils

If that works, it is looking good. Here is some more code to actually use one of the functions:

q_seqs = [
    "GATAGTTTTTTTTTTTTGTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTGTTTTTTTTTTTTTTTGACAG",
    "GATAGTTTTTTTTTTTTGGTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTGGTTTTTTTTTTTTTTTTGACAG",
    "GATAGTTTTTTTTTGGTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTGGTTTTTTTTTTTTTTTTGACAG",
    "GATAGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTGGGGGTTTTTTTTTTTTTTTTTGACAG",
    "TCTACTTTTTTTTTGGTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "TCTACTTTTTTTTTTGTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "TCTACTTTTTTTTTTTTTGGTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTTGTTTTTTTTTTTTTTGACAG",
    "TCTACTTTTTTTTTTTGGGGGTTTTTTTTTTTTTTAAAAATTTTTTTTTTTGGGGGTTTTTTTTTTTTTTGACAG",
    "CATCATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTGGTTTTTTTTTTTTTTTTTTGACAG",
    "CATCATTTTTTTTTTTTTTTTTTGGTTTTTTTTTTAAAAATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "CATCATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "CATCATTTTTTTTTTTTTTTGGGTTTTTTTTTTTTAAAAATTTTTTTTTTTGGGGGTTTTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTGTTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTGGTTTTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTTGTTTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTTTGGTTTTTTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTGGTTTTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTTTTTTGGGGTTTTTTTTTTTTAAAAATTTTTTTTTTTTTTGGGGGTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTTGGGGGGTTTTTTTTTTTTTTAAAAATTTTGGGGGGTTTTTTTTTTTTTTTTTTTTGACAG",
    "TGCATTTTTTTTTTTTTTTTGGGGGGTTTTTTTTTAAAAATTGGGGGGTTTTTTTTTTTTTTTTTTTTTTGACAG",
    "CTGTCAAAAAAAAAAAAAAACAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAACAAAAAAAAAAAACTATC",
    "CTGTCAAAAAAAAAAAAAAAACCAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAACCAAAAAAAAAAAACTATC",
    "CTGTCAAAAAAAAAAAAAAAACCAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAACCAAAAAAAAACTATC",
    "CTGTCAAAAAAAAAAAAAAAAACCCCCAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACTATC",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAACCAAAAAAAAAGTAGA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAACAAAAAAAAAAGTAGA",
    "CTGTCAAAAAAAAAAAAAACAAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAACCAAAAAAAAAAAAAGTAGA",
    "CTGTCAAAAAAAAAAAAAACCCCCAAAAAAAAAAATTTTTAAAAAAAAAAAAAACCCCCAAAAAAAAAAAGTAGA",
    "CTGTCAAAAAAAAAAAAAAAAAACCAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATGATG",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTAAAAAAAAAACCAAAAAAAAAAAAAAAAAATGATG",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATGATG",
    "CTGTCAAAAAAAAAAAAAACCCCCAAAAAAAAAAATTTTTAAAAAAAAAAAACCCAAAAAAAAAAAAAAATGATG",
    "CTGTCAAAAAAAAAAAAAACCAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAAACAAAAAAAAAATGCA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAAAACAAAAAAAAAAATGCA",
    "CTGTCAAAAAAAAAAAAAACCAAAAAAAAAAAAAATTTTTAAAAAAAAAAAAAAAAACCAAAAAAAAAAAATGCA",
    "CTGTCAAAAAAAAAAACCCCCAAAAAAAAAAAAAATTTTTAAAAAAAAAAAACCCCAAAAAAAAAAAAAAATGCA",
    "CTGTCAAAAAAAAAAAAAAAAAAAACCCCCCAAAATTTTTAAAAAAAAAAAAAACCCCCCAAAAAAAAAAATGCA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAACCCCCCAATTTTTAAAAAAAAACCCCCCAAAAAAAAAAAAAAAATGCA"
]

forwardPrimers = [
    "GATAGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT",
    "TCTACTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT",
    "CATCATTTTTTTTTTTTTTTTTTTTTTTTTTTTTT",
    "TGCATTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT"
]

reversePrimers = [
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "CTGTCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
]

forwardNames = ["1", "3", "4", "5"]

reverseNames = ["A", "A", "A", "A"]

using NextGenSeqUtils
demultiplexing = NextGenSeqUtils.deMux(q_seqs,forwardPrimers,reversePrimers,forwardNames,reverseNames)

This all works. Any ideas how I can make this work a bit more smoothly?

3

I revised my workaround solution a bit. Wrote a blog post (mainly for myself to keep track) here - Create Docker and Singularity images for NextGenSeqUtils | by Dean Kayton | Medium