Using/workaround for unsupported/EOLd operating systems, e.g. CentOS 6. Was: Test julia basic functionality

General Usage
1

Hi,

I work on CentOS 6 and it is known that it ships with GLIBC 2.12 but some packages requires higher version of it (and julia 1.9.4 itself requires at least GLIBC 2.14).

I updated GLIBC in custom dir (I can’t risque to install it in root dir).
Then using patchelf I updated julia executable and runtime and interpreter and also its library.
So it started to work with the single WARNING: failed to select UTF-8 encoding, using ASCII

I would like to be confident that basic Julia functionality works correctly. So I would try to run some tests for that. Are there such tests?

Also how can I try to solve the issue with the UTF-ASCII warning?

Julia 1.9.4

1 Like
2

You could try to run Base.runtests() although you may want to eventually choose certain tests to run.

The error is being generated from here:

You might also need to set the environment variable LOCPATH.

1 Like
3

Thank you very much! I will try it!

4

Just so people understand correctly, you mean to say that you replaced the requirements for glibc 2.14 with 2.12 using patchelf? Or do you mean that you directed julia to use a non-system copy of glibc 2.14 using patchelf?

5

The default GLIBC that comes with CentOS 6 is 2.12 (it is in /lib64 folder).
Julia 1.9.4 requires at least GLIBC 2.17. Moreover some libraries requires GLIBC at leas 2.14 (like HDF5 as I remember).

I’m not allowed to modify root directories (because it may break some linux functionality) so I can’t simply follow the instructions like this. But if could modify root dirs then I would not need to patch julia but I’m not.

The solution was to build GLIBC from sources and install it to custom directory for example $HOME/glibc.

But to run Julia with custom GLIBC in non-root dir one must patch julia executable and at least julia library so they load our custom built GLIBC at run time.
One can check what libraries an executable/library uses by typing: ldd julia.
After we patched it we can see the paths that lead to $HOME/glibc/lib/somelib.so instead of defalt /lib64/somelib.so.

Of course I want to test whether Julia works correctly after such manipulations.

The commands that I’ve used:

# patch julia executable
patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.17.so   $JULIA
patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib                  	$JULIA

# patch julia lib
patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia            	$JL_ROOT/lib/libjulia.so

# patch all executables in /libexec/julia
patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.17.so   					$JL_ROOT/libexec/julia/7z
patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia            	$JL_ROOT/libexec/julia/7z
patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.17.so   					$JL_ROOT/libexec/julia/dsymutil
patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia           	$JL_ROOT/libexec/julia/dsymutil
patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.17.so   					$JL_ROOT/libexec/julia/lld
patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia            	$JL_ROOT/libexec/julia/lld

and prebuilt patchelf may be downloaded from here

2 Likes
6

Hi kerim,
Thank you very much. I have been going through this exact problem last week.
I have CentOS 6.7 on a cluster given to me, on which I have installed julia 1.10 through juliaup locally. I don’t have sudo access.

When I execute /home/keshav/.juliaup/bin/julia, this is the error I get

/home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/bin/julia: /lib64/libc.so.6: version `GLIBC_2.14’ not found (required by /home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/bin/…/lib/libjulia.so.1.10)

ldd version is 2.12, and ldd /home/keshav/.juliaup/bin/julia doesn;t give me any libraries, like you suggested. It says: statically linked.

For glibc, I have downloaded via anaconda in environment newglibc, through which I have libc-2.19.so in folder /home/keshav/.conda/envs/newglibc/lib/. I got to known about patchelf today (preinstalled version 0.12), and I wish to use these commands suggested by you, which seem most helpful, but I don’t have variables $JULIA, $GLIBC_ROOT or $JL_ROOT defined (as in echo gives nothing).

If I understand correctly, in my case, GLIBC_ROOT=/home/keshav/.conda/envs/newglibc and JULIA=/home/keshav/.juliaup/bin/. Do I understand this correctly? Also, what would be $JL_ROOT?

Also, will these commands need to executed once, or have to be added to .bashrc?

Thank you very much. I was confused, until I found your answer, as setting LD_LIBRARY_PATH=/home/keshav/.conda/envs/newglibc/lib only was giving segmentation fault.

7

Hi,

Glad to hear that.

This should be run only once. No need to modify .bashrc.

I don’t know how conda installs it. In my case I compiled it from source but I believe conda may be used as well.

Basically take a look where GLIBC libraries reside. See in my case the interpreter and other GLIBC libraries were in $GLIBC_ROOT/lib so you should find the directory with it and do the same by analogy (replace my $GLIBC_ROOT/lib with your path to the GLIBC libraries).
$JULIA is the path to the Julia executable.
$JL_ROOT is the directory where you have installed julia: it usually contains bin and lib subfolders

Also you can download more recent prebuilt patchelf 0.18 by this link.

Happy New Year!

2 Likes
8

Thank you very much. This definitely clears it. I will have to try this later, as the same julia binary is used by different cluster (this is a shared folder), which is running a job. I am not sure changing links in between will affect the already running job or not. (Will it?)

Thanks for the updated patchelf. I would also like to be conda independent. Can you share the source from where the GLIBC libraries can be downloaded . (Is it this?)

Thanks again. Happy new year indeed.

1 Like
9

Sorry I didn’t understand the question.
But I also work with cluster and the idea is to build GLIBC and Julia in shared folder so all the nodes can use it. It should work.

I can share the script I use to build GLIBC.
I don’t remember whether the default GCC 4 is able to build GLIBC 2.17 (I guess it should), but if not then you will need either to build GCC from source or use scl:

sudo yum install centos-release-scl
sudo yum install devtoolset-9-gcc*
# scl enable devtoolset-9 bash         # activation is needed for every terminal session

after enabling it you will be able to use GCC-9.

And to build GLIBC:

# ======================================================
# make shared folders for app and general purpose
APP_DIR=$HOME/shared_app

# GLIBC, CentOS 6 ships with 2.12 while Julia and some its packages requires at least 2.14 (CentOS 7 ships with 2.17)
GLIBC_MAJ="2"
GLIBC_MIN="17"
GLIBC_PREFIX=$APP_DIR/glibc/$GLIBC_MAJ.$GLIBC_MIN
GLIBC_ROOT=$GLIBC_PREFIX

# ======================================================
# INSTALL GLIBC FROM SOURCE WITH DEFAULT GCC
wget --no-check-certificate http://ftp.gnu.org/gnu/glibc/glibc-$GLIBC_MAJ.$GLIBC_MIN.tar.gz
tar -zxvf glibc-$GLIBC_MAJ.$GLIBC_MIN.tar.gz

# to overcome the error: 
# ldconfig: Can't open configuration file /opt/glibc-2.14/etc/ld.so.conf: No such file or directory 
mkdir -p $GLIBC_PREFIX/etc
touch $GLIBC_PREFIX/etc/ld.so.conf
sh -c "echo '/usr/local/lib' >> $GLIBC_PREFIX/etc/ld.so.conf"  
sh -c "echo '/opt/lib' >> $GLIBC_PREFIX/etc/ld.so.conf" 

cd glibc-$GLIBC_MAJ.$GLIBC_MIN
mkdir build
cd build
../configure --prefix=$GLIBC_PREFIX
make
make install

# remove unnesessary files/dirs
cd ../..
rm glibc-$GLIBC_MAJ.$GLIBC_MIN.tar.gz
rm -f -r glibc-$GLIBC_MAJ.$GLIBC_MIN
2 Likes
10

I am not sure changing links in between will affect the already running job or not. (Will it?)

I mean I have two clusters c1 and c2. While c1 has outdated GLIBC_2.12, c2 libraries are updated enough, and julia works fine on it ( CentOS 7 and ldd --version is 2.17). So I submitted a job there using script file with julia code.jl line where code.jl runs on c2, where produces outputs in some output file. Now when the code is running on c2, if I use patchefl on the same julia binaries now (so I can run jobs on c1 as well), will the jobs already submitted be affected? Basically I don’t know if I can interfere with binary links, while the job is on running which uses that binary.

Thanks again.

11

Since the Julia executable already has a RUNPATH perhaps you could simply copy or symlink libc.so.6 into the lib/julia folder instead of using patchelf.

$ readelf -d julia-1.10.0/bin/julia | grep RUNPATH     0x000000000000001d (RUNPATH)            
Library runpath: [$ORIGIN/../lib:$ORIGIN/../lib/julia]

$ ln -s /lib/aarch64-linux-gnu/libc.so.6 julia-1.10.0/lib/julia/

$ ldd julia-1.10.0/bin/julia                           
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007451b29000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007451af8000)
        libc.so.6 => ~/julia-1.10.0/bin/../lib/julia/libc.so.6 (0x0000007451982000)
        libjulia.so.1.10 => ~/julia-1.10.0/bin/../lib/libjulia.so.1.10 (0x000000745194d000)
        /lib/ld-linux-aarch64.so.1 (0x0000003000000000)

Also, have you checked the cluster modules system to see jf there is one that loads a newer libc?

edit:
Link to notes about dynamic shared library search order, including runpath:

12

Hi Mark,

Thank you. The given link is informative, and will help me understand these stuff better.

I tried what you suggested, but running julia after creating a symbolic link to ..../lib/julia , gives segmentation fault now.

Also the maximum gcc version the module has is gcc/8.2.0 , and

(base) [keshav@c1-m1 ~]$ module avail

-------------------------------------------------- /usr/share/Modules/modulefiles --------------------------------------------------
dot         module-git  module-info modules     null        use.own

--------------------------------------------------------- /etc/modulefiles ---------------------------------------------------------
mpich-x86_64

-------------------------------------------------------- /c1scratch/Module --------------------------------------------------------
anaconda3/21      gcc/5.4.0         gromacs/5.1.5     intel/2016        openmpi/4.0.1     python/3.8        qe/7.0
anaconda3/23      gcc/8.2.0         gsl/2.6           intel/2018        python/2.7.16     python/3.9        qe/7.1
armadillo/9.900.2 gnuplot/5.2.7     hdf5/1.8.21       lib/openssl       python/3.11       qe/6.3            vasp/5.4.4
cmake/3.21.1      gromacs/2019.6    intel/2013        mathematica/11    python/3.6.5      qe/6.4
(base) [keshav@c1-m1 ~]$

So I think there is any no newer version of libc.

13

My rather offensive answer is to consider upgrading the CentOS version.
I know fine well that saying this is a lot easier than getting existing software packages compatible with a higher CentOS or better still Rocky release.

However, have you looked at Spack and Easybuild? They will download and install locally the gcc versions which you need to build a given package. I am less sure about downloading glibc versions - it would be interesting to find out.

Spack - Spack

2 Likes
14

Spack will bring in quite a few libraries at the correct versions. I see no glibc though

pkgs = [
“curl”,
“dsfmt”,
“gmp”,
“libgit2”,
“libssh2”,
“libunwind”,
“mbedtls”,
“mpfr”,
“nghttp2”,
“openblas”,
“pcre2”,
“suite-sparse”,
“utf8proc”,
]

spack/var/spack/repos/builtin/packages/julia/package.py at develop · spack/spack · GitHub

There is an open issue Have Spack compile its own libc · Issue #39560 · spack/spack · GitHub

15

Hi john,

Unfortunately, the Sys admin has refused to update CentOS.
I will look into spack. Thanks

16

CentOS 6 has been EOL (end-of-life) for over 3 years now.

2 Likes
17

Could you get your sysadmin to install Singularity on your cluster? If so, maybe you could launch a Singularity container with a newer glibc?

1 Like
18

Hi kerim,

Thank you for glibc. I guess I will have to try to build it from scratch. Unfortunately, I don’t have sudo access, so have to see what I can do.

Here are the commands I used

export JL_ROOT=/home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu
export GLIBC_ROOT=/home/keshav/.conda/envs/newglibc
export JULIA=/home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/bin/julia

# patch julia executable
/home/keshav/local/bin/patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.19.so                  $JULIA
/home/keshav/local/bin/patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib                $JULIA

# patch julia lib
/home/keshav/local/bin/patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia          $JL_ROOT/lib/libjulia.so

# patch all executables in /libexec/julia
/home/keshav/local/bin/patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.19.so                  $JL_ROOT/libexec/julia/7z
/home/keshav/local/bin/patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia          $JL_ROOT/libexec/julia/7z
/home/keshav/local/bin/patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.19.so                  $JL_ROOT/libexec/julia/dsymutil
/home/keshav/local/bin/patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia          $JL_ROOT/libexec/julia/dsymutil
/home/keshav/local/bin/patchelf --set-interpreter   $GLIBC_ROOT/lib/ld-2.19.so                  $JL_ROOT/libexec/julia/lld
/home/keshav/local/bin/patchelf --set-rpath         $GLIBC_ROOT/lib:$JL_ROOT/lib/julia          $JL_ROOT/libexec/julia/lld

They didn’t gave any errors.
After that, executing julia gives the following error

Inconsistency detected by ld.so: get-dynamic-info.h: 134: elf_get_dynamic_info: Assertion `info[15] == ((void *)0)' failed!

of which I have no idea about.

ldd /home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/bin/julia

gives the following

    linux-vdso.so.1 =>  (0x00007fff1e548000)
	libdl.so.2 => /home/keshav/.conda/envs/newglibc/lib/libdl.so.2 (0x00007f3b05dd8000)
	libpthread.so.0 => /home/keshav/.conda/envs/newglibc/lib/libpthread.so.0 (0x00007f3b05bb9000)
	libc.so.6 => /home/keshav/.conda/envs/newglibc/lib/libc.so.6 (0x00007f3b0580e000)
	libjulia.so.1.10 => /home/keshav/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/lib/libjulia.so.1.10 (0x00007f3b057e4000)
	/home/keshav/.conda/envs/newglibc/lib/ld-2.19.so => /lib64/ld-linux-x86-64.so.2 (0x0000003f7aa00000)

I have no idea what it is about.

19

I will ask him. Thanks a lot.

20

Hi,

I can see two ways of resolving this issue:

  1. try to use GLIBC 2.17 with Julia 1.9.4 (that was my setup that seems to be working)
  2. investigate the detected problem

Probably you should start with the second option.
First of all reinstall Julia.
Then do ldd to julia executable and julia library. It would be nice if share the output of these commands.
After that try to patch only Julia executable and try to run it. I guess there will be an exception (you can share it here as well).
Then patch library in julia/lib folder and see if that works.
Probably patching executables in /libexec/julia is not necessary. But you should be sure that Julia uses only GLIBC 2.19 only that means you should find its libraries ldd them. Probably you will need to use some more functionality of patchelf like --add-needed or --replace-needed (see the list of them in the github repo).

Generally saying the assertion that you get is most likely because Julia tries to use two GLIBC version at the same time.

And one more thing: please be sure to have julia 64 bit (I dont know wether there is julia 32 bit but still) and compile GLIBC with GCC 64 bit compiler.

By the way I use jill.py project to install Julia.

2 Likes