Inconsistent CondaPkg environments

I’m having some issues with CondaPkg.jl in order to get a conda environment configured and running for one of my packages. I basically have a list of dependencies, which if I add to an environment.yml file and and create an conda environment with those using micromamba everything works great.

However, if I try to mimic exactly the same environment.yml file with the CondaPkg.toml file, even if I pin the exact same versions I don’t get the same behaviour.

This is the conda environment I’m trying to reproduce:

name: oggm_env
channels:
  - conda-forge
dependencies:
  - python<3.12
  - numpy<2.0
  - scipy
  - pandas
  - shapely
  - matplotlib
  - Pillow
  - netcdf4
  - scikit-image
  - configobj
  - xarray
  - pytest
  - dask
  - bottleneck
  - pyproj
  - cartopy
  - geopandas
  - rasterio
  - rioxarray
  - seaborn
  - pytables
  - salem
  - motionless
  - pip
  - pip:
    - joblib
    - progressbar2
    - git+https://github.com/OGGM/pytest-mpl
    - oggm

and this is how my CondaPkg.toml file looks like:

[deps]
bottleneck = ""
netcdf4 = ""
pyproj = ""
matplotlib = ""
pytables = ""
pip = ""
configobj = ""
seaborn = ""
rioxarray = ""
pandas = ""
dask = ""
xarray = ""
python = "<3.12"
Pillow = ""
rasterio = ""
cartopy = ""
salem = ""
geopandas = ""
pytest = ""
shapely = ""
motionless = ""
numpy = "<2.0"
scipy = ""
scikit-image = ""

[pip.deps]
oggm = ""
certifi = ""
progressbar2 = ""
joblib = ""
pytest-mpl = "@ git+https://github.com/OGGM/pytest-mpl"
MBSandbox = "@ git+https://github.com/OGGM/massbalance-sandbox"

[extras]
channels = ["conda-forge"]

For some strange reason, when CondaPkg.jl installs it automatically using micromamba, it doesn’t produce the same library versions as if I do it directly with micromamba. This results in the following error when using the oggm library which calls pandas, which in turn has pytables as a dependency:

ERROR: Python: ImportError: Pandas requires version '3.8.0' or newer of 'tables' (version '3.7.0' currently installed).

If I try to pin pytables to >=3.8, it just never manages to solve the environment. Strangely, this doesn’t happen directly with micromamba, and version 3.10 is installed there.

Any ideas what could explain these inconsistencies and how to tackle this problem? Strangely, this used to work a few weeks ago, but something must have changed in the process. Thanks!

One thing is that the pip deps aren’t quite the same. You have MBSandbox in one but not the other.

CondaPkg prints out the commands it runs to create the environment. I suggest starting with running these commands yourself, then changing the arguments to be closer to what you were originally doing. Some experimenting like this should reveal where the difference arises.

Yes, I checked that. The main differences are in some commands that are automatically done by PythonCall.jl, even if they are not in the CondaPkg.toml.

These are the commands that are run:

CondaPkg Creating environment
             │ /Users/Bolib001/.julia/artifacts/c26faeb828dad2bd10fa4beb155cf9bbde1c25bb/bin/micromamba
             │ -r /Users/Bolib001/micromamba
             │ create
             │ -y
             │ -p /Users/Bolib001/.julia/dev/Sleipnir/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ bottleneck[version='*']
             │ cartopy[version='*']
             │ configobj[version='*']
             │ dask[version='*']
             │ geopandas[version='*']
             │ matplotlib[version='*']
             │ motionless[version='*']
             │ netcdf4[version='*']
             │ numpy[version='<2.0']
             │ openssl[version='>=3, <3.1']
             │ pandas[version='*']
             │ pillow[version='*']
             │ pip[version='*']
             │ pyproj[version='*']
             │ pytables[version='*']
             │ pytest[version='*']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             │ python[version='<3.12']
             │ rasterio[version='*']
             │ rioxarray[version='*']
             │ salem[version='*']
             │ scikit-image[version='*']
             │ scipy[version='*']
             │ seaborn[version='*']
             │ shapely[version='*']
             │ xarray[version='*']
             └ -c conda-forge

I see that openssl is pinned (I think due to some issue that I found on GitHub), and python is pinned twice, first with my specs and then automatically with python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']. The rest of the commands are exactly as the ones I’ve created directly with micromamba.

Any clues where this could come from?

PS: Yes, I forgot to add MBsandbox, but it doesn’t change the result.

1 Like

Also, I realized that micromamba installs the packages in two different channels, conda-forge and pypi.

# packages in environment at /Users/Bolib001/micromamba/envs/oggm_env:
#
# Name                    Version                   Build  Channel
affine                    2.4.0                    pypi_0    pypi
aom                       3.9.1                hf036a51_0    conda-forge
attrs                     24.2.0                   pypi_0    pypi
aws-c-auth                0.7.26               h77ec9d9_2    conda-forge
aws-c-cal                 0.7.4                h3e75f19_0    conda-forge
aws-c-common              0.9.27               hfdf4475_0    conda-forge
aws-c-compression         0.2.19               h3e75f19_0    conda-forge
aws-c-event-stream        0.4.3                h324d61a_0    conda-forge
aws-c-http                0.8.8                h504e0bf_1    conda-forge
aws-c-io                  0.14.18              hef79b51_7    conda-forge
aws-c-mqtt                0.10.4              h03607b6_18    conda-forge
aws-c-s3                  0.6.4               ha70045c_11    conda-forge
aws-c-sdkutils            0.1.19               hf37c103_2    conda-forge
aws-checksums             0.1.18              hf37c103_10    conda-forge
aws-crt-cpp               0.28.0               h617b8c7_1    conda-forge
aws-sdk-cpp               1.11.379             hf77b78f_6    conda-forge
azure-core-cpp            1.13.0               hf8dbe3c_0    conda-forge
azure-identity-cpp        1.8.0                h60298e3_2    conda-forge
azure-storage-blobs-cpp   12.12.0              h646f05d_0    conda-forge
azure-storage-common-cpp  12.7.0               hf91904f_1    conda-forge
azure-storage-files-datalake-cpp 12.11.0              h14965f0_1    conda-forge
blosc                     1.21.6               h7d75f6d_0    conda-forge
bokeh                     3.5.2                    pypi_0    pypi
bottleneck                1.4.0                    pypi_0    pypi
branca                    0.7.2                    pypi_0    pypi
brotli                    1.1.0                    pypi_0    pypi
brotli-bin                1.1.0                h0dc2134_1    conda-forge
brotli-python             1.1.0           py311hdf8f085_1    conda-forge
brunsli                   0.1                  h046ec9c_0    conda-forge
bzip2                     1.0.8                hfdf4475_7    conda-forge
c-ares                    1.33.1               h44e7173_0    conda-forge
c-blosc2                  2.15.1               hb9356d3_0    conda-forge
ca-certificates           2024.7.4             h8857fd0_0    conda-forge
cairo                     1.18.0               h37bd5c4_3    conda-forge
cartopy                   0.23.0                   pypi_0    pypi
certifi                   2024.7.4                 pypi_0    pypi
cffi                      1.17.0                   pypi_0    pypi
cfitsio                   4.4.1                ha105788_0    conda-forge
cftime                    1.6.4                    pypi_0    pypi
charls                    2.4.2                he965462_0    conda-forge
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
click-plugins             1.1.1                    pypi_0    pypi
cligj                     0.7.2                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
configobj                 5.0.8                    pypi_0    pypi
contourpy                 1.3.0                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cytoolz                   0.12.3                   pypi_0    pypi
dask                      2024.8.1                 pypi_0    pypi
dask-core                 2024.8.1           pyhd8ed1ab_0    conda-forge
dask-expr                 1.1.11                   pypi_0    pypi
dav1d                     1.2.1                h0dc2134_0    conda-forge
distributed               2024.8.1                 pypi_0    pypi
exceptiongroup            1.2.2                    pypi_0    pypi
expat                     2.6.2                h73e2aa4_0    conda-forge
fmt                       11.0.2               h3c5361c_0    conda-forge
folium                    0.17.0                   pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_2    conda-forge
fontconfig                2.14.2               h5bb23bf_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.53.1                   pypi_0    pypi
freetype                  2.12.1               h60636b9_2    conda-forge
freexl                    2.0.0                h3ec172f_0    conda-forge
fsspec                    2024.6.1                 pypi_0    pypi
gdal                      3.9.2                    pypi_0    pypi
geopandas                 1.0.1                    pypi_0    pypi
geopandas-base            1.0.1              pyha770c72_0    conda-forge
geos                      3.12.2               hf036a51_1    conda-forge
geotiff                   1.7.3                h4bbec01_2    conda-forge
gflags                    2.2.2             hb1e8313_1004    conda-forge
giflib                    5.2.2                h10d778d_0    conda-forge
glog                      0.7.1                h2790a97_0    conda-forge
h2                        4.1.0                    pypi_0    pypi
hdf4                      4.2.15               h8138101_7    conda-forge
hdf5                      1.14.3          nompi_h687a608_105    conda-forge
hpack                     4.0.0                    pypi_0    pypi
hyperframe                6.0.1                    pypi_0    pypi
icu                       75.1                 h120a0e1_0    conda-forge
idna                      3.8                      pypi_0    pypi
imagecodecs               2024.6.1                 pypi_0    pypi
imageio                   2.35.1                   pypi_0    pypi
importlib-metadata        8.4.0                    pypi_0    pypi
importlib_metadata        8.4.0                hd8ed1ab_0    conda-forge
iniconfig                 2.0.0                    pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
json-c                    0.17                 h6253ea5_1    conda-forge
jxrlib                    1.1                  h10d778d_3    conda-forge
kealib                    1.5.3                hb2b617a_1    conda-forge
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.21.3               h37d8d59_0    conda-forge
lazy-loader               0.4                      pypi_0    pypi
lazy_loader               0.4                pyhd8ed1ab_1    conda-forge
lcms2                     2.16                 ha2f27b4_0    conda-forge
lerc                      4.0.0                hb486fe8_0    conda-forge
libabseil                 20240116.2      cxx17_hf036a51_1    conda-forge
libaec                    1.1.3                h73e2aa4_0    conda-forge
libarchive                3.7.4                h20e244c_0    conda-forge
libarrow                  17.0.0          h3be88d5_11_cpu    conda-forge
libarrow-acero            17.0.0          hac325c4_11_cpu    conda-forge
libarrow-dataset          17.0.0          hac325c4_11_cpu    conda-forge
libarrow-substrait        17.0.0          hba007a9_11_cpu    conda-forge
libavif16                 1.1.1                ha49a9e2_1    conda-forge
libblas                   3.9.0           22_osx64_openblas    conda-forge
libbrotlicommon           1.1.0                h0dc2134_1    conda-forge
libbrotlidec              1.1.0                h0dc2134_1    conda-forge
libbrotlienc              1.1.0                h0dc2134_1    conda-forge
libcblas                  3.9.0           22_osx64_openblas    conda-forge
libcrc32c                 1.1.2                he49afe7_0    conda-forge
libcurl                   8.9.1                hfcf2730_0    conda-forge
libcxx                    18.1.8               hd876a4e_6    conda-forge
libdeflate                1.21                 hfdf4475_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 h10d778d_2    conda-forge
libevent                  2.1.12               ha90c15b_1    conda-forge
libexpat                  2.6.2                h73e2aa4_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgdal                   3.9.2                h694c41f_0    conda-forge
libgdal-core              3.9.2                h4b9bb65_0    conda-forge
libgdal-fits              3.9.2                h5d197d2_0    conda-forge
libgdal-grib              3.9.2                h385febf_0    conda-forge
libgdal-hdf4              3.9.2                h86719f3_0    conda-forge
libgdal-hdf5              3.9.2                h513f0eb_0    conda-forge
libgdal-jp2openjpeg       3.9.2                hc5f35ca_0    conda-forge
libgdal-kea               3.9.2                h3b8d0bf_0    conda-forge
libgdal-netcdf            3.9.2                h3127c03_0    conda-forge
libgdal-pdf               3.9.2                ha7d2355_0    conda-forge
libgdal-pg                3.9.2                h1b48671_0    conda-forge
libgdal-postgisraster     3.9.2                h1b48671_0    conda-forge
libgdal-tiledb            3.9.2                ha63beff_0    conda-forge
libgdal-xls               3.9.2                h597966e_0    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_3    conda-forge
libgfortran5              13.2.0               h2873a65_3    conda-forge
libglib                   2.80.3               h736d271_2    conda-forge
libgoogle-cloud           2.28.0               h721cda5_0    conda-forge
libgoogle-cloud-storage   2.28.0               h9e84e37_0    conda-forge
libgrpc                   1.62.2               h384b2fc_0    conda-forge
libhwy                    1.1.0                h7728843_0    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
libintl                   0.22.5               hdfe23c8_3    conda-forge
libjpeg-turbo             3.0.0                h0dc2134_1    conda-forge
libjxl                    0.10.3               hfb90b89_0    conda-forge
libkml                    1.3.0             h9ee1731_1021    conda-forge
liblapack                 3.9.0           22_osx64_openblas    conda-forge
libnetcdf                 4.9.2           nompi_h7334405_114    conda-forge
libnghttp2                1.58.0               h64cf6d3_1    conda-forge
libopenblas               0.3.27          openmp_h8869122_1    conda-forge
libparquet                17.0.0          hf1b0f52_11_cpu    conda-forge
libpng                    1.6.43               h92b6c6a_0    conda-forge
libpq                     16.4                 h4501773_0    conda-forge
libprotobuf               4.25.3               h4e4d658_0    conda-forge
libre2-11                 2023.09.01           h81f5012_2    conda-forge
librttopo                 1.1.0               he2ba7a0_16    conda-forge
libspatialite             5.1.0                hdc25a2c_9    conda-forge
libsqlite                 3.46.0               h1b8f9f3_0    conda-forge
libssh2                   1.11.0               hd019ec5_0    conda-forge
libthrift                 0.20.0               h75589b3_1    conda-forge
libtiff                   4.6.0                h603087a_4    conda-forge
libutf8proc               2.8.0                hb7f2c08_0    conda-forge
libwebp-base              1.4.0                h10d778d_0    conda-forge
libxcb                    1.16                 h00291cd_1    conda-forge
libxml2                   2.12.7               heaf3512_4    conda-forge
libzip                    1.10.1               hc158999_3    conda-forge
libzlib                   1.3.1                h87427d6_1    conda-forge
libzopfli                 1.0.3                h046ec9c_0    conda-forge
llvm-openmp               18.1.8               h15ab845_1    conda-forge
locket                    1.0.0                    pypi_0    pypi
lz4                       4.3.3                    pypi_0    pypi
lz4-c                     1.9.4                hf0c8a7f_0    conda-forge
lzo                       2.10              h10d778d_1001    conda-forge
mapclassify               2.8.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.9.2                    pypi_0    pypi
matplotlib-base           3.9.2           py311hf31e254_0    conda-forge
mbsandbox                 0.0.1                     dev_0    <develop>
minizip                   4.0.7                h62b0c8d_0    conda-forge
motionless                1.3.3                    pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
msgpack-python            1.0.8           py311h46c8309_0    conda-forge
munkres                   1.1.4                    pypi_0    pypi
ncurses                   6.5                  hf036a51_1    conda-forge
netcdf4                   1.7.1                    pypi_0    pypi
networkx                  3.3                      pypi_0    pypi
nspr                      4.35                 hea0b92c_0    conda-forge
nss                       3.103                he7eb89d_0    conda-forge
numexpr                   2.10.0                   pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
oggm                      1.6.2                    pypi_0    pypi
openjpeg                  2.5.2                h7310d3a_0    conda-forge
openssl                   3.3.1                hd23fc13_3    conda-forge
orc                       2.0.2                h22b2039_0    conda-forge
packaging                 24.1                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
partd                     1.4.2                    pypi_0    pypi
patsy                     0.5.6                    pypi_0    pypi
pcre2                     10.44                h7634a1b_2    conda-forge
pillow                    10.4.0                   pypi_0    pypi
pip                       24.2                     pypi_0    pypi
pixman                    0.43.4               h73e2aa4_0    conda-forge
pluggy                    1.5.0                    pypi_0    pypi
poppler                   24.08.0              h65860a0_1    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                16.4                 h9b73963_0    conda-forge
progressbar2              4.5.0                    pypi_0    pypi
proj                      9.4.1                hf92c781_1    conda-forge
psutil                    6.0.0                    pypi_0    pypi
pthread-stubs             0.4               hc929b4f_1001    conda-forge
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   17.0.0                   pypi_0    pypi
pyarrow-core              17.0.0          py311h073f6b9_1_cpu    conda-forge
pyarrow-hotfix            0.6                      pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pyogrio                   0.9.0                    pypi_0    pypi
pyparsing                 3.1.4                    pypi_0    pypi
pyproj                    3.6.1                    pypi_0    pypi
pyshp                     2.3.1                    pypi_0    pypi
pysocks                   1.7.1                    pypi_0    pypi
pytables                  3.10.1          py311h2310c21_1    conda-forge
pytest                    8.3.2                    pypi_0    pypi
pytest-mpl                0.170.0                  pypi_0    pypi
python                    3.11.9          h657bba9_0_cpython    conda-forge
python-dateutil           2.9.0                    pypi_0    pypi
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python-utils              3.8.2                    pypi_0    pypi
python_abi                3.11                    5_cp311    conda-forge
pytz                      2024.1                   pypi_0    pypi
pywavelets                1.7.0                    pypi_0    pypi
pyyaml                    6.0.2                    pypi_0    pypi
qhull                     2020.2               h3c5361c_5    conda-forge
rasterio                  1.3.10                   pypi_0    pypi
rav1e                     0.6.6                h7205ca4_2    conda-forge
re2                       2023.09.01           hb168e87_2    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
requests                  2.32.3                   pypi_0    pypi
rioxarray                 0.17.0                   pypi_0    pypi
salem                     0.3.11                   pypi_0    pypi
scikit-image              0.24.0                   pypi_0    pypi
scikit-learn              1.5.1                    pypi_0    pypi
scipy                     1.14.1                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
seaborn-base              0.13.2             pyhd8ed1ab_2    conda-forge
setuptools                72.2.0                   pypi_0    pypi
shapely                   2.0.6                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
snappy                    1.2.1                he1e6707_0    conda-forge
snuggs                    1.4.7                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
spdlog                    1.14.1               h325aa07_1    conda-forge
sqlite                    3.46.0               h28673e1_0    conda-forge
statsmodels               0.14.2                   pypi_0    pypi
svt-av1                   2.2.1                hac325c4_0    conda-forge
tables                    3.10.1                   pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tifffile                  2024.8.28                pypi_0    pypi
tiledb                    2.25.0               h5683d06_9    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzcode                    2024a                h10d778d_0    conda-forge
tzdata                    2024.1                   pypi_0    pypi
uriparser                 0.9.8                h6aefe2f_0    conda-forge
urllib3                   2.2.2                    pypi_0    pypi
wheel                     0.44.0                   pypi_0    pypi
xarray                    2024.7.0                 pypi_0    pypi
xerces-c                  3.2.5                hfb503d4_1    conda-forge
xorg-libxau               1.0.11               h0dc2134_0    conda-forge
xorg-libxdmcp             1.1.3                h35c211d_0    conda-forge
xyzservices               2024.6.0                 pypi_0    pypi
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
zfp                       1.0.1                h28dbb38_1    conda-forge
zict                      3.0.0                    pypi_0    pypi
zipp                      3.20.1                   pypi_0    pypi
zlib                      1.3.1                h87427d6_1    conda-forge
zlib-ng                   2.2.1                hf036a51_0    conda-forge
zstandard                 0.23.0                   pypi_0    pypi
zstd                      1.5.6                h915ae27_0    conda-forge

I added this channel into the CondaPkg.toml but still all the packages were installed in conda-forge.

I think the issue might come from automatically pinning openssl, because if I pin the versions like in micromamba I get:

error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ netcdf4 1.7.1  is installable and it requires
    │  └─ hdf5 [>=1.14.3,<1.14.4.0a0 |>=1.14.3,<1.14.4.0a0 mpi_mpich_*|>=1.14.3,<1.14.4.0a0 mpi_openmpi_*] with the potential options
    │     ├─ hdf5 1.14.3 would require
    │     │  └─ openssl >=3.2.0,<4.0a0 , which can be installed;
    │     ├─ hdf5 1.14.3 would require
    │     │  └─ openssl >=3.3.0,<4.0a0 , which can be installed;
    │     └─ hdf5 1.14.3 would require
    │        └─ openssl >=3.3.1,<4.0a0 , which can be installed;
    └─ openssl >=3,<3.1  is not installable because it conflicts with any installable versions previously reported.
critical libmamba Could not solve for environment specs

These both from PythonCall’s CondaPkg.toml file, because PythonCall has these as dependencies.

In particular the OpenSSL constraint is chosen assuming that Julia loads OpenSSL before Python does. This means that the Python OpenSSL package has to be no newer than the Julia OpenSSL package, otherwise the Python one will try to use features from a newer version of OpenSSL than is available (because the Julia OpenSSL library is the one actually loaded).

Unfortunately the newest version of OpenSSL_jll is 3.0.* so this means you can’t use Python OpenSSL above 3.0.

None of this is directly related to your original issue with pandas/parables, but it might be the root cause if one of these packages declared their dependencies too loosely.

Yes, the error seems to have been reduced to this. I installed the environment manually using micromamba by pinning the versions and everything worked well. So it boils down to this version of OpenSSL which is too restrictive.

Is there any way to work around this? This is really blocking and it has blown up our CI. Thanks!

I can’t think of an existing workaround, other than pinning to an older version of CondaPkg.

I think a good workaround would be if there were settings to override some dependencies but that’s not in CondaPkg yet.

Ok, do you happen to know which version of CondaPkg would do the trick? Thanks!

See the CondaPkg change log.

OK, thanks. We pinned CondaPkg to version 0.2.22 and the problem went away as expected.

We are currently facing other issues related to the rather complicated architecture of our packages, which are split into multiple ones. Basically, our use case is the following one:

ODINN.jl
/
Huginn.jl
/
Muninn.jl
/
Sleipnir.jl → calls PythonCall.jl and CondaPkg.jl

We have a low-level package (i.e. Sleipnir.jl) which handles the Python dependencies using PythonCall.jl and CondaPkg.jl. Then, this package is called by other higher-level packages in a nested way (i.e. Muninn.jl, Huginn.jl and ODINN.jl). Since by default, PythonCall is creating a .CondaPkg folder at the root level of each package, basically the structure keeps repeating for each on of the packages which has the low-level package as a dependency. I looked this up in the PythonCall or CondaPkg docs, but I couldn’t find a usecase like that. Ideally we would like to keep the Python dependencies ONLY in the low level package, and they by using @reexport, to have access to those directly from the rest of packages. By doing this, we’re encountering some issues with the paths that we don’t have when we directly run PythonCall on the low level package.

Is there any template, example, or recommendations on how to handle these cases? I guess a lot can go wrong when moving through different modules, and knowing that each one of the nested packages will have a .CondaPkg folder.

Thanks in advance!

2 Likes

This isn’t quite right, CondaPkg creates single .CondaPkg folder at the root of the active project to house all of that project’s dependencies.

However if you start Julia in your ODINN project, then in your Huginn project, then in your Muninn project, you will indeed get a .CondaPkg folder in each of those.

CondaPkg doesn’t currently do anything to reuse environments that share the same set of dependencies. For that you’ll have to explicitly specify the path where to put the environment - see the env preference.

I am having this same issue when using PythonCall.

Thank you @cjdoris for you comment about the incompatibility problem between the SSL versions used by Python and Julia, but there is any work around this? I am fixing my CondaPkg=0.2.22 but I also have the same problem… maybe using an old version of Julia can help?

I am just concern this will be a problem also in future releases of Julia/Python/SSL, so I am trying to think about an stable solution here.

The long-term best solution is for someone to update the build script in Yggdrasil to build a newer version of OpenSSL_jll. That someone could be you!

And then preferably for Julia to ship with it by default.

A solution that CondaPkg could offer is to add a setting to override this behaviour. I’ve been meaning to add various overrides for a while.

We’re sticking to the LTS of OpenSSL, which until 2026-09-07 will be OpenSSL 3.0.

Fair enough. We could still build later versions of OpenSSL_jll though right so people can opt-in to later versions? Or is OpenSSL_jll not upgradeable?

We’re trying to make it a stdlib in Try migration to OpenSSL by fxcoudert · Pull Request #53891 · JuliaLang/julia · GitHub (where help from maintainers of Python interface packages would be most helpful, BTW)

1 Like

Oh so OpenSSL_jll isn’t a stdlib, I had the impression it was? If it’s just some ordinary package, then what is stopping us from building and releasing higher versions of OpenSSL_jll? When you said “we’re sticking to the LTS of OpenSSL”, who are “we”?

The plan to make it a stdlib, and sticking to the LTS, for security reasons.