Python pyfiles and cis.read_data() function error

raman_kumar · June 17, 2024, 11:21am

Sorry, Please ignore this topic. This topic is useless and confusing.

I am using python cis.read_data() function to read hdf5 file and its config file. I am getting errors shown in image below:

raman_kumar · June 20, 2024, 4:17pm

Detailed error is

    ~  julia Flux_Ropes.jl                                   ✔  2.7.18  
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Dependencies already up to date
ERROR: LoadError: Python: ClassNotFoundError: Product cannot be found for given file.
Supported products and signatures are:
cis: ['.*\\.nc']
NCAR_NetCDF_RAF: ['.*\\.nc$']
ASCII_Hyperpoints: ['.*\\.txt']
Aeronet: ['.*\\.lev20', '.*\\.ONEILL_lev20', '.*\\.ONEILL_20', '.*\\.lev15', '.*\\.ONEILL_lev15', '.*\\.ONEILL_15', '.*All_Sites_Times.*dat', '.*\\.all']
Aerosol_CCI_L2: ['.*ESACCI-L2P_AEROSOL.*']
Aerosol_CCI_L3: ['.*ESACCI-L3C_AEROSOL.*nc']
Caliop_L1: ['CAL_LID_L1.*hdf']
Caliop_L2: ['CAL_LID_L2_05kmAPro.*hdf']
Caliop_L2_NO_PRESSURE: ['CAL_LID_L2_05kmAPro.*hdf']
CloudSat: ['.*_CS_.*GRANULE.*\\.hdf']
Cloud_CCI_L2: ['.*ESACCI-L2_CLOUD.*']
Cloud_CCI_L3: ['.*ESACCI-L3C_CLOUD.*nc', '.*ESACCI-L3U_CLOUD.*nc']
HadGEM_CONVSH: ['[a-z]{6}[\\._][pamd]{2}[0-9]{4,6}.*\\.nc']
HadGEM_PP: ['.*\\.pp']
MODIS_L2: ['.*MYD06_L2.*\\.hdf', '.*MOD06_L2.*\\.hdf', '.*MYD04_L2.*\\.hdf', '.*MOD04_L2.*\\.hdf']
MODIS_L3: ['.*MYD08_D3.*\\.hdf', '.*MOD08_D3.*\\.hdf', '.*MYD08_M3.*\\.hdf', '.*MOD08_M3.*\\.hdf', '.*MOD08_E3.*\\.hdf']
NetCDF_Gridded: ['.*\\.nc']

Python stacktrace:
 [1] __get_class
   @ cis.data_io.products.AProduct ~/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/site-packages/cis/data_io/products/AProduct.py:180
 [2] get_data
   @ cis.data_io.products.AProduct ~/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/site-packages/cis/data_io/products/AProduct.py:193
 [3] read_data_list
   @ cis.data_io.data_reader ~/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/site-packages/cis/data_io/data_reader.py:90
 [4] read_data_list
   @ cis ~/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/site-packages/cis/__init__.py:70
 [5] read_data
   @ cis ~/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/site-packages/cis/__init__.py:41
Stacktrace:
 [1] pythrow()
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/err.jl:92
 [2] errcheck
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/err.jl:10 [inlined]
 [3] pycallargs(f::Py, args::Py)
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:212
 [4] pycall(::Py, ::String, ::Vararg{Any}; kwargs::@Kwargs{})
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:230
 [5] pycall(::Py, ::String, ::Vararg{Any})
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:220
 [6] (::Py)(::String, ::Vararg{Any}; kwargs::@Kwargs{})
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:339
 [7] top-level scope
   @ ~/Flux_Ropes.jl:23
in expression starting at /home/raman/Flux_Ropes.jl:23

get_class()

def __get_class(filename, product=None):
    """
    Identify the subclass of :class:`.AProduct` to a given product name if specified.
    If the product name is not specified, the routine uses the signature (regex)
    given by :meth:`get_file_signature` to infer the product class from the filename.

    Note, only the first filename of the list is use here.

    :param filename: A single filename
    :param product: name of the product
    :return: a subclass of :class:`.AProduct`
    """
    import re
    import os
    import cis.plugin as plugin
    from cis.exceptions import ClassNotFoundError

    # Ensure the filename doesn't include the path
    basename = os.path.basename(filename)

    product_classes = plugin.find_plugin_classes(AProduct, 'cis.data_io.products')
    product_classes = sorted(product_classes, key=lambda cls: cls.priority, reverse=True)

    for cls in product_classes:

        if product is None:
            # search for a pattern that matches file signature
            class_instance = cls()
            patterns = class_instance.get_file_signature()
            for pattern in patterns:
                # Match the pattern - re.I allows for case insensitive matches.
                # Appending '$' to the pattern ensures we match the whole string
                if re.match(pattern+'$', basename, re.I) is not None:
                    logging.debug("Found product class " + cls.__name__ + " matching regex pattern " + pattern)
                    errors = class_instance.get_file_type_error(filename)
                    if errors is None:
                        return cls
                    else:
                        logging.info("Product class {} is not right because {}".format(cls.__name__, errors))
        else:
            # product specified directly
            if product == cls.__name__:
                logging.debug("Selected product class " + cls.__name__)
                return cls
    error_message = "Product cannot be found for given file.\nSupported products and signatures are:\n"
    for cls in product_classes:
        error_message += cls().__class__.__name__ + ": " + str(cls().get_file_signature()) + "\n"
    raise ClassNotFoundError(error_message)

it works fine with python 2.7

mkitti · June 20, 2024, 4:28pm

Rename the file to .h5 or something. This is a problem with the Python package cis. It is not a Julia problem. Please ask the cis folks.

raman_kumar · June 20, 2024, 4:32pm

Please look at this link. My python code below works fine on python2.7.

from pyfiles import * 
runname='./' #define start dir and file, uncomment below
sane = 1#0 for MAD98, 1 for SANE00
if sane == 1:
    inputfile=runname+'sane00.athinput'
datafile='sane00.prim.01800.athdf'
data=read_data(datafile,athinput=inputfile)

I don’t know how to explain my problem to cis community.

mkitti · June 20, 2024, 4:38pm

If you would like to use HDF5.jl, I might be able to help. However, the error you are seeing comes from the cis Python package.

I’m not sure why you would be using Python 2.7, but that is far outside of support.

I suggest downloading miniforge, creating an environment with cis, and then trying to load this with Python 3. If you have problems there, contact the CIS authors:

Once you have working Python 3 code, please show us what that is.

raman_kumar · June 20, 2024, 4:51pm

See this issue in CIS issues. Python2.7 reads that file while CondaPkg python3.11 don’t read it so there may be something wrong with file address. This .athinput is a .txt file.

mkitti · June 20, 2024, 5:09pm

This is not a good place to address Python issues.

See https://discuss.python.org/

Provide them with the Python code you posted above.

mkitti · June 20, 2024, 5:13pm

Also, I do not understand why you are not using pyfiles to load your data with Python 3.

raman_kumar · June 20, 2024, 5:33pm

Yes, there is line in python 2.7 code that is

from pyfiles import read_athinput

mkitti · June 20, 2024, 5:39pm

Try the same with Python 3.

raman_kumar · June 21, 2024, 2:00am

I am getting error.

julia> using CondaPkg

(@v1.10) pkg> conda add pyfiles
    CondaPkg Found dependencies: /home/raman/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/raman/.julia/packages/PythonCall/S5MOg/CondaPkg.toml
    CondaPkg Resolving changes
             + pyfiles
    CondaPkg Installing packages
             │ /home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
             │ -r /home/raman/micromamba
             │ install
             │ -y
             │ -p /home/raman/.julia/environments/v1.10/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ cis[version='*']
             │ libstdcxx-ng[version='>=3.4,<13.0']
             │ pyfiles[version='*']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge
conda-forge/noarch                                  15.1MB @ 716.6kB/s 21.1s
conda-forge/linux-64                               @ 982.8kB/s 36.2s
error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ pyfiles *  does not exist (perhaps a typo or a missing channel).
critical libmamba Could not solve for environment specs
ERROR: failed process: Process(`/home/raman/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba -r /home/raman/micromamba install -y -p /home/raman/.julia/environments/v1.10/.CondaPkg/env --override-channels --no-channel-priority "cis[version='*']" "libstdcxx-ng[version='>=3.4,<13.0']" "pyfiles[version='*']" "python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']" -c conda-forge`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:398
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:299
  [6] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:287
  [7] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:525
  [8] resolve()
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/resolve.jl:405
  [9] add(pkgs::AbstractVector; resolve::Any, file::Any, kw...)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/deps.jl:222
 [10] add(pkgs::AbstractVector)
    @ CondaPkg ~/.julia/packages/CondaPkg/MmzxM/src/deps.jl:215
 [11] add(args::Vector{String})
    @ CondaPkg.PkgREPL ~/.julia/packages/CondaPkg/MmzxM/src/PkgREPL.jl:125
 [12] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:412
 [13] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:390
 [14] do_cmd
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [15] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:557
 [16] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [17] invokelatest
    @ ./essentials.jl:889 [inlined]
 [18] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2656
 [19] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
 [20] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:386

abraemer · June 21, 2024, 4:55am

This really has nothing to do with Julia anymore. I suggest you get helf in the Python forum @mkitti mentioned until you have a Python envirnoment+script that can load the data. Then you can essentially copy paste script into Julia with PythonCall.jl and get the data into here. With this last part we can help you. But this is the wrong place for your Python problems. Sorry.

raman_kumar · June 21, 2024, 7:43am

I have posted thread in Python forum. But i think it is CondaPkg related issue.

kellertuer · June 21, 2024, 8:06am

I think this reads like you specified python package versions that do conflict each other in some way, so it’s a Python issue (python environment issue).

Benny · June 21, 2024, 8:33am

Does the code you need now run in Python >=3.8 alone? Because your versioning and use of PythonCall is requiring that. PyCall is the package that handles Python 2.7. Only try to mix with Julia after you make the code and environment work in pure Python of a compatible version.

All you did was copy and paste your question from here and linked this thread. You also shouldn’t expect Python users to be familiar with CondaPkg, post a fresh question there or to the cis developers about any issues with the pure Python work.

IIRC CondaPkg was created for PythonCall, does CondaPkg alone share the >=3.8 limit?

raman_kumar · June 21, 2024, 10:52am

I need to run it within Julia script so whatever version it may be, i don’t mind. It would be better if it work in Python>=3.8. My python code given in 4th post works fine in python2.7 and not in 3.12. In Python i am able to install pyfiles easily but got error while installing with CondaPkg in Julia.

    ~  pip install pyfiles                                              ✔ 
Requirement already satisfied: pyfiles in ./.pyenv/versions/3.12.2/lib/python3.12/site-packages (0.3.1)

Is there any plan to have CIS library api in julia in future?

kellertuer · June 22, 2024, 8:23am

There we have the reason and the solution. CondaPkg does not work with Python 2.x
So the code you posted in your 4th post has to be changed to run on Python 3. Then you can use it again from Julia.

For the plan of a CIS library API – I am not aware and could not find any approaches. So you either have to start that yourself and/or find people to do that together with.

Benny · June 22, 2024, 8:45am

pip and conda manage packages differently and thus cannot install from the same repositories. Mixing conda and pip is so brittle there are guidelines to limit it. You should really be learning this stuff on a Python forum, but this is rudimentary enough to mention quickly.

That matches the pyfiles package I found on PyPI, which I said did not appear to have the function you mentioned. Did you attempt to run the line from pyfiles import read_athinput and the rest of the code with this package in Python 3?

raman_kumar · June 22, 2024, 10:57am

@cjdoris said this in github
Make sure you read the full error message! Here is the relevant bit:

The following package could not be installed
└─ pyfiles * does not exist (perhaps a typo or a missing channel).

You’re trying to install a package that isn’t in conda-forge.

What else should i use to install pyfiles so that it works for my case? What is solution to this issue?

Benny · June 22, 2024, 8:25pm

Did you try to install all the packages you need and run all the related Python code you have in a pure Python (aka no Julia at all) environment of any version, as repeatedly suggested in this thread? What are the outcomes? There was already more than one indication you are dealing with limitations of the Python packages themselves, and you’re the only person here who has the code base to verify it. Doing this work properly is vital to determine where you can look for a solution, if it even exists.