Python downstream testing, can't figure out CI to install the libraries?

General Usage
1

For our diffeqpy library, we used to have downstream testing setup in SciMLBase.jl so that way our core changes wouldn’t break the core functionality when accessed through Python. However, it’s not working now after the removal of PyCall.jl, and I’m not entirely sure why. The core code is here:

using DifferentialEquations, PythonCall

pyexec(
    """ # This is a mess because normal site-packages is not writeable in CI
import subprocess, sys, site
subprocess.run([sys.executable, '-m', 'pip', 'install', '--user', 'julia'])
subprocess.run([sys.executable, '-m', 'pip', 'install', '--user', 'diffeqpy'])
sys.path.append(site.getusersitepackages())
""", @__MODULE__)

then we use it:

    pyexec("""
    from diffeqpy import ode

    def simple_system_b(du, u, p, t):
        du[0] = -1*p[0]*u[0]

    def condition(u, t, integrator):
        return t == 0.5

    def affect_b(integrator):
        integrator.u[0] += 10

    cb = ode.DiscreteCallback(
        condition, affect_b
    )

but that fails

Use of DifferentialEquations through PythonCall with user code written in Python: Error During Test at /home/runner/work/SciMLBase.jl/SciMLBase.jl/test/python/pythoncall.jl:11
  Got exception outside of a @test
  Python: ModuleNotFoundError: No module named 'diffeqpy'
  Python stacktrace:
   [1] <module>
     @ <string>:1
  Stacktrace:
    [1] pythrow()
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/err.jl:92
    [2] errcheck
      @ ~/.julia/packages/PythonCall/WMWY0/src/Core/err.jl:10 [inlined]
    [3] pycallargs(f::PythonCall.Core.Py, args::PythonCall.Core.Py)
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:220
    [4] pycall(::PythonCall.Core.Py, ::String, ::Vararg{Any}; kwargs::@Kwargs{})
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:243
    [5] pycall(::PythonCall.Core.Py, ::String, ::Vararg{Any})
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:233
    [6] (::PythonCall.Core.Py)(::String, ::Vararg{Any}; kwargs::@Kwargs{})
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/Py.jl:357
    [7] pyexec
      @ ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:1326 [inlined]
    [8] pyexec
      @ ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:1331 [inlined]
    [9] pyexec(code::String, globals::Module)
      @ PythonCall.Core ~/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:1331
   [10] macro expansion
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/python/pythoncall.jl:84 [inlined]
   [11] macro expansion
      @ /opt/hostedtoolcache/julia/1.11.4/x64/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
   [12] top-level scope
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/python/pythoncall.jl:12
   [13] include(mod::Module, _path::String)
      @ Base ./Base.jl:557
   [14] include(x::String)
      @ Main.var"##PythonCall#267" ~/.julia/packages/SafeTestsets/raUNr/src/SafeTestsets.jl:28
   [15] macro expansion
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/runtests.jl:158 [inlined]
   [16] macro expansion
      @ /opt/hostedtoolcache/julia/1.11.4/x64/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
   [17] macro expansion
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/runtests.jl:158 [inlined]
   [18] top-level scope
      @ ~/.julia/packages/SafeTestsets/raUNr/src/SafeTestsets.jl:30
   [19] eval(m::Module, e::Any)
      @ Core ./boot.jl:430
   [20] macro expansion
      @ ~/.julia/packages/SafeTestsets/raUNr/src/SafeTestsets.jl:28 [inlined]
   [21] macro expansion
      @ ./timing.jl:581 [inlined]
   [22] macro expansion
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/runtests.jl:157 [inlined]
   [23] macro expansion
      @ ./timing.jl:581 [inlined]
   [24] top-level scope
      @ ~/work/SciMLBase.jl/SciMLBase.jl/test/runtests.jl:315
   [25] include(fname::String)
      @ Main ./sysimg.jl:38
   [26] top-level scope
      @ none:6
   [27] eval
      @ ./boot.jl:430 [inlined]
   [28] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:296
   [29] _start()
      @ Base ./client.jl:531

CI link for reference: Check more test fixes · SciML/SciMLBase.jl@74b2f50 · GitHub

I thought I’m adding the package right there :sweat_smile: , anyone know why it’s not being found?

2 
/home/runner/work/SciMLBase.jl/SciMLBase.jl/test/python/.CondaPkg/.pixi/envs/default/bin/python: No module named pip
/home/runner/work/SciMLBase.jl/SciMLBase.jl/test/python/.CondaPkg/.pixi/envs/default/bin/python: No module named pip

The pip-install lines are failing because pip itself is not installed in your test environment.

3

Your best bet is probably to do

CondaPkg.add_pip("diffeqpy")

in your test code instead (CondaPkg currently has no means to declare test-time dependencies).

4

Also pip install julia is not required I think. You are using JuliaCall instead, which is made available automatically by PythonCall.

5

That did it thanks Fix Python tests by ChrisRackauckas · Pull Request #960 · SciML/SciMLBase.jl · GitHub

1 Like