TravisCI build with Python deps

Internals & Design
1

Hello,

not sure if it’s the right place for such a question. Feel free to say so if I should ask elsewhere!

In my package (ClimateTools.jl) I use some Python dependencies that are used mostly for mapping purpose. Right now, one of the dependency is somewhat tricky to install (basemap does not install on Python 3.6 for instance).

I’m wondering how I could set up TravisCI to

  1. create a virtual env, installing the proper versions of the packages and a specific python version
  2. in Julia, install PyCall, set-up ENV["PYTHON"]="virtualenv python"; Pkg.build("PyCall")

Step #2 is OK on my side, I already do it in my .travis file, but I can get the syntax right for the first step.

Any hint would be really appreciated.

Cheers!

2

If you put setups in .travis.yml, you’ll have trouble with CIBot when registering the package (see The current METADATA release process). Maybe put something like this in your deps/build.jl (untested):

using Pkg

if lowercase(get(ENV, "CI", "false")) == "true"
    let basepython = get(ENV, "PYTHON", "python")
        envpath = joinpath(@__DIR__, "env")
        run(`virtualenv --python=$basepython $envpath`)

        if Sys.iswindows()
            python = joinpath(envpath, "Scripts", "python.exe")
        else
            python = joinpath(envpath, "bin", "python")
        end
        run(`$python -m pip install basemap`)

        ENV["PYTHON"] = python
        Pkg.build("PyCall")
    end
end

It’s a bit ugly to build PyCall again in your build process, but you can’t avoid it if you want to change Python path.

2 Likes
3

Thanks!

I guess going the deps/build.jl approach is something that might be more robust.

I’ll build on your suggestion and come back with the results!

4

FYI I made a PR to add virtualenv support Support virtual environment by tkf · Pull Request #578 · JuliaPy/PyCall.jl · GitHub. I think it’d make setting up test-specific Python environment easier and reproducible. In particular, you can skip Pkg.build("PyCall").

2 Likes
5

Very interesting! I also think this is much more reproducible if we can set-up virtualenv. Will keep an eye open on this PR.

Still haven’t had time to test your solution.

Cheers!

6

For reference, here’s what I did. It works on TravisCI but not JuliaCI though. I guess it’s because the if lowercase(get(ENV, "CI", "false")) == "true" evaluate to false on JuliaCI?

using Pkg

if lowercase(get(ENV, "CI", "false")) == "true"

    let basepython = get(ENV, "PYTHON", "python2")
        envpath = joinpath(@__DIR__, "env")
        run(`virtualenv --python=$basepython $envpath`)

        if Sys.iswindows()
            python = joinpath(envpath, "Scripts", "python.exe")
        else
            python = joinpath(envpath, "bin", "python2")
        end

        run(`$python -m pip install numpy`)
        run(`$python -m pip install scipy`)
        run(`$python -m pip install matplotlib`)
        run(`$python -m pip install https://github.com/matplotlib/basemap/archive/v1.0.7rel.tar.gz`)
        run(`$python -m pip install git+https://github.com/matplotlib/cmocean`)
        run(`$python -m pip install netcdf4`)

        ENV["PYTHON"] = python
        Pkg.build("PyCall")
    end
end
1 Like
7

Assuming you are talking about the build here Tag ClimateTools.jl v0.10.0 by attobot · Pull Request #19702 · JuliaLang/METADATA.jl · GitHub, this is because JuliaCIBot does not have virtualenv program:

┌ Error: Error building `ClimateTools`: 
│ ERROR: LoadError: IOError: could not spawn `virtualenv --python=python2 /home/jrun/ClimateTools/.julia/packages/ClimateTools/KQv9Y/deps/env`: no such file or directory (ENOENT)

Maybe you can run pip install --user virtualenv first. Or alternatively just use pip in user mdoe, i.e., run(`$basepython -m pip --user install numpy`) etc.

1 Like
8

Thanks! Will try that asap. :slight_smile: