Pyimport works from repl defined module but not in my package

General Usage
1

Hi,
I’m trying to use PyCall to call the hnswlib python bindings from julia. The following MWE with a module defined at the repl works as expected. And also a slightly expanded version of that module where the find_ann function actually does non-trivial work with hnswlib also works.

julia> module A
              using PyCall
              hnswlib = pyimport("hnswlib")
              function find_ann(X::AbstractArray{T,2};
                                metric = "l2") where T<:Real

                  n_elements,ndims = size(X)
                           
                  # Declaring index
                  p = hnswlib.Index(space=metric, dim=ndims)  # possible options are l2, cosine or ip
              end
       end
Main.A

julia> using .A

julia> X = randn(100,10);

julia> A.find_ann(X)
PyObject <HNSW-lib index>

So then I start to develop a small package that I want the find_ann function to be a part of. I’m getting an error I don’t understand when I try to define my find_ann function inside the package. Right now my package consists of three files, MyPackage.jl, nn.jl, and Utils.jl. Utils.jl currently just has one simple function that only calls stuff from base julia. MyPackage.jl is:

module MyPackage

using LinearAlgebra, SparseArrays, Statistics
using PyCall

include("Utils.jl")
include("nn.jl")

end

And then nn.jl is:

export find_ann
hnswlib = pyimport("hnswlib")

function find_ann(X::AbstractArray{T,2};
    metric = "l2") where T<:Real

    n_elements,ndims = size(X)

    # Declaring index
    p = hnswlib.Index(space=metric, dim=ndims)  # possible options are l2, cosine or ip
end

This goes into the standard package folder structure:

  • MyPackage/
    -Project.toml
    -src/
    -MyPackage.jl
    -nn.jl
    -Utils.jl

And I do ] dev /path/to/MyPackage/ to add the package to my julia environment.

The problem is that find_ann doesn’t work when it’s defined inside MyPackage, even though, as you can see, it’s literally been copy-pasted from the module I defined in the repl. When I try it I get:

julia> using MyPackage
[ Info: Precompiling MyPackage [UUID-omitted for Discourse]

julia> X = randn(100,10);

julia> find_ann(X)
ERROR: ArgumentError: ref of NULL PyObject
Stacktrace:
 [1] getproperty(::PyCall.PyObject, ::String) at /home/patrick/.julia/packages/PyCall/zqDXB/src/PyCall.jl:296
 [2] getproperty(::PyCall.PyObject, ::Symbol) at /home/patrick/.julia/packages/PyCall/zqDXB/src/PyCall.jl:306
 [3] find_ann(::Array{Float64,2}; metric::String) at /home/patrick/julia-dev/SSLClustering/src/nn.jl:10
 [4] find_ann(::Array{Float64,2}) at /home/patrick/julia-dev/SSLClustering/src/nn.jl:7
 [5] top-level scope at REPL[4]:1

julia>

Does anyone know why I get that ref of NULL PyObject error only when find_ann is defined in MyPackage and not in the repl defined module? Thanks!

P.S. This is on linux, julia 1.4.1, PyCall v1.91.4.

2

Could you verify that you don’t have multiple versions of python on your system, and the ones in the package’s environment is using the same python as the one in the default REPL environment?

1 Like
3

My guess is that this line may not survive pre-compilation:

hnswlib = pyimport("hnswlib")

Can you try to define an object and then initialize it inside __init__ function, like this:

const hnswlib = PyNULL()

function __init__()
    copy!(hnswlib, pyimport("hnswlib"))
end

Ref: https://github.com/JuliaPy/PyCall.jl#using-pycall-from-julia-modules

3 Likes
4

Thanks for the suggestions @jishnub and @tk3369! @tk3369, you correctly identified the problem. Thank you. Not having the pyimport call in the module’s __init__ function was the issue. Failure to read the docs on my part. Glad it was a simple fix.