Creating a module with python dependencies

General Usage
,
1

Hi everyone!

I wrote some Julia code that I’m trying to convert into a module so I can interface it with some python code through JuliaCall. My module uses PythonCall to run some python code. I am currently using the lines:

UTILS_DIR = pwd()*"/src/UTILS/"
sys.path.append(UTILS_DIR)
ham = pyimport("ham_utils")
...

to include my python module “ham_utils.py”. This works fine when I run the code, but when I try to precompile my module with Pkg.precompile() I get the error

ModuleNotFoundError("No module named 'ham_utils'")

I am wondering how I can fix this, since I don’t know how to tell the precompilation that the UTILS folder should be added to the path. Thanks so much for your help!

2

Be happy that you got an error instead of a crash. pyimports must not be done by precompilation because that would store a stale memory pointer. See Guide · PythonCall & JuliaCall for the right solution.

1 Like
3

I’ve been looking at the PythonCall guide but I don’t really understand how to fix my error. I have tried instead of importing the full module just importing individual functions as stated in the guide:

UTILS_DIR = pwd()*"/src/UTILS/"
sys = pyimport("sys")
sys.path.append(UTILS_DIR)
bar = pyimport("ham_utils").bar
bar()

But I am still getting ModuleNotFoundError. I don’t really understand what is the problem with the pyimports or how to fix this.

4

If you follow my link you will see that you must do the pyimports in the module’s __init__() function. It’s possible that you will run into similar problems when you do that but then it at least has a chance to work.

5

Ok so now I tried:

const ham_ref = Ref{Py}()
function __init__()
	sys_ref[] = pyimport("sys")

	sys_ref[].path.append(UTILS_DIR)
	UTILS_DIR = pwd()*"/src/UTILS/"

	ham_ref[] = pyimport("ham_utils")
end

get_system = ham_ref[].get_system

And when I try to precompile the package I get the error access to undefined reference, it seems that the __init__ function never ran?

6

Yes, the idea of __init__() is that it should run when the module is loaded, not when it’s precompiled. I’m not sure what you want to do with get_system but fundamentally all symbols you retrieve from Python will be memory pointers, which are valid within the process but will be stale if you save them to disk and later reload them in a new Julia process, which is what happens when you precompile something. Then when you try to use that stale pointer you will typically get a crash because in the new process it points to some memory address you don’t have access to or to some random garbage. Thus all Python modules and functions must be loaded from __init__ so they are valid for the running process.

If you need to run some Python code during precompilation to compute some value, which is not tied to the memory layout of the current process, that’s fine. Then I would suggest to do that from a function, to reduce the risk that you happen to leave some stale pointers in the global scope that you might accidentally call from a new process.

By the way, the reason for the ModuleNotFoundError is possibly that you are using pwd() to navigate. It’s a better idea to use @__DIR__ to obtain the directory of the source code file as a starting point for building your path.

1 Like
7

Seems like the @__DIR__ fixed what needed to be fixed, thanks so much! From your answer I made a bit more sense of how Julia is interfacing to Python :slight_smile: .

Many thanks for your help!