For context, I’m jotting down a list of Python packages for a project idea, and it will be beneficial to do some very simple number crunching without allocations. I considered the
out argument of NumPy and Numba, but the current limitations will make some things very painful to get working. AFAIK the program will allocate everything at the start and update them at regular timepoints given touchscreen positions, so I’m thinking I could write the number crunching part in Julia (it’s an LLVM-based JIT compiler like Numba anyway) and use PythonCall to share the array with the Python side.
Before I start experimenting, I want some tips from people who’ve done Python-Julia interop. My wants:
- Easily reproducible environment that manages dependencies from both languages
- be able to import the package into either Julia or Python and run the program.
- probably will end up as Kivy app, so if there’s any issues with doing that from Julia or PackageCompiler, a warning would be appreciated
Initially I had considered working with Conda for multi-language support and making this a Julia-in-Python project, but I read through this thread where several people opined that Conda is still primarily a Python package manager that tacked on support for some more languages, so Julia support was still impractical then. It’s still not clear to me if Julia-in-Python or Python-in-Julia is more practical or what sort of package manager setup is smoothest right now.
It’s still tentative because:
- I also want to consider the relative memory overhead of the Numba compiler vs the Julia process
- I plan to have a customizable GUI and use higher order functions in the number-crunching side to customize the processing, but it seems unwieldy to have extensible code in two languages. This might work itself out when I get good enough at Kivy and PythonCall to make something.
, but managing dependencies the right way and getting it started comes first, hence this post.
I did find this somewhat outdated stackoverflow post about profiling the memory cost of imports in Python, but I’m not certain if it covers the full overhead. If anybody has a similar thing for the Julia side that can measure the process’s overhead or Numba usage, let me know too.