Workflow for converting Python scripts

New to Julia
1

Hi,
I have a rather vague question about your typical workflows to convert python scripts to Julia.

First off all, I’m amazed how nice and performant PyCall works and this has really made converting (and comparing) some of my scripts from Python to Julia quite easy.
My smaller projects are like 500-lines data-evaluation scripts that I have in Python, usually not organized in modules.

Somehow, however, I cannot find “the optimal” workflow to get these scripts converted to Julia.
Last I tried to have much python functionality organized in functions, rewrite the top-level python code in Julia, include and call the functions via pyimport. My idea was to get the Julia script running as quickly (and easily) as possible and then later to convert the python functions one by one.

However, I regluarly run into the situation, that calling a python function throws an exception, and I personally usually cannot read much information from the PyCall exceptions, like:

ERROR: LoadError: PyCall.PyError("\$(Expr(:escape, :(ccall(#= C:\\Users\\joach\\.julia\\packages\\PyCall\\ttONZ\\src\\pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))", PyObject(Ptr{PyCall.PyObject_struct} @0x000000005ca37a60), PyObject(Ptr{PyCall.PyObject_struct} @0x000000003f0ab4c0), PyObject(Ptr{PyCall.PyObject_struct} @0x0000000040196c08))

Usually, the easiest way to get around such hurdles is to rewrite the function in Julia (which is OK, but not the workflow I planned to pursue).

So, while my question is not about this (exemplary) exception, I want to ask

  • What is your workflow to convert Python scripts to Julia?
  • Are there any tools to debug Python-side errors in Julia, or some easy ways to switch between Julia and Python in such a way that one can find out what the problem on the python side is?
  • What would you do, if you encounter a PyCall Exception you cannot understand (given that the function works in Python => must likely be related to the arguments passed).

Looking forward to your input.

2

Whenever I need to explore Python code I turn to IPython.jl.

4 Likes
3

My recipe for converting from one language to another is quite simple, at least when the second language is able to fully wrap the first one, which is the case for Julia thanks to PyCall. This is what I usually do when I go from Python -> Julia:

  1. Create a script/workflow which you want to port to Julia, this can be something easy like calling a function, or more complex, like full analysis chain. Smaller steps are easier though.
  2. Write unit tests and/or high level tests in Julia to have an automated way of testing if the code does what it should. If you have a test suite in the Python project, you can start converting those first!
  3. Start to rip the code apart on the Julia side by expanding it and replacing the Python calls with Julia functions. You have the test suite on your side to make sure you have not changed the logic.
  4. Do not care much about how “Julian” your code is at this stage, just “make it work”.
  5. If there is still a Python call in the code: goto 3
  6. Remove using PyCall :wink:
  7. Now you have a full conversion, time to refactor the code and make it more Julian.

I know, it’s a quite rough plan, but this has worked very well for many projects I have converted in the past. Again, thanks to PyCall, the transition process is extremely smooth.

8 Likes
4

Thank you very much for this tip. This is roughly what I was looking for.
So… if I encounter a PyCall exception

ERROR: LoadError: PyCall.PyError("\$(Expr(:escape, :(ccall(#= C:\\Users\\joach\\.julia\\packages\\PyCall\\ttONZ\\src\\pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))", PyObject(Ptr{PyCall.PyObject_struct} @0x000000005ca32920), PyObject(Ptr{PyCall.PyObject_struct} @0x000000005bc5a5c8), PyObject(Ptr{PyCall.PyObject_struct} @0x0000000000000000))
  • I now store the function arguments in to Julia global variables (e.g., temp) right before the problematic PyCall
  • I open IPython in REPL, and import the same python functions I want to use in julia
  • I call the function in IPython with the globalized arguments through Main.temp.

This way, I get a (for me) more informative output that hints me to what I’ve done wrong.

In [24]: plot_histograms(Main.temp_var,[5,5])
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-24-9fa2719821a2> in <module>
----> 1 plot_histograms(Main.temp_var,[5,5])

<ipython-input-23-d6fd14765296> in plot_histograms(infos_on_current_stack, A5_landscape_in_inches)
     17     fig.canvas.set_window_title(title)
     18     plt.tight_layout()
---> 19     temp_filename = str(export_folder+'\\res_test - ' + stackname + ' - FWHM_distribution.png')
     20     plt.savefig(temp_filename, format='png', dpi=300)
     21     infos_on_current_stack["images_for_docx"].append(temp_filename)

NameError: name 'export_folder' is not defined

(usually some global variable dependencies that should of coarse have been avoided in the first place.)

So, functionality-wise, this is what I was searching for.
Is there maybe a more elegant way to acchieve such output, e.g. to configure PyCall to emit it?

5

Hi,
besides the systematic testing this is roughly what I had aimed for.
Thank you for your input.

1 Like