I want Best Practices for Working With Python & Julia Together

[Olivia]

Hi everyone,

I am enjoying how fast & expressive it feels compared to some other languages I have used before. I still rely a lot on Python for certain libraries & workflows. I have been exploring ways to combine the two but I am not sure what the best practices are when it comes to performance, package management & keeping things clean when switching back & forth.

Is PyCall still the go-to approach for most people or are there newer, more efficient ways to integrate Python code into Julia projects?? Also I want to know how others organize their projects when both languages are involved — do you keep separate environments or mix them within the same workflow?

I came across a Ruby On Rails course that mentioned how people often bridge different tools in a single project & I want to know about how Julia folks think about this balance.

Want to hear your tips.

Thank you.

[adienes]

@Olivia

Is PyCall still the go-to approach for most people or are there newer, more efficient ways to integrate Python code into Julia projects??

without comment on some of the bigger picture questions posed here, I think specifically with respect to Python interop, PythonCall.jl (and its paired Python package juliacall to allow calling Julia from Python) are probably better choices now than PyCall.jl

[Palli]

Is PyCall still the go-to approach for most people or are there newer

I would say no, PythonCall.jl is now typo not the mainstream, and the future. PyCall is still used by some python package wrappers, so you might end up using both at the same time even without knowing it. See the docs on if such is still a problem.

Packages like:
https://github.com/SciML/diffeqpy

previously used PyCall.jl but switched to PythonCall.jl (on my recommendation).

You need to decide which is your main language to call from, if Python, you install the associated juliacall Python package and do something like:

from juliacall import Main as jl

That way you can e.g. use Django, or whatever Python uses for GUIs, even for SQL (Julia have GUI, maybe less developed, web and SQL capabilities too if you want Julia the main language) and call Julia for some stuff.

Note there similar for Ruby [on Rails]:

gem install jl4rb

I don’t know how good it is, or if in-process, as the other solutions for python, or if calling in both directions is then possible. I knew of this years ago, but rarely see Ruby discussed for interop with Julia if ever. It might still just work.

Re-writing a ruby side project into julia to learn

I have a few side projects which are running fine, but never let stability get in the way of learning a new language! I’ve been using julia for two weeks now and I figure an easy walk into learning more is to port the projects. In a few hours on Friday night, I ported the code and ran some performance comparisons between the current scripts in ruby and now in julia. This may be comparing apples to trains, and none of this is statistically valid. The Background [my desktop with 48 cores]⇋[giget…

One thing of note, is that Python dicts, are by now ordered (what I and them consider convenient), Julia’s built-in Dict isn’t ordered (to be faster or not rule out such algorithms). Ordered is available in OrderedCollections.jl, and I’ve tried to get it to be the new default in Julia…

Anything you build in pure Python will be way slower (NumPy, numba etc. compensate, but not fully).

@Olivia

I came across a Ruby On Rails course that mentioned how people often bridge different tools in a single project

What do you have in mind, can you even provide a link to it?

Welcome to the Julia community!

[ufechner7]

I still use PyCall because I find it easier and some packages I use rely on it. It installs one Python version for all of your packages per Julia version in .julia/conda.

If you use PyPlot for plotting, be aware that this can cause problems when you are using multi-threading in Julia. Workaround: Do the plotting in a separate process using DistributedNext.jl .

There was a version conflict between Julia and PyPlot recently, which can be solved by executing:

using CondaPkg
CondaPkg.add("libstdcxx", version="<14.0")

[Palli]

@ufechner7

I still use PyCall because I find it easier and some packages I use rely on it.

That’s your preference, and is ok. PythonCall is easier to set up(?), PyCall not too hard, but not automatic. The latter some package rely on it isn’t an argument to not use PythonCall (too)?

You mean it’s easier after installation? The code you write, or is it just about what you’re used to? I really want to point people to the right solution, think PythonCall is the future, didn’t want to go into all the pros and cons of each.

[Alex1]

Can you please explain shortly why PythonCall is better than PyCall?

[ufechner7]

One difference is that PythonCall maintains one Python environment per Julia project, PyCall one for all Julia projects of one major Julia version. What is better really depends on your specific needs.

[sob]

A bit of an aside, but how does DistributedNext differ from Distributed?

[ufechner7]

For me, the main reason to use DistributedNext is that it is faster:

julia> using DistributedNext
julia> @time addprocs(1)
  1.629440 seconds (2.79 M allocations: 140.291 MiB, 1.36% gc time, 75.61% compilation time)

vs

julia> using Distributed
julia> @time addprocs(1)
  3.044874 seconds (6.72 M allocations: 337.386 MiB, 3.28% gc time, 88.19% compilation time)

And if you plot in another process this time contributes to the first-time-to-plot.

[sob]

Appreciate the feedback, thank you

[stevengj]

@Alex1

Can you please explain shortly why PythonCall is better than PyCall?

See e.g. [ANN] PythonCall and JuliaCall - #24 by stevengj — PythonCall has the advantage of coming later, and was able to re-visit some of the design decisions in PyCall.

[Benny]

Don’t use Python-Julia interop much so I’m not very good at it, but I prefer how PythonCall handles, or rather lets us handle, types linking both sides. That might be more opinionated though, I just found more automatic conversions more of a burden than a feature.