[ANN] PythonCall and JuliaCall

[oschulz]

@cjdoris

See my earlier reply to marius311.

Oh, sorry, overlooked that - thanks!

[michikawa07]

Thank you for your grateful work.
Can I use syntax of @pyimport numpy as np instead of np = pyimport("numpy") ?

[stevengj]

@michikawa07

Can I use syntax of @pyimport numpy as np instead of np = pyimport("numpy")?

Note that this was deprecated some time ago in PyCall, and I wouldn’t recommend bringing it back in PythonCall. It doesn’t make the code any shorter or easier to write, and it obscures what is actually going on. The goal is to interoperate with Python code, not necessarily to mimic Python syntax in Julia.

Also, as a general principle, one should be cautious in programming languages/libraries about having multiple syntaxes for the exact same thing, since it can just add confusion and complexity for little benefit.

[joa-quim]

I’ve just to install juliacall.
At first it hanged forever at the step

[juliapkg] Locating Julia ^1.6
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json

then I decided to kill the cmd process and it suddenly started to install Julia

[juliapkg] WARNING: About to install Julia to C:\programs\miniconda3\julia_env\install.

But why? I already have Julia installed and don’t want another copy that quite likely will start its own multi GigBytes if I install packages on it.

[cjdoris]

@Seif_Shebl

However, I encountered the following error in Julia 1.8.0-DEV when trying the example above and Julia exits immediately:

julia> using PythonCall

julia> plt = pyimport("matplotlib.pyplot")
Python module: <module 'matplotlib.pyplot' from 'D:\\DeepBook\\.julia\\environments\\v1.8\\.CondaPkg\\env\\lib\\site-packages\\matplotlib\\pyplot.py'>

julia> plt.plot(randn(100))
qt.qpa.plugin: Could not find the Qt platform plugin "windows" in ""
This application failed to start because no Qt platform plugin could be initialize

Can you open an issue please? This should work.

[cjdoris]

@michikawa07

Can I use syntax of @pyimport numpy as np instead of np = pyimport("numpy") ?

You can do @py import numpy as np (note the space before import).

I’ll write about it more another time but @py is an experimental macro which maps Julia syntax to its Python equivalent - not just import but many other operations too. You can write big blocks of code with it. Despite @stevengj’s concerns above, which are perfectly correct, this macro can sometimes simplify syntax considerably. It can also make tight Python loops much faster by avoiding Julia’s GC.

[cjdoris]

@stevengj

So I’m quite happy to see people actively working on an alternative re-design, and hope that the existing PyCall and PyPlot are helpful in this process.

If I’m honest, I have partly put off writing this announcement because my intention was never to write “PyCall 2.0” - it started out as an experiment to see if I could do some things differently and just snowballed. I’m extremely pleased and relieved you have such a positive reaction.

Indeed PyCall is a long-established package and I can totally see where some of its quirks come from, historically. The above comparison was never intended as an attack on PyCall - quite the opposite, it’s a brilliant package and (not least) is responsible for some very handy syntax that PythonCall can take full advantage of without the baggage of backwards incompatibility.

I have thought about merging some of the ideas from PythonCall into PyCall - the world doesn’t really need two Python-Julia interop packages - but some of the changes would be so dramatic that, like you say, a totally new package would be no less disruptive.

[cjdoris]

@joa-quim

it suddenly started to install Julia

[juliapkg] WARNING: About to install Julia to C:\programs\miniconda3\julia_env\install.

But why ? I already have Julia installed and don’t want another copy that quite likely will start its own multi GigBytes if I install packages on it.

There should be more messages from JuliaPkg explaining why, namely it couldn’t find a compatible version of Julia to use. So either Julia isn’t in your PATH, or it isn’t new enough to work with JuliaPkg which requires Julia 1.6 or above.

Also, aside from the Julia installation itself, it won’t take up any more room on your disk because it will use your usual Julia depot.

[Seif_Shebl]

I think I’ll wait for someone else to reproduce before opening an issue. I have multiple installations of Python in different directories and some of them are added to the path and the same thing goes for Julia. I feel that this could confuse the proper installation of PythonCall, so I’ll investigate more and try on another computer before I open an issue. Many thanks.

Edit:
Well, I got it fixed now. The installation was somehow corrupted, I removed the package, then removed everything in the CondaPkg folder here: .julia\environments\v1.8\.CondaPkg and reinstalled PythonCall. When I issued plt = pyimport("matplotlib.pyplot"), the package complained about matplotlib module not found. I simply did pip install matplotlib in Windows terminal inside .julia\environments\v1.8\.CondaPkg\env\Scripts and everything worked like a charm. Thank you for the great effort and awesome package.

[joa-quim]

@cjdoris

There should be more messages from JuliaPkg explaining why, namely it couldn’t find a compatible version of Julia to use. So either Julia isn’t in your PATH, or it isn’t new enough to work with JuliaPkg which requires Julia 1.6 or above.

Nope, no messages and I have Julia1.8DEV in the path. Well, don’t know if it makes a difference but what I have in the path is C:\programs\julia-1.8 where a julia.bat takes charge of calling the julia.exe

@cjdoris

Also, aside from the Julia installation itself, it won’t take up any more room on your disk because it will use your usual Julia depot.

Good to hear that.

BTW, a better explanation/example on how to call Julia packages from Python would be nice. As is, sorry but I just gave up.

[sqr00t]

Hiya, I’m just getting back to Julia after a while.

• Built in dependency isolation is great with conda package management!
• Type inference for Python objects also another plus!

However:

• Could there be a more self-explanatory way for Tabular data conversion instead of having pytable and PyTable as separate methods?

[palday]

Is there a reason not have PyCall 2.0 simply be PythonCall and deprecate the PythonCall name? I haven’t thought about all the implications and whether it even really makes sense for any real-world use cases, but then packages that only use the common subset of features could have a compat entry allowing both “classic” PyCall as well as PythonCall.

[Palli]

Yes, some reasons. PyCall 2.0 will by definition be a breaking change, so the name might as well be PythonCall. It IS possible to use PythonCall with the current PyCall (it’s a non-default option), so that’s one argument for the new name.

At least there’s no way to use some version of a package, such as PyCall, directly or as a dependency, and also another version, through another dependency. I’ve previously thought it might be useful, even for minor versions, just because some dependency is holding back, but I think the right thing is to upgrade to a new (minor) version (and make, trivial usually,PR to packages, as I’ve done, to make it possible to use there latest versions for all packages).

Do any other language, or their package managers allow this?

A good (non-technical) reason is to give @cjdoris (full) credit for his own package, which is a rewrite, not largely (almost not at all?) built on PyCall. He deserves it. PyCall is also really good, and I’m not sure @stevengj would want it taken over (in this way, something similar has happened, I believe, at least once, with Graph.jl). I would advocate he links to PythonCall as an alternative.

[jlapeyre]

One difficulty is that it’s not just different features, but the semantics are quite different. And the scope, in terms of aspects of a project that are supported, is quite different. I don’t think just a few compat functions will be sufficient to make PythonCall a drop-in replacements. (I’m not sure if that is what you are suggesting.)

[Paul_Soderlind]

Just to clarify:

do I avoid getting a new python installation/download if I set (for instance), JULIA_PYTHONCALL_EXE="C:\\miniforge3\\python.exe"
before installing PythonCall?

[Impressium]

Can one use PythonCall from different threads?

[marius311]

I believe so, that’s how it worked for me. Fwiw, PyCall does have a dynamically switchable thing like this too (PYCALL_JL_RUNTIME_PYTHON) as long as its the same Python executable that PyCall was built with, just in a different virtualenv. I think PythonCall’s works switching between any Pythons though.

Edit: to be clear, you don’t have to set it before installing PythonCall, you can dynamically switch before any Julia session.

[cjdoris]

@palday

Is there a reason not have PyCall 2.0 simply be PythonCall and deprecate the PythonCall name?

@jlapeyre and @Palli have made very good points.

One reason to keep them separate is that you can only have one version of a package installed in a project. So if you have a package in your project only compatible with PyCall 1 then you can’t use PyCall 2. PythonCall is sufficiently different that many packages would probably never upgrade, or be very slow to, which kinda pins you down. As it is you can use PythonCall and PyCall together no problem.

Anyway it’s too soon to think about this - PythonCall definitely needs more battle testing first.

[cjdoris]

@Paul_Soderlind

do I avoid getting a new python installation/download if I set (for instance), JULIA_PYTHONCALL_EXE="C:\\miniforge3\\python.exe"
before installing PythonCall?

Yes, selecting an interpreter explicitly opts out of all the automatic dependency handling - nothing will be downloaded. This means installing any required packages is up to you.

[cjdoris]

@Impressium

Can one use PythonCall from different threads?

Sure if you like segfaults.

Out of the box no, but with manual GIL handling it might work. I’ve never tried.