PyJulia is an awesome interface to Julia from Python and seems to be the ultimate “gateway drug” for Python communities, to inject some Julia speed into their software without having to disrupt the frontend API. It’s apparently used by over 280 repositories according to GitHub, and I’ve personally found it very useful in my own projects, e.g., PySR’s Julia interface uses PyJulia, and has been downloaded 135,000 times.
However, it seems like PyJulia is not being actively maintained. The last commit was on January 7th, and there look to be many pull requests for essential features which have received zero comments (excluding bots). There are also many issues which have zero comments. This seems not optimal for such an important package, especially in expanding the use of Julia to other communities.
What I wish to learn with this posting is the following: is PyJulia still being maintained by the Julia team? Or, is there a different strategy which is now recommended for interfacing Python and Julia (that I am unaware of)? Or, is interfacing these languages not a development priority at the moment?
Thank you for all your work on this so far.
Best,
Miles
From what I’ve read, it’s an independent effort still in its v0 stage (though to be fair, PyJulia is still v0 too), and it can be used alongside PyCall. Big perk over the PyCall/PyJulia system is that PythonCall and JuliaCall share implementation and have symmetric interfaces, so those are fundamentally developed together.
Python’s package ecosystem is just much bigger, so Julia users will resort to PyCall far more often than Python users resort to PyJulia. Even when the choice of libraries between the 2 languages are more even, people might base their code more in Julia to benefit from the optimizing compiler. The symmetric PythonCall-JuliaCall design will make it far harder for uneven demand to cause uneven development.
seems to be the ultimate “gateway drug” for Python communities, to inject some Julia speed into their software without having to disrupt the frontend API
Yes.
Of course in an open-source project something like “by the Julia team” is rather nebulous. PyJulia is not getting much attention at the moment. But, there are a lot of very important improvements to be made to Julia that are not being worked on because of other very important improvements.
PythonCall is under active development, but is a one-person effort. Maybe the biggest advantage of PythonCall is that it is much easier to deploy in a robust way than PyCall/PyJulia. I wrote julia_project to help python packages manage a Julia dependency. A very large part of it is trying to make PyJulia deployment easier and more robust.
It would be great if there were a community of developers of Julia/Python projects with design guided by interactions with users.
I’ve mentioned elsewhere that the pyO3 project for python bindings in rust is far larger and more active than the corresponding Julia projects. On the other hand, they are following the very long precedent of extending scripting languages with compiled modules.
This is a good question. Let me bring in the human element here. The person who has been maintaining PyJulia, Takafumi, is still very active in the Julia community, but he’s just been doing other things, namely reworking many of the multithreading constructs. Aqua, something that has arguably become one of the most important elements of the Julia package ecosystem given how it is an auto-correctness test for many elements, is one of his projects.
In that sense, the Python → Julia bridges really are in need of a maintainer shake up. That may be someone else picking up the reigns of PyJulia, or having a new design (PythonCall.jl) disrupt it. I’m not sure. I think the packaging story could be improved with integrating PackageCompiler and StaticCompiler (i.e. julia_project) and the bindings need to get to a level where something like ModelingToolkit.jl could work on the Python side (currently there’s not enough of the dispatch carried over for the arithmetic operations). PythonCall might be a better basis on which those issues could be solved, but that is a story for the future to find out. For now, pyjulia is kept alive but without a maintainer shift I don’t see it changing too much, which is both a good and bad thing.
Oh this is a bit of a tangent (mods feel free to throw this into a different thread ), but you might have a useful case for measuring something so I wanted to document it. Pypi measures all downloads, while the Julia package server filters CI from the downloads tracking. So I’ve been curious for awhile if there’s any natural experiments to do a calculation of how much including CI in download counts inflates the difference. This seems like a nice natural experiment because every download of PySR should include a download of SymbolicRegression.jl, but it also has 0 dependents, so most of the downloads should be either direct users or users from PySR. And since PySR’s downloads of SymbolicRegression.jl hit the package server since it’s using the Pkg.add, this number would exclude CI while the other would not. The most conservative estimate for the small non-CI ratio runs would then be to assume they are all from PySR. But interestingly, the current number of downloads sits at 1552.
135,000 / 1552 ~ 87 CI downloads per unique user download. That seems higher than I thought it would be? I assumed Pypi stats would be inflated by ~10x due to including CI, but this seems to suggest an extra order of magnitude. @staticfloat did I miss something in this back of the envelope math?
(Again mods, I don’t mean to hijack this thread so feel free to start this on a new thread if it gets replies, I just found this interesting because I just realized it’s a nice natural experiment while writing the first reply )
Is that yet another example of competing efforts in the Julia ecosystem where some of the authors leave the Julia community and their packages die? Can someone share the history of PyJulia and PythonCall.jl? What has inhibited the authors from collaborating into a single repository? Is their approach too different?
Thank you @albheim. From a quick scan of the first post and announcement, it seems that a few issues were fixed and more conversions between Julia and Python types were added. I wish I had more time to dive deeper into the differences, but I am guessing that this is another case where features could have been added to PyJulia instead of forking a whole new project?
Thanks everyone for the helpful replies. I will consider switching to JuliaCall. It looks like it doesn’t currently support M-series macs so I can’t test it out just yet: Juliacall on Apple M1 · Issue #175 · cjdoris/PythonCall.jl · GitHub, but once it is supported I will try. Thanks for linking the original thread @albheim. Will also check out julia_project, thanks @jlapeyre.
@ChrisRackauckas indeed I am immensely appreciative of Takafumi’s work. This thread is more a practical question of whether it still makes sense to use PyJulia in Python+Julia packages longterm, or whether it will introduce future instability due to low maintenance.
Regarding the download statistics, I am not sure. Could it be because the python version downloads the GitHub version of SymbolicRegression.jl, not the one in the registry? As far as I know, the numbers are unrelated to CI, since the CI uses GitHub instead of PyPI. Note that pepy counts updates too, so that may make up some of the difference. Even the conda version has 2900, ~double the Julia count, so I’m really not sure why the difference is so large. (But yes this is probably worth a separate thread)
No, this is not the case in any sense. S.G. Johnson and Takafumi are still super productive, but spread thin. PythonCall is a fresh approach to an old problem. The author would not have been happy (this is a big point, you can’t force people to be happy doing what you think they should) and would not have acheived their goals, tinkering at the edges, or even a bit toward the center, of PyCall. In fact there are design choices that are very different in the two packages, and the only way to test them is to let people use them.
Again, if this gets replies, it would be better for the mods to split this.
Me too. At my company the state of Python/Julia interop is holding back adoption of Julia. This could mean anywhere from a few to many people employed to code in Julia in a high profile project. I really feel like complaining or blaming someone. But, that doesn’t make sense. People work on what they think is most important and interesting (or are paid to do). But, I do think it’s good to make people aware of the situation.
I really understand your frustration at lack of coordinated effort on projects. Chris was correct about the effect of single personalities. But, there are also many partial counter-examples, with big second order contributions and a lively user community that drive development.
To the Julia team: The return-on-investment for well-maintained Python and C++ interoperability is colossal. I can’t emphasize this enough. A couple dedicated maintainers for PyJulia now could bring contributions from hundreds of external engineers for other packages in the future.
When I was working at a FAANG company I chatted with someone researching potential languages to use in the future. I mentioned Julia as an option for high-performance compute, and most of their questions to me were about how well it could interface with existing languages in their stack (particularly Python and C++). Interoperability is absolutely essential to adoption by existing industries with mature software stacks!
I feel like this is very much the wrong attitude. Someone found some motivation to make & maintain an open source package (PythonJulia). That’s good! You often say things like “but shouldn’t they have made it as a PR to PyJulia?” Well maybe- but are those really equal options? What if they didn’t have the motivation to do that? What about the current users of PyJulia who might not want breaking changes? What if the PyJulia maintainers don’t have the bandwidth to review a giant rewrite PR and aren’t comfortable merging a PR without review?
I don’t think it’s fair or productive to dictate how others should spend their OSS time. I think if you accept different folks have different motivations and some work is easier or more motivating than other work then you will be able to better work with what exists instead of just wishing it were different. Perhaps for you, consolidating packages is motivating work. That’s great! But it won’t be for everyone, and yet we can still benefit from other forms of contributions.
I don’t know if I agree with you. The duplication of efforts that we see is a real issue in Julia and I am just pointing to it even if it hurts. OSS is about collaboration and for some reason we are failing at that in many cases. I am not saying that this is the case of PyJulia and PythonCall.jl, I would need more time to investigate it and I don’t have this time.
I just don’t think the options are what you think they are. If someone isn’t motivated to do something, it’s just not really an option. Unless they are your employees or students, you kinda just need to accept what they are willing to give, even if it’s not the most “efficient” thing from your perspective.
Not all time is equal; time doing something I like that I want to do, I will often give for free. It will make me feel excited. Time spent doing something I don’t want to do? I will feel drained. If I have the choice, I probably just won’t do it. And I think that’s true for a lot of folks.
), but you might have a useful case for measuring something so I wanted to document it. Pypi measures all downloads, while the Julia package server filters CI from the downloads tracking. So I’ve been curious for awhile if there’s any natural experiments to do a calculation of how much including CI in download counts inflates the difference. This seems like a nice natural experiment because every download of PySR should include a download of SymbolicRegression.jl, but it also has