Is PyJulia being maintained?

I think it’s just another option. It’s one I now prefer, but still not a great look to have a long standing and widely used package gathering dust.

1 Like

Python’s package ecosystem is just much bigger, so Julia users will resort to PyCall far more often than Python users resort to PyJulia. Even when the choice of libraries between the 2 languages are more even, people might base their code more in Julia to benefit from the optimizing compiler. The symmetric PythonCall-JuliaCall design will make it far harder for uneven demand to cause uneven development.

1 Like

seems to be the ultimate “gateway drug” for Python communities, to inject some Julia speed into their software without having to disrupt the frontend API

Yes.

Of course in an open-source project something like “by the Julia team” is rather nebulous. PyJulia is not getting much attention at the moment. But, there are a lot of very important improvements to be made to Julia that are not being worked on because of other very important improvements.

PythonCall is under active development, but is a one-person effort. Maybe the biggest advantage of PythonCall is that it is much easier to deploy in a robust way than PyCall/PyJulia. I wrote julia_project to help python packages manage a Julia dependency. A very large part of it is trying to make PyJulia deployment easier and more robust.

It would be great if there were a community of developers of Julia/Python projects with design guided by interactions with users.

I’ve mentioned elsewhere that the pyO3 project for python bindings in rust is far larger and more active than the corresponding Julia projects. On the other hand, they are following the very long precedent of extending scripting languages with compiled modules.

2 Likes

This is a good question. Let me bring in the human element here. The person who has been maintaining PyJulia, Takafumi, is still very active in the Julia community, but he’s just been doing other things, namely reworking many of the multithreading constructs. Aqua, something that has arguably become one of the most important elements of the Julia package ecosystem given how it is an auto-correctness test for many elements, is one of his projects.

Extended atomics is another.

With his main projects being GitHub - JuliaFolds/FLoops.jl: Fast sequential, threaded, and distributed for-loops for Julia—fold for humans™ and GitHub - JuliaFolds/Transducers.jl: Efficient transducers for Julia for new parallelism interfaces. He’s been an active participant in solving the issue so threads could be added at runtime. So with him being one of the people in the “Julia Lab’s Parallel Computing Compilers Squad” (Valentin (CUDA.jl, KernelAbstractions.jl, Enzyme.jl), Taka, Julian (AMDGPU.jl, Dagger.jl), etc.), PyJulia just hasn’t been in his focus. You can argue that it’s good to have better Python bridges etc. but I think if you look a bit more holistically at this aspect, we’re making major strides into new and improved multithreading constructs and he’s one of the architects. Any activity in PyJulia is somewhat pulling him out of what his real strength is.

In that sense, the Python → Julia bridges really are in need of a maintainer shake up. That may be someone else picking up the reigns of PyJulia, or having a new design (PythonCall.jl) disrupt it. I’m not sure. I think the packaging story could be improved with integrating PackageCompiler and StaticCompiler (i.e. julia_project) and the bindings need to get to a level where something like ModelingToolkit.jl could work on the Python side (currently there’s not enough of the dispatch carried over for the arithmetic operations). PythonCall might be a better basis on which those issues could be solved, but that is a story for the future to find out. For now, pyjulia is kept alive but without a maintainer shift I don’t see it changing too much, which is both a good and bad thing.

8 Likes

Oh this is a bit of a tangent (mods feel free to throw this into a different thread :sweat_smile:), but you might have a useful case for measuring something so I wanted to document it. Pypi measures all downloads, while the Julia package server filters CI from the downloads tracking. So I’ve been curious for awhile if there’s any natural experiments to do a calculation of how much including CI in download counts inflates the difference. This seems like a nice natural experiment because every download of PySR should include a download of SymbolicRegression.jl, but it also has 0 dependents, so most of the downloads should be either direct users or users from PySR. And since PySR’s downloads of SymbolicRegression.jl hit the package server since it’s using the Pkg.add, this number would exclude CI while the other would not. The most conservative estimate for the small non-CI ratio runs would then be to assume they are all from PySR. But interestingly, the current number of downloads sits at 1552.

https://shields.io/endpoint?url=https://pkgs.genieframework.com/api/v1/badge/SymbolicRegression

135,000 / 1552 ~ 87 CI downloads per unique user download. That seems higher than I thought it would be? I assumed Pypi stats would be inflated by ~10x due to including CI, but this seems to suggest an extra order of magnitude. @staticfloat did I miss something in this back of the envelope math?

(Again mods, I don’t mean to hijack this thread so feel free to start this on a new thread if it gets replies, I just found this interesting because I just realized it’s a nice natural experiment while writing the first reply :sweat_smile:)

2 Likes

Is that yet another example of competing efforts in the Julia ecosystem where some of the authors leave the Julia community and their packages die? Can someone share the history of PyJulia and PythonCall.jl? What has inhibited the authors from collaborating into a single repository? Is their approach too different?

I became obsessed with this issue recently: How to know if a package is good?

You can read the announcement here, where some of the differences were discussed.

3 Likes

Thank you @albheim. From a quick scan of the first post and announcement, it seems that a few issues were fixed and more conversions between Julia and Python types were added. I wish I had more time to dive deeper into the differences, but I am guessing that this is another case where features could have been added to PyJulia instead of forking a whole new project?

Thanks everyone for the helpful replies. I will consider switching to JuliaCall. It looks like it doesn’t currently support M-series macs so I can’t test it out just yet: Juliacall on Apple M1 · Issue #175 · cjdoris/PythonCall.jl · GitHub, but once it is supported I will try. Thanks for linking the original thread @albheim. Will also check out julia_project, thanks @jlapeyre.

@ChrisRackauckas indeed I am immensely appreciative of Takafumi’s work. This thread is more a practical question of whether it still makes sense to use PyJulia in Python+Julia packages longterm, or whether it will introduce future instability due to low maintenance.

Regarding the download statistics, I am not sure. Could it be because the python version downloads the GitHub version of SymbolicRegression.jl, not the one in the registry? As far as I know, the numbers are unrelated to CI, since the CI uses GitHub instead of PyPI. Note that pepy counts updates too, so that may make up some of the difference. Even the conda version has 2900, ~double the Julia count, so I’m really not sure why the difference is so large. (But yes this is probably worth a separate thread)

1 Like

I thought the story detailed by Chris said the exact opposite.

2 Likes

No, this is not the case in any sense. S.G. Johnson and Takafumi are still super productive, but spread thin. PythonCall is a fresh approach to an old problem. The author would not have been happy (this is a big point, you can’t force people to be happy doing what you think they should) and would not have acheived their goals, tinkering at the edges, or even a bit toward the center, of PyCall. In fact there are design choices that are very different in the two packages, and the only way to test them is to let people use them.

Again, if this gets replies, it would be better for the mods to split this.

4 Likes

Thank you all for clarifying the history. I am just sad to hear that we are lacking maintainers again.

1 Like

Me too. At my company the state of Python/Julia interop is holding back adoption of Julia. This could mean anywhere from a few to many people employed to code in Julia in a high profile project. I really feel like complaining or blaming someone. But, that doesn’t make sense. People work on what they think is most important and interesting (or are paid to do). But, I do think it’s good to make people aware of the situation.

I really understand your frustration at lack of coordinated effort on projects. Chris was correct about the effect of single personalities. But, there are also many partial counter-examples, with big second order contributions and a lively user community that drive development.

4 Likes

Good points @jlapeyre.

To the Julia team: The return-on-investment for well-maintained Python and C++ interoperability is colossal. I can’t emphasize this enough. A couple dedicated maintainers for PyJulia now could bring contributions from hundreds of external engineers for other packages in the future.

When I was working at a FAANG company I chatted with someone researching potential languages to use in the future. I mentioned Julia as an option for high-performance compute, and most of their questions to me were about how well it could interface with existing languages in their stack (particularly Python and C++). Interoperability is absolutely essential to adoption by existing industries with mature software stacks!

9 Likes

I feel like this is very much the wrong attitude. Someone found some motivation to make & maintain an open source package (PythonJulia). That’s good! You often say things like “but shouldn’t they have made it as a PR to PyJulia?” Well maybe- but are those really equal options? What if they didn’t have the motivation to do that? What about the current users of PyJulia who might not want breaking changes? What if the PyJulia maintainers don’t have the bandwidth to review a giant rewrite PR and aren’t comfortable merging a PR without review?

I don’t think it’s fair or productive to dictate how others should spend their OSS time. I think if you accept different folks have different motivations and some work is easier or more motivating than other work then you will be able to better work with what exists instead of just wishing it were different. Perhaps for you, consolidating packages is motivating work. That’s great! But it won’t be for everyone, and yet we can still benefit from other forms of contributions.

20 Likes

I don’t know if I agree with you. The duplication of efforts that we see is a real issue in Julia and I am just pointing to it even if it hurts. OSS is about collaboration and for some reason we are failing at that in many cases. I am not saying that this is the case of PyJulia and PythonCall.jl, I would need more time to investigate it and I don’t have this time.

1 Like

I just don’t think the options are what you think they are. If someone isn’t motivated to do something, it’s just not really an option. Unless they are your employees or students, you kinda just need to accept what they are willing to give, even if it’s not the most “efficient” thing from your perspective.

Not all time is equal; time doing something I like that I want to do, I will often give for free. It will make me feel excited. Time spent doing something I don’t want to do? I will feel drained. If I have the choice, I probably just won’t do it. And I think that’s true for a lot of folks.

5 Likes

That is the key. If someone isn’t motivated to collaborate on an existing project, there is a reason, and as a community we can investigate this reason and try to improve the status quo.

We can’t simply close our eyes to a serious issue like this one where we have PyJulia today without active maintainers. This is super bad.

Anyways, I don’t want to deviate too much from the original question the OP asked. I think he has a better clue now of which package is more likely to get maintenance moving forward.

2 Likes

I want to make the point that I don’t think PyJulia should be considered a toy project that people work on only if they are interested in it in their spare time. Many other Julia libraries are like that and that’s fine, but my opinion is that PyJulia (or JuliaCall if it is chosen instead) should be considered an essential part of Julia infrastructure, having a major role in Julia’s future wider adoption by industry, and should receive commensurate dedicated support. Does this seem reasonable?

7 Likes

It’s still not clear to me what’s the point. People are entitled to be interested in taking care of whatever projects they feel like, unless they’re employed to maintain a specific package, which is not the case here.

3 Likes