Merge Python and Julia, i.e. an (unofficial) integration proposal, sort of like the Mojo language

Community
1

@cjdoris has an excellent package PythonCall.jl, and it means you can call to or from Python. I just don’t think that most Python users know of it (or the alternative pyjulia/PyCall.jl).

I know we have excellent integration already, and maybe most Julia users also know it, but the proposal is to draw Python users in.

Julia solved the “two language problem”, at least to a degree Python doesn’t, but those users mostly don’t know or care it’s an issue, and while this proposal seems to abandon it, I think I can justify it. It might seem heretical to suggest this, like giving up, but I’m thinking of the Python users mostly (and how it could selfishly benefit the Julia community to draw them in).

What I like about Julia is e.g. the awesome REPL, and the Python REPL is just inferior (also the default Ruby REPL, let alone Perl one and in any other mainstream language). [Does Python have some good opt-in REPL, similar to non-default Ruby REPL?]

Phase 1:

I’m thinking Python users, could start python, the same default python binary executable, as quickly as before, and they would not bear any interaction cost, until or if they they interop with Julia.

Same for Julia if you run the julia binary. It’s only that this unofficial self-extracting bundle.exe would install both languages/binaries. The combined download size is of course larger, roughly sum of both, but not a big issue, a one-time cost for all users (until we think of compiled Python and/or Julia programs).

Phase 1 seems like really easy to implement, possibly useful to (some) Julia users, but premature to announce to Python users until at least Phase 2:

Phase 2:

The Julia REPL is too good for Python users to pass up on. Only having it additionally for the python project seems doable, you could edit python code, it would be less of a drag, and you would only pay the slower Julia startup cost for an interactive section of Python.

This seems like an easy additional step, also what I would prefer, in 2b, a keystroke to go back and forth from Python to Julia. And from either language mode of course ; to enter the shell.

Phase 3:

We’ve had JuliaPro before, with a curated list of Julia packages. Here things get interesting. I think we should include Make.jl plotting package, and it would really interest interactive data science Python users. They could of course install their matplotlib as they’re used to, and use whatever plotting package from either language, but it seems warranted to include Makie.jl in the download. It might not need to be included into a default environment. I’m not sure we should have anything preinstalled, that way. At least we need to allow upgrading independently, unlike is done for Julia’s stdlibs. I’m open to a list of packages preinstalled in the package environment, that people could of course just delete from it.

This could go both ways, if there are some excellent Python packages Julia users should be aware of or might overlook, that could be added and usable from the Julia side.

@ChrisRackauckas’ SciML/DifferentialEquations.jl seems possibly very specific, too specific for most users(?), but I’m not opposed to including it in some way, at least diffeqpy (something I think is not too large to bloat the download, and I assume it autodownoads the rest on use):

Polyglot Userbase
While the majority of the tooling for SciML is built using the Julia programming language, SciML is committed to ensure that these methodologies can be used throughout the greater scientific community. Tools like diffeqpy and diffeqr bridge the DifferentialEquations.jl solvers to Python and R respectively

What I’m not proposing is also integrating the R language… I mean we could go all in, and RCall.jl already has an excellent R Mode, that I want emulated for Python. It’s just that Julia 1.10 is finally GPL-free, and R is GPL, and I want to keep Julia, and this project, GPL-free. Possibly the package RCall.jl only should be included though… RCall.jl doesn’t install R for you but it uses R that is preinstalled. That’s unlike PythonCall.jl that downloads Python for you or Julia if calling from the Julia side. Which would means no longer needed, to have an internet connection, after first install.

It seems obvious we would want to use the latest stable Python version (currently 3.11), but 3.12 will have a release candidate by the end of this month, and should be released 2023-10-02, so it’s tempting to use that also since:

The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623.

Python plans to adopt UTF-8 fully (internally) and that might be it (or just part of the plan?). Currently passing arrays (of Float64) between Julia and Python is no-copying, i.e. fast, but passing strings work, while I think it means on O(n) operation. With both using UTF-8 internally it should be O(1).

On the Julia side again latest version (currently 1.9), or maybe latest LTS (will likely be 1.10, now in alpha2?).

Mojo promises very fast “Python” but really it’s also two languages in one, the new one resampling Python syntax, new semantics. I’m really proposing similar, it’s just two in one with different syntax and different semantics.

It will e.g. be confusing to Python users to use 1-based by default, but most wouldn’t write their own Julia code, at least at first. You could add OffsetArays to the mix, installed by default if it help to have 0-based. I expect most Python users would be tempted by just calling state-of-the-art Julia code, and get them interested in Julia potentially.

I think we need to preempt Mojo getting larger mindshare than Julia, since we are ahead in many ways (not all) already. Mojo and Julia have many similarities, like speed.

Chris Lattner, Mojo and LLVM guy on Julia:

2 Likes
2

What I like about Julia is e.g. the awesome REPL, and the Python REPL is just inferior (also the default Ruby REPL, let alone Perl one and in any other mainstream language). [Does Python have some good opt-in REPL, similar to non-default Ruby REPL?]

IPython is closer in capabilities to the Julia REPL. It is nice that the Julia REPL is good by default, though.

5 Likes
3

First, I like the idea of heading Mojo off at the pass :stuck_out_tongue_winking_eye:

I have some questions:

What workflow are you imagining? I’m writing a project in Python and then how does this change it? Can I throw in a import Makie at the top of my file and the hybrid language interpreter will pull in Makie with PythonCall?

Or if I’m working on the REPL and define const str = "Hello World" on the Julia side, what happens to str on the Python side? Are the Julia values all isolated under the Main module? If so, how is this any different from what PythonCall already is?

As it stands, this proposal seems like it’s just PythonCall + a Python REPL mode. What am I missing?

4

What workflow are you imagining? […] Can I throw in a import Makie at the top of my file and the hybrid language interpreter will pull in Makie with PythonCall? […] this proposal seems like it’s just PythonCall + a Python REPL mode. What am I missing?

Yes, ideally (right now, without this project) you should be able to start Python, and do import Makie, or probably rather import GLMakie (and do same or similar to the example in the Makie docs) or e.g. import

But you can’t do that, and I’m not proposing that for the 1st or 2nd phase at least.

Plausibly, but not for phase 1, nor what I had in mind for phase 2.

I thin you mean you start for python_with_added Julia, i.e. you start with python syntax only (until you change to a Julia REPL mode with a keystroke at least). But hypothetically you would have name collisions in Python package ecosystem and Julia package ecosystem.

I believe all Python packages use lower-case names, or it’s at least the convention. Luckily all in Julia are start with an upper case letter (sometimes all uppercase is allowed).

There are exceptions on the Julia side e.g. cuDNN.jl, and while JLL packages are most often uppercase too but not always and they probably have more exceptions, e.g. libjulia_jll, yaml_cpp_jll and jlqml_jll, are available.

I don’t like much having a priority system, i.e. if package not found in Python ecosystem, then look into Julia’s. It seems bad if you were to later add that package… but possibly this could be ok for only packages starting with an upper-case letter.

Right now Python doesn’t use UTF-8 by default (it though sometimes uses UTF-8 as a redundant representation). Plausibly all variables (and functions and classes) could share the same namespace with Julia’s Main namespace. For string variables at least it’s technically less ideal until Python uses only UTF-8, as planned, like Julia does.

That was my first idea, and separate namespaces and REPL modes. I guess it’s non-ideal and not what Mojo does. But it could be done first that way as a proof-of-concept.

Julia is now run with julia but juliax and juliac is coming, and that new infrastructure could help.

1 Like
5

I apologize for the off topic question, but what are juliax and juliac?

1 Like
6

Ok, I guess I invited that question… See Keno’s PR, and read his words on it, not just my comment just after:

juliac - Compiler driver for the julia programming language
juliax - Experimental driver for the julia programming language

Before you get too excited, the PR is not yet merged (and neither helpful even if, right away), is seemingly related to an issue (with merged PR), on entry points, a very long discussion.

Note, my idea preceded juliax (and juliac), i.e. that infrastructure, that was my main point pointing out.

It’s possible to implement my idea unofficially without that PR merged. But with the PR merged, it doesn’t just add those two new CLI drivers, but also a way to add more e.g. julia_and_python_combined.

Most of what the PR does could be handled with an alternative sysimage. Not all, e.g. the distinction in Python with python (for .py) and pythonw (for .pyw) for Windows is not possible with just one CLI driver.

I very much welcome juliax, since it’s (my understanding at least, or could be) a playground for new ideas, Julia 2.0, changing/dropping stuff from Julia 1.x that can’t otherwise be done. Julia run the old way would still be compatible.

See also response to my 2.0, i.e. Julia1 package proposal, here, and my answer:

1 Like
7

I know we have @py_str in PyCall.jl and @pyexec in PythonCall.jl, but I’m surprised no one has tried a @python_cmd macro. Then you could write inline Python code like this:

bar(x) = x^2

macro python_cmd(ex)
    # to be implemented
end

python```
def foo(x):
    return x**2
```
7 Likes
8

I’ve used:

py"“”
def foo(x):
return x**2
“”"

so what’s the difference you mean? That’s py_str macro vs python_cmd, and I’m not sure what the difference is with _cmd and ```. I was well aware of, had to, Python being in a different namespace. Are you suggesting foo would be in Julia’s namespace? I would think _cmd, what you suggest, to shell out?

I would want a Python REPL mode (similar to the R REPL mode), even if not the same namespace, at least helpful for REPL work, to define variables if not functions.

2 Likes
9

The ticks are how we bracket code here in Discourse, so perhaps it would make sense to use them to bracket foreign code as well. Initially, the _cmd macros were not available, but now that they are they my provide a slightly more intuitive mechanism for including foreign code. In practice there is little difference between the string and cmd macros.

The other thing would be adapting VSCode to recognize foreign code and do proper syntax highlighting.

4 Likes
10

Let’s just keep saying that “Julia is just Python!” and maybe people will believe it. Seems to work for others. Maybe throw in a lot of :fire: emoji to really sell it.

28 Likes
11

I am not sure that fair. It basically is python, for better or worse.

Among the “worse” parts is: python without any foreseeable auto-differentiation packages to my knowledge… So not worth investing much effort in learning for much of the scientific computing community, in my mind.

12

Is it though? Literally every time I’ve tried simple but non-trivial Python code in Mojo, it fails. I just tried it again (in case it’s gotten better) with some example code that ChatGPT gave me:

def fibonacci(n):
    if n <= 0:
        return []
    elif n == 1:
        return [0]
    elif n == 2:
        return [0, 1]
    else:
        fib = [0, 1]
        for _ in range(2, n):
            fib.append(fib[-1] + fib[-2])
        return fib

n = 10
print(fibonacci(n))

Works fine in a Python REPL. The Mojo playground, on the other hand, gives me:

expression failed to parse (no further compiler diagnostics)

I tried inputting it all at once and statement by statement. Both give the same error. As far as I can tell, despite the marketing, Mojo is not only not “just Python”, it is not compatible with even pretty basic Python code. It is a completely separate language, which (unlike Julia) has similar syntax to Python and (like Julia) has excellent interop with Python.

Early on it seemed plausible to me that they actually intended to implement Python compatibility and just hadn’t yet had time to implement it. But at this point it seems clear that they’re not even trying to do that. Their official docs very carefully skirt the issue and imply Python compatibility but never outright claim it. Their marketing seems to have done an incredibly clever bait-and-switch and fooled an awful lot of people into believing that Mojo can run Python code. I guess very few people have tried actually doing that? The project is interesting and their MLIR interop is excellent—that’s the real secret sauce of the project—but this business of tricking so many people into believing that it’s Python-compatible as a language feels really icky.

So my comment above was really about this: Julia is “just Python” every bit as much as Mojo is, except that their syntax is more similar.

20 Likes
13

I think a lot of what you are saying is correct. And no doubt that in the current form it is nowhere close to being a superset of Python, but bugs galore are different from inherent design flaws. The whole thing is pre-alpha and roughly at the julia 0.0.0.1 level of usefulness.

I am not sure why they would go on a marketing rampage before things are operational for the basics of their vision. None of my business.

But I can see how they can become more and more compatible with python over time, but I don’t see any way that is true with auto-differentiation. The reason AD works in python so well is that the language is insanely dynamic - so not sure how they could even wrap existing AD frameworks. But maybe I just lack imagination here. Until then, it is of minimal interest . When I use python, the primary reason is for the AD.

1 Like
14

If they don’t maintain decent Python interop, it will be tough for people to adopt Mojo, which I think is unfortunate. Porting over libraries would be tricky and anyone that remember the 2->3 debacle would likely avoid Mojo.

15

I think that’s true at the moment, but don’t think it’s shaping up to be true in even just a couple months. For starters, Mojo is planning to support OOP features like multiple inheritance and protocols that are missing from Julia (which leads many/most people familiar with Python to avoid it).

Besides that, the tiny stylistic differences seem to matter more than I thought, and I think the Mojo team might be onto something there. The reason Python’s syntax is so distinctive/“clean” for introductory computing is because it was tested extensively as part of ABC, using focus groups and surveys of students and engineers to see what they preferred. I’ve had a handful of ML engineers used to Python try both Julia and Mojo, then come away preferring Mojo because it drops end keywords, avoids curly braces (preferring [] for parametric types, like Python), and allows for dot methods for piping/chaining.

The MLIR support claim feels kind of overblown, in that MLIR is designed to be easily retrofitted onto existing IR dialects, and is already the basis for most practical Python code in ML (because PyTorch, TF, and Jax all compile down to MLIR).

From what I’m seeing, the biggest actual semantic change is that they plan on making a much more rigid language than Julia or Python, with an (inferred) static type system, borrow checker, traits, and protocols. The ML engineers I talked to didn’t seem to have much trouble with these (actually, they seemed to enjoy learning about them) as long as there was some way to go back to dynamic typing with an opt-out (which Mojo seems to provide).

1 Like
16

(BTW, in this particular case, I think Mojo just doesn’t happen to support recursion yet.)

17

:joy:

2 Likes
18

That is picking the car based on the color of the hubcaps.

9 Likes
19

Agreed, that’s why I said it’s surprising; but it’s still what they told me.

Some of these are more reasonable than others. The lack of built-in piping/chaining in Julia is a real problem/missing feature; it definitely makes it harder to discover methods in Julia.

(For these users) end keywords are a mild annoyance, both for typing out and because they take up a lot of space (maybe 10% of lines in Julia are just end, so dropping end is like getting a 10% taller monitor). Scala did a similar experiment where they made curly braces optional and found programmers were slightly more productive after the change.

I can’t make an argument for Type[Float] being meaningfully different from Type{Float}, except that you might get confused while switching. I think Python just instills an irrational hatred for curly braces. Which, to be fair, I can empathize with that after having to learn Java.

20

Let’s just keep saying that “Julia is just Python!” and maybe people will believe it. Seems to work for others. Maybe throw in a lot of :fire: emoji to really sell it.

Where is Matt the snark slayer?