RFC – ergonomic juliacall syntax for passing Python variables to Julia

General Usage
, , ,
1

I want to get feedback on an API for ergonomically passing Python variables to Julia code via juliacall. Here are a couple of ideas:

from juliacall import Main as jl

# proposal
jl.teval("const MY_CONST = $(x)", x=my_python_const)

a “template” eval, or

# proposal
with jl.let(x=my_python_const):
    jl.seval("const MY_CONST = x")

which is a juliacall “let.”

This is designed to address the following issue. Currently, passing Python objects to Julia via seval requires creating closure functions:

# already works
jl.seval("x -> @eval const MY_CONST = $x")(my_python_const)

While Python’s PEP 750 proposes Template Strings which could offer one solution:

# proposal
jl.seval(t"const MY_CONST = {my_python_const}")

We could implement something similar with a new teval method:

# proposal
jl.teval("const MY_CONST = {my_python_const}")

However, this has two key issues:

  1. The {} syntax conflicts with Julia type signatures (e.g., Vector{Float64}), which could cause subtle bugs. Even if t-strings are eventually merged to Python, we would still face this issue!
  2. teval wouldn’t have access to local variables to reference my_python_const

Now, here are the two proposed approaches. The first is similar to Python’s .format method, but with {} -> $() to avoid conflicts.

# proposal
jl.teval("const MY_CONST = $(x)", x=my_python_const)

The second approach uses a context manager for temporary variable binding:

# proposal
with jl.let(x=my_python_const):
    jl.seval("const MY_CONST = x")

Basically the seval would have access to a stack of Python objects pushed via juliacall.let, and put them into the Julia context via a Julia-evaluated let statement.

Thoughts?

(I wanted to get broader feedback so am cross-posting this PythonCall.jl/juliacall issue here)

2 Likes
2

Wouldn’t it be easier and more natural to pass variables as function arguments, rather than defining globals or evaluating strings? Why is the latter something you find yourself needing frequently?

Going in the other direction, PyCall has long provided a way to interpolate Julia objects into Python-code strings, but it’s something I’ve virtually never used. Evaluating strings is a terrible, last-resort way to do cross-language calls.

3

@stevengj That’s actually what I’m currently doing - passing variables as function arguments via a closure, as shown in my first example:

jl.seval("x -> @eval const MY_CONST = $x")(my_python_const)

The proposed APIs are just meant to provide a more ergonomic way to do this and other similar patterns, while maintaining proper variable scoping under the hood.

1 Like
4

This happens quite a bit in PySR - users frequently need to define Julia variables from Python to customize aspects of the symbolic regression search. It is very common for users to want to insert a single snippet of Julia code in their Python scripts. It comes up frequently enough that creating closures like this is a real problem, which is the entire reason I’m interested in a syntax like this.

Here are some recent examples of PySR’s users needing similar sorts of patterns for specific use-cases:

5

I think there’s a disconnect here, not sure. I think stevengj is talking about importing or executing files (like pyexec(read(...))) from another language and just using the interop wrappers afterward. There wouldn’t be any evaluation of inline strings of other languages at all.

There is evidently a demand for it, and I agree there are benefits for inline code strings from other languages:

  • Say we have code with a PJPJ structure (P for Python code, J for Julia code). Ideally we have a PP module and a JJ module to interop, but maybe we only had separate P and J scripts before and they weren’t designed with APIs. Sure, doing that work would be preferable, but pasting and interpolating is simpler and saves us time for other work.
  • While it’s often exaggerated, lower technical skill of a team using scripting languages can affect how code is written and shared; one file or notebook is easier to read and attach in emails. Performance and best practices are not the priority or even needed for their job.
  • Inline code strings can ironically stave off something worse in practice. A very common interop myth is that one can unconditionally optimize by moving source code into a “faster” language, so you’ll see forum questions about how to translate every single line in R code to RCall.jl code, completely unaware that Julia’s compiler cannot touch the underlying R code and the granular wrapping is adding significant overhead. If they’ll struggle to reimplement in Julia (many things lack neat equivalents) or write APIs for Julia interop, then interpolating into an @R_str is the best practice they can do.
6

@Benny Thanks. To clarify - my focus is on making it more ergonomic to pass Python variables into Julia when using juliacall. Currently, any time we want to use Python values in Julia code (from python), we need to create a separate closure:

from juliacall import Main as jl
import numpy as np

my_python_const = np.array([1.0, 2.0])

jl.seval("x -> @eval const MY_CONST = $x")(my_python_const)
loss = jl.seval("""
threshold -> begin
    (x, y) -> min(abs2(x - y), threshold)
end
""")(1.0)

This pattern comes up a lot in PySR where Python users need to customize all sorts of Julia-side behavior. I’m exploring if we can make this variable-passing pattern more ergonomic and more intuitive to the Python user. I think the above would look cleaner as follows:

with jl.let(x=my_python_const):
    jl.seval("const MY_CONST = x")

with jl.let(threshold=1.0):
    loss = jl.seval("(x, y) -> min(abs2(x - y), threshold)")

or

jl.teval("const MY_CONST = $(x)", x=my_python_const)

loss = jl.teval("(x, y) -> min(abs2(x - y), $(threshold))", threshold=1.0)
2 Likes
7

I’m not a user of PySR so don’t prioritize my opinion, but I’m not a fan of getting stuck in with blocks, and while I would glady use this:

I would probably never define constants across a language barrier like this:

Makes namespaces harder to read unless I have ALL the Julia code inlined to the Python file, in which case I wouldn’t be evaluating line by line like that.

Something to consider is that you’re doing this from the Python side, and $-interpolation is not a thing there. $-interpolation in foreign language code strings on the Julia side don’t treat $ like native string or expression interpolation either, they need custom behavior for the interop. No idea how you’d do that from the Python side, probably best to implement it in Julia anyway.

8

Actually maybe it’s best if we just extend seval to pass keyword arguments by itself? So that you could just use

jl.seval("const MY_CONST = x", x=my_python_const)

and have it work.

Potential implementation

and it would get parsed into the Julia code

(; x) -> @eval begin
    const MY_CONST = $x
end

and then called with x=my_python_const as a keyword argument.

The current handling for seval is here: PythonCall.jl/src/JlWrap/module.jl at a61c0223a60b803016e51a0dde39371222abf773 · JuliaPy/PythonCall.jl · GitHub

function pyjlmodule_seval(self::Module, expr::Py)
    Py(Base.eval(self, Meta.parseall(strip(pyconvert(String, expr)))))
end

Perhaps there’s a way to preprocess the expr and convert it into a closure if keyword arguments are passed, and then execute it.

This is precisely why I thought it would be a better option. See my comments about {} and t-strings/PEP 750 in the first post. Using {} is not possible until t-strings arrive, and even then, it would cause a lot of bugs by the conflicting usage in Julia type parameters.

9

Yeah that would be my issue with it too. f-strings already {}-interpolate, just eagerly with the available variables, and you have to double braces f"vec::Vector{{Int}}" to escape the interpolation, which is a Julia code pasting issue we’d rather avoid.

I took another look, and turns out Python already has Template strings, just from string import Template instead of the proposed t-string literal, and it does use $-interpolation (${} instead of $() like ours though), also escaped by doubling. It must be niche because I’ve never even heard of it, and PEP 292 says it’s inspired by the older %-interpolation in str. I’d probably lean toward % for more familiarity, but not sure if Python users would like that either. But hey if you decide to use $() (maybe it’s easier to handle in a Julia implementation), you can say Python has a precedent, it’s not just a Julia practice forced onto Python users.

Template strings — Python 3.13.1 documentation

1 Like
10

Cool. That string.Template looks a bit different from a “t-string” though? The main advantage of a t-string (Python 3.14+ if PEP 750 passes) is you could write:

jl.seval(t"x = {y}")

and it would pass the entire variable for y (i.e., without converting it to a string) along with the parsed string object to seval for further processing. Only if you call print on it would it convert y to a string. I guess it’s most similar to a LazyString in Julia.

From PEP 750:

Templates provide developers with access to the string and its interpolated values before they are combined. This brings native flexible string processing to the Python language and enables safety checks, web templating, domain-specific languages, and more.

Then in the juliacall side we would convert it to a Julia variable before finally passing it to the Base.eval

But yeah, sadly we couldn’t use it because {} conflicts too much with other syntax in Julia.

11

Indeed, string.Template only stores the raw uninterpolated str. I was mostly shocked they’re reusing the terminology, as niche as it was.

Not the variable y exactly, its name and its assigned object at the time. It’s not like closures capturing variables that react to reassignments. However, it does mean that you can extract those then-assigned objects OR evaluate the name for reassigned objects. In Julia, $-interpolation in Expr or custom string literals behave like the former.

No, LazyStrings are mutable, it’s how they can lazily interpolate for printing. t-string Templates are immutable, and they are inputs for processing to separate str. Being more similar to Expr serves your purpose of evaluating Julia code. Again, just the really annoying need to escape {{}} that justifies custom parsing of raw str.

12

What do you think about just extending seval to handle keyword arguments directly? Could be the simplest solution for Python users.

13

You mean like the threshold keyword argument here?

Sure I’d use it, it just delays interpolation to a function call, and Python users would be familiar with that through the older printf-style %-formatting built into str or the str.format function using {} (both escape by doubling). But I’d (and my hunch is most users) prefer directly interpolating from variables in the scope, and that can’t be delayed at all. The proposed t-strings immediately store objects even if they don’t instantiate the full str like f-string literals. However, you can’t just make a non-standard string literal in Python, let alone for custom interpolation, which is why interpolation of arguments is common.

14

My point is that this assumes that the Julia code is of the form of a “script”, whose parameters are defined in terms of global variables.

Ordinary best practice should be that your Julia code defines functions already, in which case you don’t need to create a separate closure with an @eval to define Julia globals from Python values. All nontrivial Julia code should normally be inside functions.

15

FWIW, Pluto/HypertextLiterals have a very convenient way to share Julia values with JavaScript: interpolation like

@htl """<script>
x = $myval
</script>"""

doesn’t do string transformation, but efficiently packs and sends Julia myval to JavaScript (I think through msgpack). It’s seamless while it works (for a lot of cases!), with one downside that debugging can be challenging when anything goes wrong.

Maybe something similar could be done for Python → Julia direction, worth looking for inspiration.

1 Like
16

The difficulty is rooted in Python lacking macros that can transform code within an arbitrary scope (proposed in PEP 638). The similar-looking decorators are actually higher order functions that take arguments. There’s no customizable nonstandard string literals, it’s all builtin.

17

The closure pattern with @eval has a helpful purpose here - it lets users automatically use Python objects directly in Julia functions, like in this covariance matrix example where they just reference a numpy array. While I agree that pure Julia code should generally avoid this pattern, Python users seem to prefer to keep as much of their code in Python as possible – considering the popularity of Cython, Triton compiler, CuPy, etc. Keeping the barriers as low as possible for the Python community is a priority of PySR.

That said, we actually find that as users get more advanced, many graduate to using Julia and SymbolicRegression.jl directly where they can follow these better patterns! PySR’s more permissive style is really about providing an accessible entry point for Python users. I view this as a nice aspect of juliacall in general. And syntaxes like discussed in this thread could strengthen it even more.

@aplavin Thanks for the pointer! Will take a look. The msgpack approach is particularly intriguing.

18

It’s a real pity they don’t have this. It looks like it was proposed but got rejected as part of PEP 750:

Screenshot 2024-12-30 at 14.32.24
Screenshot 2024-12-30 at 14.32.241026×865 89.3 KB

^I feel like their reasons for rejection don’t match, at all, my experience working with _str macros in Julia which I have really liked. Maybe the proposed design was presented with a particular flaw attached to it, and that aspect took down the whole idea with it. Bummer!

19

The Julia code is defining a function in that example, so why not just make INV_COV_MATRIX a parameter of your Julia function? Using globals to pass data to functions is an anti-pattern.

(Even if you then want to pass the function to another function that doesn’t expect the extra parameter, that’s what closures are for, e.g. you pass lambda tree, dataset, options: custom_loss(tree, dataset, options, inv_cov).)

20

@stevengj The loss function interface in PySR is fixed - it must take (tree, dataset, options) as arguments since it’s called internally by the search algorithm. A scoped closure approach would look like:

jl.seval("""
inv_covar -> begin
    function my_loss(tree, dataset, options)
        ...
    end
end""")(inv_cov)

This is effectively equivalent to the current pattern - we’ve just moved the closure around. The core question here isn’t about code organization or global-vs-scoped, but rather about making Python->Julia variable passing more ergonomic.