# Building a Transpiler from Python to Julia

**URL:** <https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349>\
**Category:** Offtopic\
**Tags:** announcement, python\
**Created:** [August 25, 2022, 5:09pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349 "2022-08-25T17:09:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 25, 2022, 5:09pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/1 "2022-08-25T17:09:09Z")

</div>

Greetings!  
This is my first-ever post on Julia’s discourse, so I hope it turns out ok!  
I have been developing a transpiler called PyJL for my Master Thesis over the last year, which translates Python source code to human-readable Julia source code. PyJL is part of the [Py2Many](https://github.com/py2many/py2many) transpiler, which transpiles Python to many C-like programming languages.

One of the last things I would like to do, is to have some feedback on the code that I am generating. I have recently created a small quiz containing Python code fragments and their corresponding translations in Julia generated by the transpiler. The intent is to assert the quality of the generated code. If you have any spare time, we would like for you to give it a try! You also have space in the quiz to leave your own suggestions, so leave them if you feel it is necessary (And yes, we will make an effort to consider all of them 🙂).  
Link to quiz (anonymous): [https://forms.gle/XZFcDBu8J8XafXAX9](https://forms.gle/XZFcDBu8J8XafXAX9)

Ps.: If you would like to contribute to Py2Many, feel free to check out the [repository](https://github.com/py2many/py2many). We are always open for new contributors 🙂.

---

<div class="post-metadata">

**Author:** ![ikirill](https://avatars.discourse-cdn.com/v4/letter/i/43a26b/32.png) [@ikirill](https://discourse.julialang.org/u/ikirill)\
**Post date:** [August 26, 2022, 7:35am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/2 "2022-08-26T07:35:13Z")

</div>

This is pretty cool.

If you integrated this with PyCall and managed to transpile the python parts of pytorch, would the result be capable of running pytorch code in Julia? That would be more “native” than calling a DL framework through PyCall yourself by wrapping every single call by hand.

---

<div class="post-metadata">

**Author:** ![SortofDamocles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sortofdamocles/32/15709_2.png) [@SortofDamocles](https://discourse.julialang.org/u/SortofDamocles)\
**Post date:** [August 26, 2022, 11:28pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/3 "2022-08-26T23:28:06Z")

</div>

I’m always happy to help a student project! I have a quick question, first, if you don’t mind. The instructions are: “How good do you consider the following translations to be?” My question is: “in what way?” Presumably, you’re not just interested in correctness because you could automate that. So what human touch are you looking for? For instance, simple if-else statements in julia are idiomatically done by short-circuiting `||` or `&&`. Since the translation doesn’t do this, I consider it imperfect. Do you?

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 27, 2022, 2:47pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/4 "2022-08-27T14:47:42Z")

</div>

Not sure if we are at that stage yet. One of the current issues we have is related to Python’s dependencies on external API’s and their dynamic features, leaving us with sometimes incomplete data-flow analysis.  
We are currently trying to transpile the [pywin32-ctypes](https://github.com/enthought/pywin32-ctypes) project, which is a reimplementation of pywin32 that uses Python’s ctypes FFI.  
We also mapped a subset of NumPy to native Julia, which gave us good results. PyTorch is on our list as well, but even if we are aiming for a subset, it would take a long time.

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 27, 2022, 3:04pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/5 "2022-08-27T15:04:53Z")

</div>

What I intended was for people to analyse the generated code to see if it respects the pragmatics of Julia. Our end-goal is to make the generated code as close to how a Julia programmer would write it. That leaves a little room for personal judgement, but that is to be expected, as one can write the same thing in multiple different ways.

I also agree with your last statement and I would also consider the translation to be imperfect in that scenario.

In general, all suggestions will help guide me in improving the transpiler.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [August 27, 2022, 6:57pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/6 "2022-08-27T18:57:36Z")

</div>

Wow, nice project!

I don’t feel comfortable “grading” the translations, but I can give my comments on the translations if you want, here or wherever.

One thing I noticed looking at the first few examples is that your transpiler assumes one-based indexing when accessing array members. Although this is correct for `Array`, I think it would make sense to do `a[begin + 1]` instead of `a[2]` and `a[(begin + 3):5]` instead of `a[4:5]`, for example. This would make the generated code easier to convert to use other AbstractArray types.

Another suggestion is to do `isone(x)` and `iszero(x)` instead of `x == 1` and `x == 0`, although you’ll be the judge whether the added complexity is worth it for you.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [August 28, 2022, 5:03am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/7 "2022-08-28T05:03:25Z")

</div>

> [@nsajko](#):
>
> `a[begin + 1]` instead of `a[2]`

I think this is fair considering they’re translating from Python code, which lacks an analog for `begin` and `end`. Of course, you’re right that the transpiler assumes that the `a` in Python is 0-based, specifically when incrementing indices by +1 for the 1-based `a` in Julia. I think this is a fair approach if `a` is proven to be a handful of Python built-in iterables (range, list) or a NumPy array, but something like `def get_index(A, i): return A[i]` cannot assume 0-based indexing always e.g. `A` might reference either `list`s or `dict`s. Translating to an array type with the same indices, perhaps a 0-based `OffsetArray`, would be safer generally; unfortunately, a lot of Julia code has yet to be updated to play nice with non-1-based arrays, and even it were, Julians would still have to convert their conventionally 1-based arrays to whatever the Python code expects, which are conventionally 0-based.

> [@mig](#):
>
> PyTorch is on our list as well

I read PyTorch is really a C++ implementation with a Python interface, so I’m guessing the intention is to transpile to a Julian interface that composes better with other Julia code? All the Python examples in your quiz make heavy use of mypy/PEP484 type hints that reflect Julian type annotations, but a lot of Python code lack them because hints don’t affect implementation as in Julia. This may not be a big problem for functions but translating classes to `::Any`-annotated types will introduce a lot of type instability. I assume you know PyTorch (more than me at least), so what are your thoughts on this issue when transpiling that? How about this issue when transpiling Python code that lack type hints in general?

---

<div class="post-metadata">

**Author:** ![ikirill](https://avatars.discourse-cdn.com/v4/letter/i/43a26b/32.png) [@ikirill](https://discourse.julialang.org/u/ikirill)\
**Post date:** [August 28, 2022, 11:22am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/8 "2022-08-28T11:22:06Z")

</div>

> [@Benny](#):
>
> All the Python examples in your quiz make heavy use of mypy/PEP484 type hints that reflect Julian type annotations, but a lot of Python code lack them because hints don’t affect implementation as in Julia.

PyTorch and JAX use type hints pretty actively (e.g., [https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py](https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py)), at least whenever the argument type is a primitive type, I believe. Sometimes they define type aliases like `Array = Any`, and then use `arg: Array` in code, which is not usable directly but also, in principle, could be converted to a proper type manually.

---

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [August 29, 2022, 5:55pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/9 "2022-08-29T17:55:58Z")

</div>

Unfortunately there is no option to skip a question. Not every respondent feels qualified to answer **all** question, thus many would probably stop in the middle, like I did.

As for different options (`begin` vs. `1`, `OffsetArrays` vs. index translation etc.). Some of them are not equivalent: They may both work correctly, but break on minimal code changes. Thus it could make sense to provide corresponding options to the user.

However I think you are starting a pretty big project, and the most important thing now would be to get transpiler working, i.e. producing correct results, on at least medium-sized projects. The translations you produce now are all readable enough IMO.

One small note though (as I didn’t finish the survey) - combining `for x = 0:n` and `for i in 0:n` in the same code (binomial\_coef) is not esthetically pleasing. Personally I prefer the second form.

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 30, 2022, 6:58pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/10 "2022-08-30T18:58:23Z")

</div>

Thank you for the suggestions @nsajko.  
Can you point to the examples where the transpiler is _assuming_ one-based indexing? It is possible that I optimized (that is, I told the transpiler to try to optimize) some of the examples and that is why you are seeing the use of one-based indexing. By default, the transpiler always uses zero-based indexing.

Using `a[begin + 1]`, and converting comparisons to `isone(x)` and `iszero(x)` are all good ideas for future implementations.

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 30, 2022, 7:01pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/11 "2022-08-30T19:01:00Z")

</div>

Thank you @Eben60. Regarding the use of `for x = 0:n` and `for i in 0:n`, this was my mistake when adding the code to the quiz. By default, the transpiler will use the syntax `for i in 0:n`. The `for x = 0:n` syntax is applied by the formatted we are using (see [JuliaFormatter](https://github.com/domluna/JuliaFormatter.jl)).

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [August 30, 2022, 7:15pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/12 "2022-08-30T19:15:17Z")

</div>

Hi @Benny. TBH, I haven’t looked into PyTorch and similar large libraries a lot, because our focus is to first guarantee translation correctness. I hope we will get into that in the future.

Regarding the use of type hints, we have integrated the [pytype](https://github.com/google/pytype) type-inference mechanism to try to decrease the amount of work required to annotate large code bases. Still, as pytype is a static inference mechanism, it also has its limitations. As an example, consider the following implementation of bubble sort written in Python:

```julia
def bubble_sort(seq):
    l = len(seq)
    for _ in range(l):
        for n in range(1, l):
            if seq[n] < seq[n - 1]:
                seq[n - 1], seq[n] = seq[n], seq[n - 1]
    return seq

```

The following is pytype’s output when analysing this function:

```julia
_T0 = TypeVar('_T0')

def bubble_sort(seq: _T0) -> _T0: ...

```

As you can see, this does not add any relevant type information to the code. This becomes a problem if you want to perform any operations with the function that require knowledge about its return type. In such situations, we still require programmers to manually annotate the Python source code.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [August 30, 2022, 11:57pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/13 "2022-08-30T23:57:39Z")

</div>

> [@mig](#):
>
> This becomes a problem if you want to perform any operations with the function that require knowledge about its return type.

As I mentioned before, this might not always be a problem for functions. Sure, `def bubble_sort` is not statically type-inferrable, but neither Python nor Julia are in general. Julia’s type inference only kicks in at function calls, using the types of the runtime arguments. The equivalent Julia version of `def bubble_sort` (with no type annotations) would generally have type-stable calls, though there is an assumption that `seq` is 0-based-indexable by integers.

That’s just 1 particular case, however. Python functions are often written without caring about the variables’ type stability (because like type hints, there’s no runtime benefit). An example is how often Python code prepares a summing loop with `sum_seq = 0` instead of how Julia does `sum_seq = zero(eltype(seq))`, but an _accurate_ translation would have that type instability.

My concern was more with classes’ ` __init__ ` being non-annotated. pytype’s static inference looks like it can help a little, but you’re probably right that those have to be manually annotated in general, or else the translated Julia `mutable struct` would be given `::Any` fields (introduces type instabilities) or given type parameters `::T` (which may require excessive compilation, or error when a field is intentionally reassigned an instance of a different type).

An aside: if possible, it would be good to show your examples in a README file (which can be put in the PyJL part of the Py2Many Github) or a linked blogpost, so readers can scroll through examples without being stopped by unskippable required answers. It’s not just a matter of encouraging readers to give feedback. I had actually completed your quiz before, but my answers are not saved for me after submission, so I was still hampered by multiple pages of unskippable answers when I tried to look at your classes examples again.

---

<div class="post-metadata">

**Author:** ![mig](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mig/32/36443_2.png) [@mig](https://discourse.julialang.org/u/mig)\
**Post date:** [September 14, 2022, 10:03pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/14 "2022-09-14T22:03:54Z")

</div>

I had some time to read through some of the quiz’s comments and made some improvements. I also noticed there were some bugs that I had accidentally overlooked. The most notable ones were:

- Python’s `int` has to be translated to `Int`, not `Int64`
- Some annotations in the Fasta benchmark were left untranslated. This was a copy-paste bug, which was already fixed.
- The Python expression `a = [0] * m` has to be translated to `fill(0, m)`, not `repeat([0], m)`

Furthermore, I wanted to make clear that the intent is to evaluate the pragmatics of the **Julia** source code, not Python. Therefore, one should evaluate if the code is readable to Julia programmers.

Lastly, as some people also requested access to the code samples in a more accessible format, I also decided to provide a [document](https://drive.google.com/file/d/1_OQreLo1XDSq28NLmfcv0v-krtbwnf6d/view?usp=sharing) containing all the samples. This should allow you to scroll and compare any translations.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [September 14, 2022, 10:18pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/15 "2022-09-14T22:18:24Z")

</div>

> [@mig](#):
>
> Python’s `int` has to be translated to `Int`, not `Int64`

Of course, if you care about correctness, you should translate it to `BigInt` (because Python integers never overflow).

(See also the Julia [FAQ on transpilers](https://docs.julialang.org/en/v1/manual/faq/#Why-don't-you-compile-Matlab/Python/R/%E2%80%A6-code-to-Julia?).)

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [September 15, 2022, 12:15am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/16 "2022-09-15T00:15:23Z")

</div>

> [@stevengj](#):
>
> Of course, if you care about correctness, you should translate it to `BigInt` (because Python integers never overflow).

Unless it’s in a numpy array or similar and then it will. 😬

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [September 15, 2022, 2:14am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/17 "2022-09-15T02:14:28Z")

</div>

That’s a different type, though. The type annotation would presumably be something like `numpy.int32`, not just `int`. Just guessing though, I don’t know anything about python type annotations.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [September 15, 2022, 6:14am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/18 "2022-09-15T06:14:29Z")

</div>

Python doesn’t have type annotations or parameters. Numpy arrays have an attribute (analogous to field in Julia) called `dtype` that specifies the size and structure of an element. You can (but shouldn’t) reassign that attribute to accomplish something like `reinterpret` (though you should use `.view(new_dtype)` instead). It is not part of Python’s type system, but a static analyzer could distinguish the eventually-`np.int64` instances in `np.array` calls from the `BigInt`-like `int` literals.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [September 15, 2022, 9:53am UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/19 "2022-09-15T09:53:45Z")

</div>

> [@Benny](#):
>
> Python doesn’t have type annotations or parameters.

Well, but Python has type hints, see [typing — Support for type hints — Python 3.12.0 documentation](https://docs.python.org/3/library/typing.html)

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [September 15, 2022, 9:41pm UTC](https://discourse.julialang.org/t/building-a-transpiler-from-python-to-julia/86349/20 "2022-09-15T21:41:46Z")

</div>

Yes, but those have different purpose than Julia’s annotations. Rather than being part of the type system for dispatches, assertions, or conversions, they’re an auxiliary system for static analysis that is not enforced during runtime. So they’re allowed and intended to deviate from the type system, especially to write type parameters. Numpy has an open issue [16544](https://github.com/numpy/numpy/issues/16544) drafting type parameters like `StaticArrays`’ for Numpy arrays, basically indicating the mutable attributes for the element type and shape. These parameters would deviate from Julia’s `Array`, but this transpiler is already taking some liberties with translated types e.g. `list`s are not always `Vector{Any}`, `int`s are not `BigInt`s.