# Pyarrow conversion with PythonCall

**URL:** <https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485>\
**Category:** General Usage\
**Tags:** dataframes, pythoncall, arrow\
**Created:** [May 27, 2023, 2:58pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485 "2023-05-27T14:58:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 2:58pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/1 "2023-05-27T14:58:04Z")

</div>

It would appear to me that PythonCall.jl does not convert pyarrow tables to DataFrames. I get the error

```julia
ERROR: cannot convert this Python 'Table' to a Julia 'PyTable'

```

Since `PyTable(x)` wraps Tables.jl compatible tables, I was hoping that it would “just work”.

Is there a quick way to get this working? My goal is no-copy conversion from the python object to the Julia object.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 2:59pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/2 "2023-05-27T14:59:23Z")

</div>

> [@tbeason](#):
>
> My goal is no-copy conversion from the python object to the Julia object.

> <https://github.com/apache/arrow-julia/issues/92>
>
> Hi @quinnj , thank you for just willing to consider this wild attempt! The only …pkg you need to re-create is PyArrow and \[awkward-1.0\](https://github.com/scikit-hep/awkward-1.0) on the python side and PyCall.jl on Julia side.
> 
> Create example \`arr\`:
> \`\`\`julia
> julia\> using PyCall
> 
> julia\> ak = pyimport("awkward");
> 
> julia\> arr = ak.Array(py"\[\[1,2,3\], \[\], \[4,5\]\]")
> PyObject \<Array \[\[1, 2, 3\], \[\], \[4, 5\]\] type='3 \* var \* int64'\>
> 
> julia\> arr.layout
> PyObject \<ListOffsetArray64\>
> \<offsets\>\<Index64 i="\[0 3 3 5\]" offset="0" length="4" at="0x0000037ff330"/\>\</offsets\>
> \<content\>\<NumpyArray format="l" shape="5" data="1 2 3 4 5" at="0x000003a256a0"/\>\</content\>
> \</ListOffsetArray64\>
> \`\`\`
> 
> Then you can get an pyarrow object via:
> \`\`\`julia
> julia\> arr\_arrow = ak.to\_arrow(arr)
> PyObject \<pyarrow.lib.ListArray object at 0x7f2144343048\>
> ...
> ..
> 
> julia\> @time \[Int64\[x...\] for x in arr\]
> 0.034800 seconds (38.72 k allocations: 2.031 MiB)
> 3-element Array{Array{Int64,1},1}:
> \[1, 2, 3\]
> \[\]
> \[4, 5\]
> 
> \`\`\`
> 
> Currently the fastest / least copy method of re-using as been:
> \`\`\`julia
> function view\_ak(arr)
> c = PyArray(arr.layout."content")
> o = PyArray(arr.layout."offsets")
> @views \[c\[o\[i\]+1:o\[i+1\]\] for i in 1:length(o)-1\]
> end
> 
> julia\> @time view\_ak(arr)
> 0.000089 seconds (37 allocations: 1.609 KiB)
> 3-element Array{SubArray{Int64,1,PyArray{Int64,1},Tuple{UnitRange{Int64}},false},1}:
> \[1, 2, 3\]
> 0-element view(::PyArray{Int64,1}, 4:3) with eltype Int64
> \[4, 5\]
> 
> \`\`\`

the latest “working” example is this: [Re-use Awkward / pyarrow IPC for Julia Arrow.jl · GitHub](https://gist.github.com/Moelf/de9e6be8575ed5a0399e04637fb935cf)

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 3:06pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/3 "2023-05-27T15:06:46Z")

</div>

Yikes. That looks a bit too fragile for me. The problem exists for polars dataframes as well, it seems. Is that really the only way?

Is pandas conversion “no copy”? I just thought that using pyarrow or polars would be better since they have more focus on performance.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 3:16pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/4 "2023-05-27T15:16:47Z")

</div>

think about this at a higher level. In-memory, pandas or polars or pyarrow tables are just a bunch of bytes with schema/pointers. So to do “non-copy”, you either have to make Julia able to re-interpret data bytes, or you have to manipulate Python in-memory structure through Python.

You want option #1, but understanding a impl-dependent blob is hard, which is why Arrow exists, the “batch” is intended to use as IPC blob. But, the IPC assumes the bytes are “contiguous” (otherwise you can’t easily pass it around). When you open a feather file with pyarrow or polars, it’s MMap-ed and in-memory has Rust/C++/Python stuff, the “write to buffer” is a way to say “now please give me something fully up to spec with Arrow IPC I need to use it elsewhere”

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 3:20pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/5 "2023-05-27T15:20:51Z")

</div>

That I understand. But the gist isn’t exactly written in a way that is completely certain of its correctness lol. If this functionality were to go into PythonCall, do you think that would be the way it would be implemented? How safe is it to use? I want no copy but I also want no (silently) lost data!

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 3:23pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/6 "2023-05-27T15:23:16Z")

</div>

> [@tbeason](#):
>
> exactly written in a way that is completely certain of its correctness lol

🤷‍♂️

all of the apis I used are public, for both pyarrow and Arrow.jl

you mean you want someone to put hacky code behind a function call so you don’t know what it’s actually doing? PythonCall itself is plenty hacky and there have been a few “reliability” issues.

If you have specific concerns other than “this looks hacky because it’s just some random person’s gist” I’m happy to see if I can address

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 3:30pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/7 "2023-05-27T15:30:49Z")

</div>

> [@jling](#):
>
> this looks hacky because it’s just some random person’s gist

Not that! Terms like “practically viable” and “This in principle allows one…” don’t make it seem like the final answer, that’s all.

I will try it out. I assume that all of the `awkward` stuff is not relevant since I already have a `pyarrow` table? And when I call `to_batches()`, I get a list rather than 1, so I guess I need to write them all to the sink and then do `take!`?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 3:36pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/8 "2023-05-27T15:36:52Z")

</div>

> [@tbeason](#):
>
> If this functionality were to go into PythonCall, do you think that would be the way it would be implemented?

Honestly yes. the IPC bytes written out by pyarrow is completely safe because it’s spec-ed. It would be far more unreliable to rely on Python internal objects and try to re-use pointer to each Arrow `page`, probably impossible

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 3:37pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/9 "2023-05-27T15:37:14Z")

</div>

> [@tbeason](#):
>
> And when I call `to_batches()`, I get a list rather than 1, so I guess I need to write them all to the sink and then do `take!`?

yeah, indeed, the `gist` is dealing with something that only has 1 batch

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 3:49pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/10 "2023-05-27T15:49:26Z")

</div>

Very, very cool. Only modification that I needed was to do

```julia
pywith(pa.ipc.new_stream(jl_sink, first(pa_batches).schema)) do writer
        writer.write_batch.(pa_batches)
end;

```

Works fairly fast on a 5M x 12 dataframe.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 27, 2023, 3:57pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/11 "2023-05-27T15:57:09Z")

</div>

For reference (mostly my future self), here is a function that contains the code:

```julia
# requires DataFrames, Arrow, PythonCall + pyarrow
function pyarrow_to_jldf(pyarr)

    pa_batches = pyarr.to_batches()

    jl_sink = IOBuffer()

    pywith(pa.ipc.new_stream(jl_sink, first(pa_batches).schema)) do writer
            writer.write_batch.(pa_batches)
    end;

    df = DataFrame(Arrow.Table(take!(jl_sink)))

    close(jl_sink)

    return df

end

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 27, 2023, 4:03pm UTC](https://discourse.julialang.org/t/pyarrow-conversion-with-pythoncall/99485/12 "2023-05-27T16:03:17Z")

</div>

I have had issues with pythonCall when pkged into a Julia packages, maybe try PyCall in that case