# Is it possible to do a simple example of using Arrow.jl to pass data from/to Python?

**URL:** <https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146>\
**Category:** General Usage\
**Tags:** python, arrow\
**Created:** [April 28, 2021, 4:19am UTC](https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146 "2021-04-28T04:19:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [April 28, 2021, 4:19am UTC](https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146/1 "2021-04-28T04:19:02Z")

</div>

Say I am using PyCall.jl to start a python session. Can I receive an arrow vector from python without having to write it to disk?

What’s the mechanism to do so? A simple MWE would be highly appreciated!

Also, if I call Julia from Python, is it possible to receive a Julia vector via arrow in python?

---

<div class="post-metadata">

**Author:** ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)\
**Post date:** [April 28, 2021, 4:50am UTC](https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146/2 "2021-04-28T04:50:16Z")

</div>

Yes! To pass arrow data back and forth, you need to first convert it to a byte array in the sender, and then interpret that byte array in the receiver. Here are some functions I wrote to do so, with dataframes:

Python:

```julia
import pyarrow as pa
def convert_to_arrow_bytes(df: pd.DataFrame) -> bytearray:
    """
    Efficiently convert a dataframe to arrow bytes in memory
    For transfer to other processes
    Modified from https://github.com/JuliaData/Arrow.jl/blob/main/test/pyarrow_roundtrip.jl
    """
    batch = pa.record_batch(df)
    sink = pa.BufferOutputStream()
    writer = pa.ipc.new_stream(sink, batch.schema)
    writer.write_batch(batch)
    writer.close()
    buf = sink.getvalue()
    jbytes = buf.to_pybytes()
    return bytearray(jbytes)

def receive_arrow_bytes(byte_array: bytearray) -> pd.DataFrame:
    reader = pa.ipc.open_stream(byte_array)
    pyarrow_table = reader.read_all()
    return pyarrow_table.to_pandas()

```

Julia:

```julia
function load_df(bytearray::Vector{UInt8})::DataFrame
    return bytearray |> Arrow.Table |> DataFrame |> disallowmissing!
end

function df_to_arrow_bytes(df)
    io = IOBuffer()
    Arrow.write(io, df)
    seekstart(io)
    byte_array = take!(io)
    return byte_array
end

```

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [April 28, 2021, 5:05am UTC](https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146/3 "2021-04-28T05:05:24Z")

</div>

This is awesome! I should’ve seen it in the test folder too.

This will be so cool!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [April 28, 2021, 5:08am UTC](https://discourse.julialang.org/t/is-it-possible-to-do-a-simple-example-of-using-arrow-jl-to-pass-data-from-to-python/60146/4 "2021-04-28T05:08:17Z")

</div>

how does python and julia know which byte array to read from?

Is there a MWE showing how to do that from Julia and python?

say I have file.jl, what in there allows me to call a python function go generate a dataframe and then receive it in Julia?

The other way around, in a file2.py, say I call a Julia function to create a dataframe, how do I receive it in python?