# Cannot read file written by Arrow.jl in Python
**URL:** https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889
**Category:** Data
**Tags:** python, arrow
**Created:** [April 25, 2023, 8:51am UTC](https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889 "2023-04-25T08:51:32Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)
#### Post date: [April 25, 2023, 8:51am UTC](https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889/1 "2023-04-25T08:51:32Z")
I’m using a Julia environment with Arrow v2.5.2 and DataFrames v1.5.1, and Python 3.10.2 with pandas 1.5.0 and pyarrow 11.0.0. From the [User manual](https://arrow.juliadata.org/dev/manual/#Arrow.write) I think I could do:
```julia
using Arrow, DataFrames
m = rand(20,10)
open("examplematrix.feather", "w") do io
Arrow.write(io, DataFrame(m, :auto))
end
```
And then read in Python with either:
```python
import pandas
pandas.read_feather("examplematrix.feather")
```
or
```julia
from pyarrow import feather
feather.read_feather("examplematrix.feather")
```
But in either case I get:
```julia
ArrowInvalid Traceback (most recent call last)
Cell In [19], line 1
----> 1 pd.read_feather("examplematrix.feather")
File c:\Program Files\Python310\lib\site-packages\pandas\io\feather_format.py:132, in read_feather(path, columns, use_threads, storage_options)
126 from pyarrow import feather
128 with get_handle(
129 path, "rb", storage_options=storage_options, is_text=False
130 ) as handles:
--> 132 return feather.read_feather(
133 handles.handle, columns=columns, use_threads=bool(use_threads)
134 )
File c:\Program Files\Python310\lib\site-packages\pyarrow\feather.py:226, in read_feather(source, columns, use_threads, memory_map, **kwargs)
199 def read_feather(source, columns=None, use_threads=True,
200 memory_map=False, **kwargs):
201 """
202 Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use
203 feather.read_table.
(...)
224 The contents of the Feather file as a pandas.DataFrame
225 """
--> 226 return (read_table(
227 source, columns=columns, memory_map=memory_map,
...
File c:\Program Files\Python310\lib\site-packages\pyarrow\error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()
File c:\Program Files\Python310\lib\site-packages\pyarrow\error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: Not a Feather V1 or Arrow IPC file
```
I also tried with `feather.read_table` as ssuggested in the error message, but I get the same `ArrowInvalid` error.
Am I doing something wrong? I haven’t found other messages reporting this problem.
EDIT: with Feather.jl it does work, but I want compression support, that that package does not have as far as I know.
---
### Author:  [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)
#### Post date: [April 25, 2023, 10:01am UTC](https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889/2 "2023-04-25T10:01:23Z")
Solved. It works if don’t pass `Arrow.write` an `IOStream`, but the string with the file path.
> [@heliosdrm](#):
>
> `Arrow.write("examplematrix.feather"., DataFrame(m, :auto))`
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [April 25, 2023, 10:28am UTC](https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889/3 "2023-04-25T10:28:44Z")
I don’t know what you did exactly in the first case, but it doesn’t seem to be a valid Arrow file (could you even read it back in, in Julia?). It’s 2704 bytes, and the other 3266 bytes (and you have an extra period, otherwise that example worked), and only the latter starts with he text ARROW1, I suppose a magic cookie to signal the Arrow file format. Still the “file” command may not be aware of it and shows is both cases:
```julia
shell> file examplematrix.feather
examplematrix.feather: data
```
---
### Author:  [@heliosdrm](https://discourse.julialang.org/u/heliosdrm)
#### Post date: [April 25, 2023, 12:36pm UTC](https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl-in-python/97889/4 "2023-04-25T12:36:55Z")
Yes, both files are read back successfully in Julia, and have the same contents. However, you are right: according to the specification ([Arrow Columnar Format — Apache Arrow v14.0.0](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format)) the file created with `open` looks wrong.
I opened an issue commenting on that:
>
>
> Context:
> https://discourse.julialang.org/t/cannot-read-file-written-by-arrow-jl…-in-python/97889/2
>
> As mentioned in one of the answers, the file that results from:
>
> \`\`\`julia
> open(filename, "w") do io
> Arrow.write(io, table)
> end
> \`\`\`
> does not include the magic "ARROW1" in the start/end, and there may be other problems.
>
> It does work with \`Arrow.write(filename, table)\`, though (\`filename\` being the path to the file).
EDIT: And then I have just closed it, after re-reading the docstring and finding that if I pass a `IOStream` for a file, I should add the keyword argument `file=true`.