# Setindex! issue with DataFrame saved as Arrow file
**URL:** https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307
**Category:** Data
**Tags:** dataframes, arrow
**Created:** [November 15, 2022, 6:00pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307 "2022-11-15T18:00:32Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)
#### Post date: [November 15, 2022, 6:00pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/1 "2022-11-15T18:00:32Z")
Working with a DataFrame saved as an Arrow file:
`fn = download("https://www.dropbox.com/s/322g26p3apdeqpf/granules.remote?dl=0")`
Loading in the file as a df
`g = DataFrame(Arrow.Table(fn))`
Copy the DataFrame
`g0 = copy(g)`
Set row 1 of g0 == row 1 of g
`g0[1,:] = g[1,:]`
Results in the following error:
ERROR: CanonicalIndexError: setindex! not defined for Arrow.List{String, Int32, Vector{UInt8}}
Stacktrace:
[1] error\_if\_canonical\_setindex(#unused#::IndexLinear, A::Arrow.List{String, Int32, Vector{UInt8}}, #unused#::Int64)
@ Base ./abstractarray.jl:1352
[2] setindex!(A::Arrow.List{String, Int32, Vector{UInt8}}, v::String, I::Int64)
@ Base ./abstractarray.jl:1343
[3] insert\_single\_entry!(df::DataFrame, v::String, row\_ind::Int64, col\_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/JZ7x5/src/dataframe/dataframe.jl:631
[4] setindex!(df::DataFrame, v::String, row\_ind::Int64, col\_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/JZ7x5/src/dataframe/dataframe.jl:659
[5] setindex!(df::DataFrame, v::DataFrameRow{DataFrame, DataFrames.Index}, row\_ind::Int64, col\_inds::Colon)
@ DataFrames ~/.julia/packages/DataFrames/JZ7x5/src/dataframerow/dataframerow.jl:255
[6] top-level scope
@ REPL[53]:1
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [November 15, 2022, 6:25pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/2 "2022-11-15T18:25:47Z")
I cannot reproduce it. I get:
```julia
julia> g0[1,:] = g[1,:]
DataFrameRow
Row │ id center polygon granules
│ String NamedTup… NamedTup… Array…
─────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ lat[+72+74]lon[-76-74] (lat = 73.0, lon = -75.0) (min_x = -76.0, min_y = 72.0, ma… NamedTuple{(:id, :url, :bbox, :i…
```
under Julia 1.8.2 and:
```julia
(@v1.8) pkg> st DataFrames Arrow
Status `C:\Users\bogum\.julia\environments\v1.8\Project.toml`
[69666777] Arrow v2.3.0
[a93c6f00] DataFrames v1.4.3
```
---
### Author:  [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)
#### Post date: [November 15, 2022, 6:34pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/3 "2022-11-15T18:34:33Z")
Sigh… I was using `DataFrames v1.3.2` updating to `v1.3.6` resolves the issue. Thanks @bkamins
---
### Author:  [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)
#### Post date: [November 15, 2022, 7:59pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/4 "2022-11-15T19:59:05Z")
@bkamins looks like the issue persists using:
[69666777] Arrow v2.3.0
[a93c6f00] DataFrames v1.4.3
This works:
`fn = download("https://www.dropbox.com/s/322g26p3apdeqpf/granules.remote?dl=0")`
`g = DataFrame(Arrow.Table(fn))`
`g0 = copy(g)`
`g0[1,:] = g[1,:]`
But when I define g0 from file it does not:
`g0 = DataFrame(Arrow.Table(fn))`
`g0[1,:] = g[1,:]`
throws the following error:
ERROR: CanonicalIndexError: setindex! not defined for Arrow.List{String, Int32, Vector{UInt8}}
Stacktrace:
[1] error\_if\_canonical\_setindex(#unused#::IndexLinear, A::Arrow.List{String, Int32, Vector{UInt8}}, #unused#::Int64)
@ Base ./abstractarray.jl:1352
[2] setindex!(A::Arrow.List{String, Int32, Vector{UInt8}}, v::String, I::Int64)
@ Base ./abstractarray.jl:1343
[3] insert\_single\_entry!(df::DataFrame, v::String, row\_ind::Int64, col\_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/KKiZW/src/dataframe/dataframe.jl:659
[4] setindex!(df::DataFrame, v::String, row\_ind::Int64, col\_ind::Symbol)
@ DataFrames ~/.julia/packages/DataFrames/KKiZW/src/dataframe/dataframe.jl:688
[5] setindex!(df::DataFrame, v::DataFrameRow{DataFrame, DataFrames.Index}, row\_ind::Int64, col\_inds::Colon)
@ DataFrames ~/.julia/packages/DataFrames/KKiZW/src/dataframerow/dataframerow.jl:255
[6] top-level scope
@ REPL[37]:1
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [November 15, 2022, 8:32pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/5 "2022-11-15T20:32:08Z")
But that’s expected, but default the data isn’t copied and read only, so you can’t mutate it.
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [November 15, 2022, 8:35pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/6 "2022-11-15T20:35:22Z")
> [@alex-s-gardner](#):
>
> `g = DataFrame(Arrow.Table(fn))`
Try
```julia
g = DataFrame(Arrow.Table(fn); copycols=true)
```
Then you shouldn’t need the `copy` later.
---
### Author:  [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)
#### Post date: [November 15, 2022, 8:42pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/7 "2022-11-15T20:42:18Z")
Ahh… ok. Thank you all for the guidance.
1. I wrongly assumed it was lazy loading and would load once operated on.
2. This error seems really cryptic for “DataFrame is read only”… maybe a new error could be added
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [November 15, 2022, 8:48pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/8 "2022-11-15T20:48:05Z")
By default, Arrow.jl uses [memory-mapping](https://docs.julialang.org/en/v1/stdlib/Mmap/) when loading a table from a file, as you say, and `copycols=true` tells it go eagerly load all the data into `Vector`’s. So if you are memory constrained, you may not want to use `copycols=true`.
The problem is Arrow.jl presents a read-only view of the data, so you can’t do in-place operations like `g0[1,:] = g[1,:]` on it. You need your DataFrame to be backed by mutable columns to do that. But you can do things like load the table with `copycols=false` to get the memory-map backed version, then `select` and `subset` (with `view=true`) to select columns or rows without making a copy, reducing the data size. _Then_ once you have only the data you need, you could make a copy to move that data into mutable vectors.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [November 15, 2022, 9:50pm UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/9 "2022-11-15T21:50:42Z")
> [@alex-s-gardner](#):
>
> This error seems really cryptic
The problem is that DataFrames.jl is not aware of this error. It is Arrow.jl that throws the error.
@quinnj - maybe there could be defined `setindex!` for immutable Arrow types that would produce a more informative error?
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [November 16, 2022, 1:03am UTC](https://discourse.julialang.org/t/setindex-issue-with-dataframe-saved-as-arrow-file/90307/10 "2022-11-16T01:03:23Z")
Yeah, that’s probably a good idea.