# How well Apache Arrow’s zero copy methodology is supported?

**URL:** <https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797>\
**Category:** Data\
**Tags:** arrow\
**Created:** [April 22, 2021, 11:52am UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797 "2021-04-22T11:52:40Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [April 28, 2021, 3:19pm UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/21 "2021-04-28T15:19:18Z")

</div>

@nilshg , sorry for the slow response; from your example, it does seem like some over-copying is going on when materializing the views on your arrow data. I’ll look into it.

@Sami , in your example, as long as there’s still a reference to the original `at` file, then that memory isn’t going to be freed/reclaimed. So I don’t see anything unexpected in your example. If you were to set `at = nothing; GC.gc(); GC.gc()` and the memory wasn’t freed, that would be a concern.

---

<div class="post-metadata">

**Author:** ![Sami](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sami/32/20726_2.png) [@Sami](https://discourse.julialang.org/u/Sami)\
**Post date:** [April 29, 2021, 7:12am UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/22 "2021-04-29T07:12:08Z")

</div>

Thanks @quinnj . I understood `let` block wrongly before. I run code

```julia
using Tables
using Arrow
using TableOperations
print("start")
let at=Arrow.Table("big.arrow")
    let t= at |> TableOperations.select(:vendor_id, :passenger_count, :total_amount) |> TableOperations.filter(x -> x.total_amount > 10) |> Tables.columntable
        print("in")
    end
end
print("out")
GC.gc(); GC.gc();
print("done")

```

and memory becomes freed at the end.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [April 29, 2021, 2:27pm UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/23 "2021-04-29T14:27:08Z")

</div>

Ah yes, that’s correct. It’s the variables in the let clause that will be temporary/i.e. live for the duration of the block.

---

<div class="post-metadata">

**Author:** ![merlin](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@merlin](https://discourse.julialang.org/u/merlin)\
**Post date:** [May 1, 2021, 4:56am UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/24 "2021-05-01T04:56:02Z")

</div>

This explanation helps in so many ways

---

<div class="post-metadata">

**Author:** ![merlin](https://avatars.discourse-cdn.com/v4/letter/m/8baadc/32.png) [@merlin](https://discourse.julialang.org/u/merlin)\
**Post date:** [May 1, 2021, 5:08am UTC](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797/25 "2021-05-01T05:08:50Z")

</div>

It would be nice if TableOperations continued this philosophy. Many jobs start needing to saw up huge data sets and that tool doesn’t have to be the same one used for fine data frame transformations.

[Previous page](https://discourse.julialang.org/t/how-well-apache-arrow-s-zero-copy-methodology-is-supported/59797.md?page=1)