# Arrow.jl reading compressed(lz4 and zstd) arrow/feather format may have memory leak

**URL:** <https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177>\
**Category:** Data\
**Tags:** arrow, data-compression\
**Created:** [December 21, 2020, 2:41pm UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177 "2020-12-21T14:41:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jesseventura](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jesseventura/32/7638_2.png) [@jesseventura](https://discourse.julialang.org/u/jesseventura)\
**Post date:** [December 21, 2020, 2:41pm UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177/1 "2020-12-21T14:41:40Z")

</div>

refer to [memory leaking when reading compressed arrow files · Issue #80 · apache/arrow-julia · GitHub](https://github.com/JuliaData/Arrow.jl/issues/80)

I found only when i have a compressed arrow format(lz4 or zstd) , the memory leak problem occurs.  
uncompressed format is OK  
may be a bug in 3rd party libs ?

test env: julia1.5.3 stable / julia 1.7-dev  
Arrow 1.0.3  
linux mint 20

```julia
using Arrow
using ProgressMeter
using DataFrames

for i in range(1,5)
    df = DataFrame(rand(Float64,(100000,1000)))
    Arrow.write("data.$i.feather.lz4",df, compress=:lz4)
    Arrow.write("data.$i.feather.zstd",df, compress=:zstd)
end

@showprogress 1 "read repeatly" for i in range(1,5)
    df = DataFrame(Arrow.Table("data.$i.feather"))
end

```

 ![Screenshot_2020-12-22_11-12-23](https://global.discourse-cdn.com/julialang/original/3X/a/5/a505322faed1a12df4db1ea4e7f096af9d019a2c.png)

and in CodecLz4 frame\_compression.jl implements initialize and finalize…

```julia
 """
    TranscodingStreams.finalize(codec::LZ4FrameDecompressor)

Finalizes the LZ4F Decompression Codec.
 """
function TranscodingStreams.finalize(codec::LZ4FrameDecompressor)::Nothing
    LZ4F_freeDecompressionContext(codec.dctx[])
    nothing
end

```

```julia
"""
Releases the memory of a `LZ4F_cctx`.
"""
function LZ4F_freeCompressionContext(cctx::Ptr{LZ4F_cctx})
    ccall((:LZ4F_freeCompressionContext, liblz4), Csize_t, (Ptr{LZ4F_cctx},), cctx)
end

```

---

<div class="post-metadata">

**Author:** ![jesseventura](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jesseventura/32/7638_2.png) [@jesseventura](https://discourse.julialang.org/u/jesseventura)\
**Post date:** [December 29, 2020, 7:16am UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177/2 "2020-12-29T07:16:28Z")

</div>

## especially reading PYTHON exported feather format…

### here is python code creating several feather files…

```julia-auto
import numpy as np
from pyarrow.feather import write_feather
import os

for i in range(5):
    for comp_type in ['lz4','zstd']:
        fout = 'pydata.{}.feather.{}'.format(i+1,comp_type)
        if os.path.isfile(fout):
            continue
        write_feather(pd.DataFrame(pd.np.random.rand(100000,1000)), fout, compression=comp_type)

```

# and here is repeatly

```julia-auto
using Arrow
using DataFrames

@showprogress 1 "read repeatly" for ii in range(1,10)
    i = mod(ii,5)+1
    @time df = copy(DataFrame(Arrow.Table("pydata.$i.feather.zstd"), copycols=true))
    finalize(df)
    sleep(1)
end

```

## and here is the mem size ,nearly up to 16G

| PID | VSZ | RSS |
| --- | --- | --- |
| 55623 | 19133564 | 17207488 |

---

<div class="post-metadata">

**Author:** ![jesseventura](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jesseventura/32/7638_2.png) [@jesseventura](https://discourse.julialang.org/u/jesseventura)\
**Post date:** [May 6, 2021, 3:41am UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177/3 "2021-05-06T03:41:34Z")

</div>

issue been fixed

---

<div class="post-metadata">

**Author:** ![jesseventura](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jesseventura/32/7638_2.png) [@jesseventura](https://discourse.julialang.org/u/jesseventura)\
**Post date:** [August 12, 2021, 1:47am UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177/4 "2021-08-12T01:47:19Z")

</div>

[https://github.com/JuliaData/Arrow.jl/issues/93](https://github.com/JuliaData/Arrow.jl/issues/93)