# Compressing arbitrary Vector{UInt8} in memory

**URL:** <https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058>\
**Category:** General Usage\
**Tags:** data-compression\
**Created:** [October 9, 2018, 7:37am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058 "2018-10-09T07:37:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![janfrancu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janfrancu/32/5673_2.png) [@janfrancu](https://discourse.julialang.org/u/janfrancu)\
**Post date:** [October 9, 2018, 7:37am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/1 "2018-10-09T07:37:04Z")

</div>

I am creating a lot of FlatBuffers, which I would like to store in S3 storage. Right now I use something akin to the following code to serialize the structure and put it to S3 bucket.

```julia
fbStruct = ... 
fbBytes = FlatBuffers.bytes(FlatBuffers.build!(fbStruct))
AWSS3.put_s3(awstoken, bucket, path, fbBytes)

```

Is there a way to compress the `fbBytes` vector using `CodecZlib`? I have found a way to do that with some temporary file, however I don’t know how to make it all in memory.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 9, 2018, 7:54am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/2 "2018-10-09T07:54:23Z")

</div>

Have you tried an `IOBuffer`?

---

<div class="post-metadata">

**Author:** ![janfrancu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janfrancu/32/5673_2.png) [@janfrancu](https://discourse.julialang.org/u/janfrancu)\
**Post date:** [October 9, 2018, 8:06am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/3 "2018-10-09T08:06:36Z")

</div>

I have tried the `IOBuffer` like this

```julia
stream = GzipCompressorStream(IOBuffer(fbBytes))
newBytes = read(stream)
close(stream)

```

however the results are different from what I get, when using some temporary file and then reading it back.

```julia
open(GzipCompressorStream, filename, "w") do stream
    write(stream, fbBytes)
end

fileBytes = = open("a.fb.gz", "r") do f
    read(f)
end 

```

I am still getting my head around how the `TranscodingStreams`, `CodecZlib` and the base `IO` work together, so maybe I am using it all wrong. 🙂

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 9, 2018, 8:10am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/4 "2018-10-09T08:10:02Z")

</div>

It seems there is a direct `Array` API:

> **[GitHub - JuliaIO/CodecZlib.jl: zlib codecs for TranscodingStreams.jl.](https://github.com/JuliaIO/CodecZlib.jl)**
>
> zlib codecs for TranscodingStreams.jl. Contribute to JuliaIO/CodecZlib.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![janfrancu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janfrancu/32/5673_2.png) [@janfrancu](https://discourse.julialang.org/u/janfrancu)\
**Post date:** [October 9, 2018, 8:22am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/5 "2018-10-09T08:22:15Z")

</div>

Well that was hiding in plain sight for me. Thanks for pointing that out, I somehow thought that the `GzipCompressor` was deprecated, however that was the case with the old `GzipCompression` types. Will try that straightaway.

---

<div class="post-metadata">

**Author:** ![bicycle1885](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bicycle1885/32/107_2.png) [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)\
**Post date:** [October 9, 2018, 8:51am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058/6 "2018-10-09T08:51:44Z")

</div>

`transcode(GzipCompressor, data)` is the simplest way to compress in-memory data. However, it allocates a working space every time you try to compress a chunk of data. If you need to compress a lot of data chunks, you can avoid lots of allocations by reusing pre-allocating a compressor object as described in this example: [https://bicycle1885.github.io/TranscodingStreams.jl/stable/examples.html#Transcode-lots-of-strings-1](https://bicycle1885.github.io/TranscodingStreams.jl/stable/examples.html#Transcode-lots-of-strings-1).