# \#data-compression

**URL:** https://discourse.julialang.org/tag/data-compression/1018.md

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

---

## [Reading compressed csv gzopen cuts file short](https://discourse.julialang.org/t/reading-compressed-csv-gzopen-cuts-file-short/137885)

<div class="topic-metadata">

**Author:** [@xor0110](https://discourse.julialang.org/u/xor0110)\
**Replies:** 4\
**Last updated:** [July 1, 2026, 5:31pm UTC](https://discourse.julialang.org/t/reading-compressed-csv-gzopen-cuts-file-short/137885 "2026-07-01T17:31:48Z")

</div>

I seem to have stumbled into some kind of bug. I am working with large compressed CSV files. I’m loading the data with: mydata = gzopen(x-\>CSV.read(x, DataFrame), mydatafile) Then I just get this warning: ┌ Warning: t…

---

## [JLD2 CodecLz4 compression stopped working](https://discourse.julialang.org/t/jld2-codeclz4-compression-stopped-working/132249)

<div class="topic-metadata">

**Author:** [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Replies:** 2\
**Last updated:** [September 10, 2025, 1:07pm UTC](https://discourse.julialang.org/t/jld2-codeclz4-compression-stopped-working/132249 "2025-09-10T13:07:37Z")

</div>

Hi, all of a sudden, compressing a Jld2 with Lz4 throws an error: using JLD2 using CodecLz4 name = rand(10) detail = rand(10) values = rand(10) jldopen("./test.jdl2", "w"; compress = LZ4FrameCompressor()) do f …

---

## [\[pre-ANN\] CompressedBeliefMDPs.jl: Solve large POMDPs in POMDPs.jl!](https://discourse.julialang.org/t/pre-ann-compressedbeliefmdps-jl-solve-large-pomdps-in-pomdps-jl/112625)

<div class="topic-metadata">

**Author:** [@FlyingWorkshop](https://discourse.julialang.org/u/FlyingWorkshop)\
**Replies:** 0\
**Last updated:** [April 7, 2024, 4:28am UTC](https://discourse.julialang.org/t/pre-ann-compressedbeliefmdps-jl-solve-large-pomdps-in-pomdps-jl/112625 "2024-04-07T04:28:35Z")

</div>

https://github.com/JuliaPOMDP/CompressedBeliefMDPs.jl Hello everyone! I’m excited to announce CompressedBeliefMDPs.jl (docs on repo), an upcoming addition to the POMDPs.jl ecosystem. CompressedBeliefMDPs.jl provides a …

---

## [Plots; pythonplot(): approximate heatmap as gradients to save space](https://discourse.julialang.org/t/plots-pythonplot-approximate-heatmap-as-gradients-to-save-space/108293)

<div class="topic-metadata">

**Author:** [@maltee1](https://discourse.julialang.org/u/maltee1)\
**Replies:** 8\
**Last updated:** [January 10, 2024, 8:02am UTC](https://discourse.julialang.org/t/plots-pythonplot-approximate-heatmap-as-gradients-to-save-space/108293 "2024-01-10T08:02:44Z")

</div>

Hi, Is it possible to turn a heatmap from individual pixels into something that’s a little easier on storage? Especially, if nothing very exciting is happening in most places of the heatmap, a collection of gradients co…

---

## [Partially decompressing Bzip2 files](https://discourse.julialang.org/t/partially-decompressing-bzip2-files/96429)

<div class="topic-metadata">

**Author:** [@tristian](https://discourse.julialang.org/u/tristian)\
**Replies:** 2\
**Last updated:** [March 23, 2023, 1:00am UTC](https://discourse.julialang.org/t/partially-decompressing-bzip2-files/96429 "2023-03-23T01:00:58Z")

</div>

As the title of the question states, I have a few \*.csv.bzip2 compressed files that are as big as 13GB when decompressed. The files have a structured format and are essentially CSV files. I’d like to read the headers of…

---

## [How to save an array to disk in compressed form?](https://discourse.julialang.org/t/how-to-save-an-array-to-disk-in-compressed-form/40325)

<div class="topic-metadata">

**Author:** [@andrey2185](https://discourse.julialang.org/u/andrey2185)\
**Replies:** 9\
**Last updated:** [January 24, 2023, 2:32pm UTC](https://discourse.julialang.org/t/how-to-save-an-array-to-disk-in-compressed-form/40325 "2023-01-24T14:32:14Z")

</div>

Hello, how to save an array to disk in compressed form?

---

## [How to save a large Float32 array on disk using data compression (failed attempt with JLD2)?](https://discourse.julialang.org/t/how-to-save-a-large-float32-array-on-disk-using-data-compression-failed-attempt-with-jld2/77413)

<div class="topic-metadata">

**Author:** [@rio](https://discourse.julialang.org/u/rio)\
**Replies:** 2\
**Last updated:** [January 24, 2023, 9:13am UTC](https://discourse.julialang.org/t/how-to-save-a-large-float32-array-on-disk-using-data-compression-failed-attempt-with-jld2/77413 "2023-01-24T09:13:54Z")

</div>

I would like to save a large array on disk using if possible some type of data compression to save disk space. Then, I would like to read the file in Julia for future usage. My attempt below using JLD2 works only for sma…

---

## [Data Storage Quo Vadis under Julia: HDF5 - JLD2 - MAT -](https://discourse.julialang.org/t/data-storage-quo-vadis-under-julia-hdf5-jld2-mat/80159)

<div class="topic-metadata">

**Author:** [@ellocco](https://discourse.julialang.org/u/ellocco)\
**Replies:** 0\
**Last updated:** [April 27, 2022, 5:35pm UTC](https://discourse.julialang.org/t/data-storage-quo-vadis-under-julia-hdf5-jld2-mat/80159 "2022-04-27T17:35:45Z")

</div>

The good news: Julia can beat the phyton data-compression package “hdf5storage”. But it is useful to have a closer look into the matter. And, if you store data, it should be better or at least as good as hdf5storage. …

---

## [Compressed array-on-disk](https://discourse.julialang.org/t/compressed-array-on-disk/69410)

<div class="topic-metadata">

**Author:** [@grero](https://discourse.julialang.org/u/grero)\
**Replies:** 9\
**Last updated:** [October 11, 2021, 1:36pm UTC](https://discourse.julialang.org/t/compressed-array-on-disk/69410 "2021-10-11T13:36:51Z")

</div>

I am working on an algorithm that fits an HMM model to some data. As part of the fitting I need to run the so-called \[forward-backward\] algorithm (Forward–backward algorithm - Wikipedia). This involves constructing matri…

---

## [Arrow.jl reading compressed(lz4 and zstd) arrow/feather format may have memory leak](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177)

<div class="topic-metadata">

**Author:** [@jesseventura](https://discourse.julialang.org/u/jesseventura)\
**Replies:** 3\
**Last updated:** [August 12, 2021, 1:47am UTC](https://discourse.julialang.org/t/arrow-jl-reading-compressed-lz4-and-zstd-arrow-feather-format-may-have-memory-leak/52177 "2021-08-12T01:47:19Z")

</div>

refer to memory leaking when reading compressed arrow files · Issue #80 · apache/arrow-julia · GitHub I found only when i have a compressed arrow format(lz4 or zstd) , the memory leak problem occurs. uncompressed forma…

---

## [A Julia-compatible alternative to zarr](https://discourse.julialang.org/t/a-julia-compatible-alternative-to-zarr/11842)

<div class="topic-metadata">

**Author:** [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Replies:** 19\
**Last updated:** [December 18, 2019, 8:17pm UTC](https://discourse.julialang.org/t/a-julia-compatible-alternative-to-zarr/11842 "2019-12-18T20:17:12Z")

</div>

Dear Julia community, we are looking for a object-storage compatible file format for quite large dense N-dimensional arrays. Our project partners suggested this Zarr — zarr 2.13.6 documentation format, but there seems t…

---

## [Compressing arbitrary Vector{UInt8} in memory](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058)

<div class="topic-metadata">

**Author:** [@janfrancu](https://discourse.julialang.org/u/janfrancu)\
**Replies:** 5\
**Last updated:** [October 9, 2018, 8:51am UTC](https://discourse.julialang.org/t/compressing-arbitrary-vector-uint8-in-memory/16058 "2018-10-09T08:51:44Z")

</div>

I am creating a lot of FlatBuffers, which I would like to store in S3 storage. Right now I use something akin to the following code to serialize the structure and put it to S3 bucket. fbStruct = ... fbBytes = FlatBuffe…

---

## [Data Import Types and Compression](https://discourse.julialang.org/t/data-import-types-and-compression/7907)

<div class="topic-metadata">

**Author:** [@ldsands](https://discourse.julialang.org/u/ldsands)\
**Replies:** 1\
**Last updated:** [December 22, 2017, 4:36am UTC](https://discourse.julialang.org/t/data-import-types-and-compression/7907 "2017-12-22T04:36:40Z")

</div>

Hello all, Sorry if this is a bad questions but I use data sets from ICPSR (a social science repository) and to use an example a dataset that I recently used is about 200 mg in uncompressed csv however I’m able to get t…

---

## [How to compress HDF5 data?](https://discourse.julialang.org/t/how-to-compress-hdf5-data/2636)

<div class="topic-metadata">

**Author:** [@programista](https://discourse.julialang.org/u/programista)\
**Replies:** 2\
**Last updated:** [March 13, 2017, 2:46pm UTC](https://discourse.julialang.org/t/how-to-compress-hdf5-data/2636 "2017-03-13T14:46:37Z")

</div>

How to compress HDF5 data ? julia\> using HDF5 julia\> fid=h5open("file.h5","w") HDF5 data file: file.h5 julia\> close(fid) julia\> julia\> fid = h5open("file.h5","r+") HDF5 data file: file.h5 julia\> write(fid,"A",rand(…