# How to read a compressed CSV file?

**URL:** <https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719>\
**Category:** New to Julia\
**Created:** [January 16, 2019, 7:25pm UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719 "2019-01-16T19:25:35Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 16, 2019, 7:25pm UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/1 "2019-01-16T19:25:35Z")

</div>

Hello.

I’m working with several large csv files. (Around 2GB each, million rows, thousands of columns).  
In order to reduce the space on disk (and speed to sync online) they are compressed.

I was using R to work with them.  
data.table’s fread let’s me read directly the compressed file with:

`myDT <- fread("7z e -y -bso0 -so mycompress.7z", stringsAsFactors=F, na.strings=c("", "NA")) # and sometimes selecting columns or rows.`

That executes transparently 7-zip and forwards the result to fread.

How can I do something similar with Julia?

I’m also considering using feather or hdf5 but I feel safer using csv for now, it’s easier for other people to access the files.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 16, 2019, 7:52pm UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/2 "2019-01-16T19:52:19Z")

</div>

If they are gz compressed, you can read them directly with [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl), see [here](https://github.com/queryverse/CSVFiles.jl/issues/33). But that won’t work for 7z compression, I’m afraid…

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 16, 2019, 11:26pm UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/3 "2019-01-16T23:26:32Z")

</div>

I first tried with “zip” but it didn’t work well together with fread.  
Would you suggest any other compressed file format to share data between R and Julia?  
For example “feather” doesn’t offer internal compression.  
Maybe some fast database? (I’m interested on Windows but something multiplatform would be nice).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 16, 2019, 11:45pm UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/4 "2019-01-16T23:45:15Z")

</div>

> [@davidanthoff](#):
>
> If they are gz compressed, you can read them directly with [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl), see [here](https://github.com/queryverse/CSVFiles.jl/issues/33). But that won’t work for 7z compression, I’m afraid…

Can’t you do

```julia-auto
myDT = open(`7z e -y -bso0 -so mycompress.7z`, "r") do io
    load(Stream(format"CSV", io)) |> DataFrame
end

```

to load it from a pipe just like in R?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 17, 2019, 12:20am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/5 "2019-01-17T00:20:36Z")

</div>

Yes, that probably would also work, I just haven’t tried it 🙂

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 17, 2019, 12:31am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/6 "2019-01-17T00:31:30Z")

</div>

```julia
using CSVFiles
using DataFrames
myDT = open(`7z e -y -bso0 -so mycompress.7z`, "r") do io
    load(Stream(format"CSV", io)) |> DataFrame
end

```

> ERROR: UndefVarError: Stream not defined  
> Stacktrace:  
> [1] (::getfield(Main, Symbol(“##9#10”)))(::Base.Process) at .\REPL[15]:2  
> [2] open(::getfield(Main, Symbol(“##9#10”)), ::Cmd, ::String) at .\process.jl:617  
> [3] top-level scope at none:0

What else do I need to do?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 17, 2019, 12:39am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/7 "2019-01-17T00:39:06Z")

</div>

Ah, you also need `using FileIO` (and first add the FileIO.jl package)!

I should just reexport `Stream` from `CSVFiles`…

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 17, 2019, 12:45am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/8 "2019-01-17T00:45:50Z")

</div>

> [@davidanthoff](#):
>
> using FileIO

OK, thanks, it seems to work.

Is it supposed to be read with possible missings?  
How can I know how the number of missings on each column?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 17, 2019, 1:16am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/9 "2019-01-17T01:16:06Z")

</div>

> [@Juan](#):
>
> Is it supposed to be read with possible missings?

I’m not entirely sure what you mean by that… Are there rows with missing values? [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl) should handle those just fine. If not, please open an issue.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 17, 2019, 1:18am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/10 "2019-01-17T01:18:31Z")

</div>

Yes, some rows on the csv have missings, it’s supposed to be like that because that value wasn’t measured.

```julia
 aa , bb  
1 , 11
2 ,
3 , 23

```

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 17, 2019, 1:23am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/11 "2019-01-17T01:23:02Z")

</div>

Ok, and I assume it loaded it properly? The way this should work is that the column `bb` in your `DataFrame` should now have a `missing` value in the second row.

I guess I’m just not sure whether there is a problem, or whether you are just reporting success 🙂

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [January 17, 2019, 1:51am UTC](https://discourse.julialang.org/t/how-to-read-a-compressed-csv-file/19719/12 "2019-01-17T01:51:48Z")

</div>

it’s OK, thanks.