# Pipe to CSV.Source

**URL:** <https://discourse.julialang.org/t/pipe-to-csv-source/3764>\
**Category:** Data\
**Tags:** question\
**Created:** [May 17, 2017, 7:31am UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764 "2017-05-17T07:31:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 17, 2017, 7:31am UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764/1 "2017-05-17T07:31:56Z")

</div>

I have a gzip-compressed `CSV` file in which uses commas for the decimal mark. I can uncompress and then use `tr` on the command line, but I would prefer doing that using a pipe from Julia. I tried

```julia
using CSV
using DataStreams
src1 = open(pipeline(`zcat /tmp/test.gz`, `tr , .`), "r") do io
    CSV.Source(io)
end

```

put the process appears to be idle (around `0%` CPU), so I guess it is waiting for something, how can I fix this?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 21, 2017, 6:35pm UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764/2 "2017-05-21T18:35:01Z")

</div>

`CSV.Source(io)` just creates a `Source` object, but nothing will be read. Did you mean to use `CSV.read`?

---

<div class="post-metadata">

**Author:** ![joshbode](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshbode/32/260_2.png) [@joshbode](https://discourse.julialang.org/u/joshbode)\
**Post date:** [May 22, 2017, 2:36am UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764/3 "2017-05-22T02:36:00Z")

</div>

Here is one way in pure Julia to translate the commas to dots in-stream using `BufferedStreams`:

```julia
using BufferedStreams

type Translator{T <: IO}
    io::T
    from::UInt8
    to::UInt8
end
Translator(io::IO; from::Char=',', to::Char='.') = Translator(io, UInt8(from), UInt8(to))

function BufferedStreams.readbytes!(source::Translator, buffer::Vector{UInt8}, from::Int, to::Int)
    v = view(buffer, from:to)
    n = readbytes!(source.io, v)
    v[v .== source.from] = source.to
    n
end
Base.eof(source::Translator) = eof(source.io)
Base.close(source::Translator) = close(source.io)
Base.nb_available{T <: Translator}(s::BufferedStreams.BufferedInputStream{T}) = !eof(s)

```

Then you can use a `BufferedInputStream{Translator}` object in place of an `IO` object, i.e.

```julia
using Libz
io = ZlibInflateInputStream(open("/tmp/test.gz", "r"))
t = BufferedInputStream(Translator(io; from=',', to='.'))
@assert isa(t, IO)
d = CSV.read(t)

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 22, 2017, 12:22pm UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764/4 "2017-05-22T12:22:30Z")

</div>

Thanks. Now I am trying to get something like this working:

```julia
function read_into_db(csv_path, db, tablename, CSV_args)
    tmp = tempname()
    println("Uncompressing $(csv_path)")
    println(" into $(tmp) (may take a few minutes) ...")
    open(io->run(pipeline(pipeline(`zcat $(csv_path)`, `tr , .`), stdout=io)), tmp, "w")
    println(" filesize is $(round(filesize(tmp)/(2^30),2)) Gb")
    println(" ... forming CSV.Source")
    src = CSV.Source(tmp; CSV_args...)
    println(" ... loading into database")
    SQLite.load(db, tablename, src)
    println(" ... removing temporary file")
    rm(tmp)
end

```

The fix from @joshbode should help with uncompression and replacement. However, `CSV.Source` seems to insist on traversing the whole file first, so it is essentially a two-pass operation. Is there anything I can do about that, to make in single-pass?

In general, what is the best strategy for dealing with data that does not fit into memory in the current Julia ecosystem? If I read into an `SQLite.jl` database, can I dump the data somehow columnwise using `Feather.jl` (columns fit in memory)?

---

<div class="post-metadata">

**Author:** ![joshbode](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshbode/32/260_2.png) [@joshbode](https://discourse.julialang.org/u/joshbode)\
**Post date:** [May 22, 2017, 10:23pm UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764/5 "2017-05-22T22:23:17Z")

</div>

Yep - looks like you’re right - `CSV.Source` ([Source.jl:59](https://github.com/JuliaData/CSV.jl/blob/master/src/Source.jl#L59)) will do a full `read` on the stream, regardless of whether the header and types are specified. The only input type `CSV.Source` won’t instantly consume is `IOBuffer`.