# Streaming gziped file to FASTQ.Reader - where to add method?

**URL:** <https://discourse.julialang.org/t/streaming-gziped-file-to-fastq-reader-where-to-add-method/36266>\
**Category:** General Usage\
**Tags:** question, biology, input-output\
**Created:** [March 20, 2020, 3:01pm UTC](https://discourse.julialang.org/t/streaming-gziped-file-to-fastq-reader-where-to-add-method/36266 "2020-03-20T15:01:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [March 20, 2020, 3:01pm UTC](https://discourse.julialang.org/t/streaming-gziped-file-to-fastq-reader-where-to-add-method/36266/1 "2020-03-20T15:01:59Z")

</div>

FASTQ is a file format used for DNA sequencing, and we have a handy reader in the [FASTX package](https://biojulia.net/FASTX.jl/stable/manual/fastq/).

```julia
using FASTX
testfile = "some/path.fastq"

for record in FASTQ.Reader(open(testfile))
    seq = FASTQ.sequence(record)
    # ...
end

```

These files are often quite large (I’m currently working with several hundred files that are ~500Mb each), but usually come gzipped (and are ~10x smaller). Unfortunately, the FASTQ reader from FASTX doesn’t seem to be able to take the stream from GZip.jl:

```julia
testfilegz = "some/path.fastq.gz"

for record in FASTQ.Reader(GZip.open(testfilegz))
    seq = FASTQ.sequence(record)
    # ...
end

```

gives:

```julia
ERROR: LoadError: MethodError: no method matching isopen(::GZipStream)
Closest candidates are:
  isopen(::Mmap.Anonymous) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:41
  isopen(::Base.Filesystem.File) at filesystem.jl:94
  isopen(::Base.BufferStream) at stream.jl:1217
  ...

```

> **full stacktrace**
>
> ```julia
> 
> Stacktrace:
> [1] TranscodingStreams.TranscodingStream{TranscodingStreams.Noop,GZipStream}(::TranscodingStreams.Noop, ::GZipStream, ::TranscodingStreams.State, ::Bool) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:25
> [2] TranscodingStreams.TranscodingStream(::TranscodingStreams.Noop, ::GZipStream, ::TranscodingStreams.State; initialized::Bool) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:39
> [3] TranscodingStreams.TranscodingStream(::TranscodingStreams.Noop, ::GZipStream, ::TranscodingStreams.State) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:39
> [4] TranscodingStreams.TranscodingStream(::TranscodingStreams.Noop, ::GZipStream; bufsize::Int64, sharedbuf::Bool) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/noop.jl:41
> [5] TranscodingStreams.TranscodingStream(::TranscodingStreams.Noop, ::GZipStream) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/noop.jl:34
> [6] TranscodingStreams.TranscodingStream{TranscodingStreams.Noop,S} where S<:IO(::GZipStream; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/noop.jl:28
> [7] TranscodingStreams.TranscodingStream{TranscodingStreams.Noop,S} where S<:IO(::GZipStream) at /home/kevin/.julia/packages/TranscodingStreams/MsN8d/src/noop.jl:28
> [8] FASTX.FASTQ.Reader(::GZipStream; fill_ambiguous::Nothing) at /home/kevin/.julia/packages/FASTX/wcfDB/src/fastq/reader.jl:25
> [9] FASTX.FASTQ.Reader(::GZipStream) at /home/kevin/.julia/packages/FASTX/wcfDB/src/fastq/reader.jl:19
> [10] top-level scope at /augusta/students/danielle/resampling/subsample.jl:33
> 
> ```

Based on the stacktrace, it seems to be an issue with `TranscodingStreams.jl` which FASTQ uses under the hood, but I’m not familiar enough with the I/O packages to know for sure. Where should this method be added and would it be straightforward to add? Both packages are pretty lightweight on their own, and I’m guessing neither would want to take the dependency on the other.

Or is there a way to do this in FASTX.jl without committing type piracy?

---

<div class="post-metadata">

**Author:** ![js135005](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/js135005/32/8219_2.png) [@js135005](https://discourse.julialang.org/u/js135005)\
**Post date:** [March 20, 2020, 3:13pm UTC](https://discourse.julialang.org/t/streaming-gziped-file-to-fastq-reader-where-to-add-method/36266/2 "2020-03-20T15:13:33Z")

</div>

Have you tried CodecZlib? I use it all the time to read gzip files directly as if I were reading an uncompressed file.

A side benefit is that it is the fastest reader I’ve found for gzip compressed files.

Oops, may not work for you as it is also based on TranscodingStreams. Would still be worth trying it.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [March 20, 2020, 3:21pm UTC](https://discourse.julialang.org/t/streaming-gziped-file-to-fastq-reader-where-to-add-method/36266/3 "2020-03-20T15:21:15Z")

</div>

> [@js135005](#):
>
> Have you tried CodecZlib?

Nope! Didn’t know about it - thanks!

```julia
for record in FASTQ.Reader(GzipDecompressorStream(open(testfilegz)))
    #...
end

```

works 😃