Streaming gziped file to FASTQ.Reader - where to add method?

kevbonham
Fri, 20 Mar 2020 15:01:59 +0000
General Usage

FASTQ is a file format used for DNA sequencing, and we have a handy reader in the FASTX package.

using FASTX
testfile = "some/path.fastq"

for record in FASTQ.Reader(open(testfile))
    seq = FASTQ.sequence(record)
    # ...
end

These files are often quite large (I’m currently working with several hundred files that are ~500Mb each), but usually come gzipped (and are ~10x smaller). Unfortunately, the FASTQ reader from FASTX doesn’t seem to be able to take the stream from GZip.jl:

testfilegz = "some/path.fastq.gz"

for record in FASTQ.Reader(GZip.open(testfilegz))
    seq = FASTQ.sequence(record)
    # ...
end

gives:

ERROR: LoadError: MethodError: no method matching isopen(::GZipStream)
Closest candidates are:
  isopen(::Mmap.Anonymous) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Mmap/src/Mmap.jl:41
  isopen(::Base.Filesystem.File) at filesystem.jl:94
  isopen(::Base.BufferStream) at stream.jl:1217
  ...

full stacktrace (click for more details)
Based on the stacktrace, it seems to be an issue with TranscodingStreams.jl which FASTQ uses under the hood, but I’m not familiar enough with the I/O packages to know for sure. Where should this method be added and would it be straightforward to add? Both packages are pretty lightweight on their own, and I’m guessing neither would want to take the dependency on the other.

Or is there a way to do this in FASTX.jl without committing type piracy?

3 posts - 2 participants