Hi there, I am new to Julia, and come to Julia for performance with fewer codes. I have been using R, and encountered a simple task: read from a gziped file and do something to each line. It was slow in R, of course. So I went to Perl first. Here is the code: perl -e ' open( DATA, "zcat ~/in2/swxx…

How does the timed performance compare? I’m not sure it will make much difference in this case, but it is generally bad for performance to refer to non-const global variables inside a function. Also, I would recommend using CodecZlib.jl over shell commands.

I will later record the times spent. But I have the impression that the python code is obviously faster than the other two. For the julia code, there is only one global variable, which was not used repeatedly. So I guess it would not affect the performance much.

The issue with non const global variables is in how they affect type inference. Basically it and every value derived from that are a type which cannot be known at compile time which results in slower code. If there are still speed differences after that it could be due to whether println buffers IO…

This could also be related to the fact that println is slower than Python’s print in some terminals: https://github.com/JuliaLang/julia/issues/36639 Do your results change if you skip the print statements?

I see. I have modified the code, but noticed no improvement in performance.

I later skipped printing, but noticed no improvement in performance - gzip was using ~45% CPU, and Julia 100%.

Yea why are you calling externally here and not using a Julia package? I read in large *.csv.gz files via streaming using CSV.jl and CodecZLib.jl and it is very fast.

Is this code appropriate in using CodecZlib? It is even more slow… using CSV, CodecZlib, Mmap, TranscodingStreams function fun1( file1) io1 = TranscodingStream( GzipDecompressor(), open( expanduser( file1))) N = 10 n1 = 0 for line1 in eachline( io1) F = split( line1, '\t') …

[image] calvin: "zcat ~/in2/swxx/sj/db/ncbi/gene/gene2accession.gz|" just a side note - with a multi core cpu : the "unpigz -c sample.gz | .... " is ~ 2x faster than a simple zcat sample.gz | ... https://unix.stackexchange.com/questions/363644/fastest-and-most-efficient-way-to-get-numb…

I know. However, gzip is not the bottleneck here.

Slow gzip streaming in julia but not in python

General Usage

Topic		Replies	Views
Speed comparison on reading a gzip file General Usage performance	8	1578	December 10, 2023
GzipDecompressionStream compared to GZip.jl? General Usage	2	1458	August 22, 2017
[ANN] LibDeflate.jl and CodecBGZF.jl - really fast blocked de/compression Package Announcements	9	1585	March 17, 2021
Julia1.0 linux GZip.jl cannout find libz.so New to Julia package	3	1148	September 2, 2018
Read/write compressed files in julia 0.7 General Usage	3	1169	June 13, 2019

Slow gzip streaming in julia but not in python

Related topics