### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 20, 2021, 9:20pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/1 "2021-02-20T21:20:20Z")
Hi
I am trying to use Julia for general purpose programming.
I am trying to use Julia to create a faster version of a Perl script, which is supposed to create a report on a large logfile, piped into STDIN. The logfile does not fit into RAM, hence the data needs to be read from STDIN line-by-line from the data stream.
My problem is that Julia (1.5.3) is twice as slow as Perl.
I was able to identify one of the main problems, which is the code to read line-by-line from the input stream.
I use the following test code only to illustrate the issue.
**Perl:**
```julia
#!/bin/perl -w
use strict;
use warnings;
my $counter = 0;
while(<>) {
$counter++;
}
print "Perl: Number of lines: $counter\n";
```
**Julia:**
```julia
#!/bin/julia
counter = 0
for line = eachline()
global counter += 1
end
println("Julia: Number of lines: $counter")
```
**Performance Perl:**
```julia
# time (cat access.log | ./testloopspeed.pl)
Perl: Number of lines: 19567100
real 0m15.251s
user 0m10.475s
sys 0m8.209s
```
**Performance Julia:**
```julia
# time (cat access.log | ./testloopspeed.jl)
Julia: Number of lines: 19557160
real 0m36.974s
user 0m27.348s
sys 0m6.368s
```
During runtime of above test code, I noticed that Perl is using much less RAM.
| Language | Resident-Memory | Virtual-Memory |
| --- | --- | --- |
| Perl | 2,4MB | 127MB |
| Julia | 185MB | 686MB |
What am I doing wrong?
Please notice, that there is another topic on a very similar subject here:
> [@Read large stream from STDIN](https://discourse.julialang.org/t/read-large-stream-from-stdin/55627):
>
> Hi I would like to use Julia to process large logfiles piped into STDIN, in order to create reports. The amount of data is too big to fit into RAM, so I need to process every line by line while reading the stream of data (rather than slurping all data into a variable before starting to process data). Like this: cat my\_huge\_logfile.log | reporting.jl In Perl, I would use something like this: while(\) { # regex matching on current line # do some preprocessing # remember sel…
Regards
Toni
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [February 20, 2021, 9:35pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/2 "2021-02-20T21:35:34Z")
First thing, never write code directly on the global scope, write inside a function (even if it is a `main` taking no arguments and returning nothing) and call the function. Do not use global variables if possible (use variables local to this function instead). For things that will run a single time, and that compilation time may be the culprit try calling `julia` passing the `--compile=min` flag.
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 20, 2021, 9:53pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/3 "2021-02-20T21:53:02Z")
Hello @Henrique_Becker
Thanks for your comments.
Do you think you general recommendations would make the Julia code run twice as fast?
Another question from me would be, how I would access STDIN from within a function? STDIN is a global handler, how would I pass a reference to it as argument to a function?
Thanks, Toni
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 20, 2021, 10:50pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/4 "2021-02-20T22:50:30Z")
in this case, `--compile=min` only makes things slower:
```julia
~ » time (cat blah.log | julia16 --startup-file=no blah.jl)
Julia: Number of lines: 100000
( cat blah.log | julia16 --startup-file=no blah.jl; ) 0.25s user 0.44s system 355% cpu 0.195 total
------------------------------------------------------------------------------------------------------------------------
~ » time (cat blah.log | julia16 --compile=min --startup-file=no blah.jl)
Julia: Number of lines: 100000
( cat blah.log | julia16 --compile=min --startup-file=no blah.jl; ) 4.30s user 0.45s system 111% cpu 4.267 total
```
Perl:
```julia
0.02s user 0.00s system 101% cpu 0.020 total
```
* * *
and putting things into function doesn’t help because you’re just running it once
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [February 20, 2021, 10:57pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/5 "2021-02-20T22:57:01Z")
> [@jling](#):
>
> and putting things into function doesn’t help because you’re just running it once
The objective of putting things inside a function was to remove variables from global scope, so I do not understand you comment.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 20, 2021, 10:58pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/6 "2021-02-20T22:58:24Z")
it doesn’t matter in this case, you will know once you’ve tried it.
```julia
function f()
counter = 0
for line = eachline()
counter += 1
end
counter
end
println("Julia: Number of lines: $(f())")
~ » time (cat blah.log | julia16 --startup-file=no blah.jl)
Julia: Number of lines: 100000
( cat blah.log | julia16 --startup-file=no blah.jl; ) 0.25s user 0.44s system 360% cpu 0.191 total
```
I think in this case `counter` being a global variable only added negligible overhead compare to the really slow `readline()`
---
### Author:  [@pixel27](https://discourse.julialang.org/u/pixel27)
#### Post date: [February 20, 2021, 11:22pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/7 "2021-02-20T23:22:30Z")
I don’t think Julia can perform better with such a small test. There is barely any logic being executed. 99% of the perl code is probably C i.e. not interpreted. With Julia you have the compile time getting in the way so you need a file and logic big enough to dwarf that time.
I used this Julia file:
```julia
using Dates
function from_stdin()
start = now()
counter = 0
for line = eachline()
counter += 1
end
finish = now()
println("STDIN read time ($counter lines): $(finish - start)");
end
from_stdin()
```
With 10 million lines perl is about 1/3 faster:
```julia
$ time ./ptest < 10_000_000.txt
Perl: Number of lines: 10000000
real 0m0.937s
user 0m0.893s
sys 0m0.043s
$ time julia read.jl < 10_000_000.txt
STDIN read time (10000000 lines): 1214 milliseconds
real 0m1.430s
user 0m1.350s
sys 0m0.569s
```
When I go up to 1,000,000,000 lines the differences get narrower:
```julia
$ time julia read.jl < 1_000_000_000.txt
STDIN read time (1000000000 lines): 113656 milliseconds
real 1m53.871s
user 1m44.570s
sys 0m9.574s
$ time ./ptest < 1_000_000_000.txt
Perl: Number of lines: 1000000000
real 1m36.142s
user 1m25.340s
sys 0m10.625s
```
My test files are not that big, the 10,000,000 line file is only 123MiB while the 1,000,000,000 line file is about 14GiB, so both can fit into memory.
Just for grins I moved the 14GiB file into /tmp which is a tmpfs file system (RAM) to remove any disk access times and I get:
```julia
$ time ./ptest < /tmp/1_000_000_000.txt
Perl: Number of lines: 1000000000
real 1m31.380s
user 1m28.799s
sys 0m2.390s
$ time julia read.jl < /tmp/1_000_000_000.txt
STDIN read time (1000000000 lines): 108426 milliseconds
real 1m50.210s
user 1m45.692s
sys 0m3.708s
```
```julia
```
---
### Author:  [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [February 20, 2021, 11:26pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/8 "2021-02-20T23:26:12Z")
> [@tferic](#):
>
> Do you think you general recommendations would make the Julia code run twice as fast?
No. However, I do not seem why to not use them either. It is easier to just use them from start than benchmarking the difference they make.
> [@tferic](#):
>
> how I would access STDIN from within a function? STDIN is a global handler,
You can just pass `stdin` to any `function`, as a parameter, in my suggestion (creating a `main` function) you could pass it to `main` in the call. However, this will problaby make little difference, as pointed out by @jling, and I was interested in making `counter` a `local` variable, I was not even thinking about `stdin`.
---
### Author:  [@simeonschaub](https://discourse.julialang.org/u/simeonschaub)
#### Post date: [February 20, 2021, 11:26pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/9 "2021-02-20T23:26:47Z")
`eachline` will collect each line into a string, which can be expensive if you only want the number of lines. Try this instead:
```julia
let counter = 0
while !eof(stdin)
counter += read(stdin, Char) == '\n'
end
println("Julia: Number of lines: $counter")
end
```
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 20, 2021, 11:38pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/10 "2021-02-20T23:38:04Z")
> [@simeonschaub](#):
>
> if you only want the number of lines.
No, I actually really need to perform some logic and use regex to extract substrings.
I tried running the full program with all logic in Julia, and it was twice as slow as Perl.
Then I tried to find the reason, and I suspect the reason for Julia being so slow is in `eachline()` itself.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 20, 2021, 11:43pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/11 "2021-02-20T23:43:05Z")
> [@tferic](#):
>
> some logic and use regex to extract substrings
apparently “some logic” is too light to make a difference.
regex is not something Julia can shine in doing, we just call [http://www.pcre.org/](http://www.pcre.org/) so… again, Perl probably is just running C code here which is already optimized. This is similar to if you benchmark some linear algebra code, everyone will just be calling OpenBLAS/MLK
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 20, 2021, 11:46pm UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/12 "2021-02-20T23:46:05Z")
> [@pixel27](#):
>
> I don’t think Julia can perform better with such a small test. There is barely any logic being executed.
I initially had ran the Julia program with all logic in it, so that it would extract substrings, store them in Dictionaries, and at the end the reporting.
As a result, Julia was twice as slow as Perl. I investigated by reducing more and more logic, and came to the conclusion, that it must be `eachline()` that’s so slow.
> [@pixel27](#):
>
> the 10,000,000 line file is only 123MiB while the 1,000,000,000 line file is about 14GiB, so both can fit into memory.
14GB won’t fit into my memory. But why would anyone even try to do this, when it can be done cheaper by stream-processing? We’ve done that 20 years ago with `awk`.
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 21, 2021, 12:10am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/13 "2021-02-21T00:10:08Z")
> [@jling](#):
>
> > [@tferic](#):
> >
> > some logic and use regex to extract substrings
>
> apparently “some logic” is too light to make a difference.
>
> regex is not something Julia can shine in doing, we just call [http://www.pcre.org/](http://www.pcre.org/) so… again, Perl probably is just running C code here which is already optimized. This is similar to if you benchmark some linear algebra code, everyone will just be calling OpenBLAS/MLK
Are you implicitly saying that Julia is not such a good fit for general purpose programming? I struggle to believe that.
I would rather suspect that `eachline()` just isn’t as optimized as it should be. I don’t see a reason why JIT compiled code would run significantly slower than statically compiled code.
In my case, the difference between Perl and Julia is 20s. (Same logic, same structure) That difference cannot be explained by the initial pre-delay of JIT compilation.
---
### Author:  [@pixel27](https://discourse.julialang.org/u/pixel27)
#### Post date: [February 21, 2021, 12:11am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/14 "2021-02-21T00:11:15Z")
> [@tferic](#):
>
> But why would anyone even try to do this, when it can be done cheaper by stream-processing? We’ve done that 20 years ago with `awk` .
I wouldn’t, I’d used `wc -l < log.txt`. But you were the one complaining about `eachline()` being “slow”, so I’m confused now.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 21, 2021, 12:17am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/15 "2021-02-21T00:17:01Z")
> [@tferic](#):
>
> Are you implicitly saying that Julia is not such a good fit for general purpose programming?
I’m not, I’m saying “run regex for each line of a big text file” is not exactly some high-level complex use-case calling for a general programming language, you can use awk grep some GNU/POSIX utility to do that.
Julia can do all of these just fine of course, and there’s certainly room for improvement in I/O.
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 21, 2021, 12:17am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/16 "2021-02-21T00:17:06Z")
> [@pixel27](#):
>
> But you were the one complaining about `eachline()` being “slow”, so I’m confused now.
I am trying to use Julia’s `eachline()` in a way one would use `awk`. I would not expect Julia to be significantly slower than any other language, including compiled C.
---
### Author:  [@pixel27](https://discourse.julialang.org/u/pixel27)
#### Post date: [February 21, 2021, 1:01am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/17 "2021-02-21T01:01:31Z")
> [@tferic](#):
>
> I am trying to use Julia’s `eachline()` in a way one would use `awk` . I would not expect Julia to be significantly slower than any other language, including compiled C.
I’m not sure I’d say significant. I suspect perl is doing some trickery since you are not using the line being read. Try these two programs they both report their speed in seconds so hopefully comparing apples to apples:
Perl:
```julia
#!/bin/perl -w
use strict;
use warnings;
use utf8;
use Time::HiRes qw(time);
my $start = time();
my $counter = 0;
while(<>) {
$counter = $counter + length($_);
}
my $total = time() - $start;
print "STDIN read time ($counter chars): $total\n";
```
Julia:
```julia
using Dates
function from_stdin()
start = now()
counter = 0
for line = eachline(;keep=true)
counter += length(line)
end
total = Dates.value(now() - start)/1000
println("STDIN read time ($counter chars): $total");
end
from_stdin()
```
For me on a 100 million line file I get:
```julia
$ ./ptest < 100_000_000.txt
STDIN read time (1388888898 chars): 11.8912749290466
$ julia test.jl < 100_000_000.txt
STDIN read time (1388888898 lines): 12.91
```
So a difference of 1 second over a runtime of 12 seconds. Perl still runs faster on this small test, no clue where the overhead is. I suspect Julia might be copying bytes out of the buffer when it converts them to a string, and Perl might not be, but that’s just a wild guess.
---
### Author:  [@tferic](https://discourse.julialang.org/u/tferic)
#### Post date: [February 21, 2021, 1:58am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/18 "2021-02-21T01:58:42Z")
@pixel27 Thanks for taking the time.
I used these exact code snippets you suggested, just added a shebang to the Julia code at the beginning.
I can still see a significant difference:
```julia
$ cat access.log | ./testloopspeed2.pl
STDIN read time (3013196840 chars): 20.3576831817627
$ cat access.log | ./testloopspeed2.jl
STDIN read time (3011271087 chars): 39.258
```
I am using a real logfile in this test.
My version of Julia is 1.5.3.
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [February 21, 2021, 3:54am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/19 "2021-02-21T03:54:11Z")
So what is your operating system, RAM, CPU, hard drive, perl version? Did you try Julia 1.6rc1?
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 21, 2021, 4:17am UTC](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701/20 "2021-02-21T04:17:02Z")
why does it matter? they have test against Perl on the same system
[Next page](https://discourse.julialang.org/t/bad-performance-of-eachline-on-stdin/55701.md?page=2)