# Performance of splitting string and parsing numbers

**URL:** <https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161>\
**Category:** Performance\
**Created:** [December 26, 2022, 7:19pm UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161 "2022-12-26T19:19:58Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 27, 2022, 11:20pm UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/22 "2022-12-27T23:20:17Z")

</div>

After seeing this several times, I’m no longer surprised. It’s important to note that this is perfectly idiomatic rust, and it uses only the nice “functional” abstractions. And it is only two times slower than the best (maybe ?) hand-tuned Julia. Of course, you could write a hand-tuned version in rust (wonder what that would look like). I think this is a big challenge for Julia. It might be difficult, given the the dynamic semantics of Julia. But my guess is that the latter isn’t really the problem here.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [December 28, 2022, 2:38am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/23 "2022-12-28T02:38:44Z")

</div>

String manipulation is also not an area where Julia exactly shines at the moment performance-wise.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [December 28, 2022, 3:31am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/24 "2022-12-28T03:31:29Z")

</div>

Okay, now this version is really high-level:

```julia
import Base.Iterators as Itr
using IterTools

newsolve(data) = begin
  itr = Itr.Stateful(Itr.map(s->something(tryparse(Int, s), -1), eachsplit(data, '\n')))
  maximum(Itr.takewhile(>(0), repeatedly(()->sum(Itr.takewhile(!=(-1), itr); init=0))))
end

```

and it actually works:

```julia
julia> data = "11\n21\n\n3\n34\n5\n\n16\n17\n10"
"11\n21\n\n3\n34\n5\n\n16\n17\n10"

julia> bigdata=(data*"\n\n")^255;

julia> newsolve(bigdata)
43

```

Performance-wise it is totally competitive. The problem with performance with the initial `solve` function was `eachsplit(..., "\n\n")`. The `eachsplit` isn’t optimized for long patterns and needs some optimization work.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 28, 2022, 3:43am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/25 "2022-12-28T03:43:08Z")

</div>

My test shows this is not very competitive on the data set from the OP.

```julia
 @btime newsolve($s)
  131.668 μs (3 allocations: 144 bytes)

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [December 28, 2022, 4:00am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/26 "2022-12-28T04:00:45Z")

</div>

Can you add some other baselines? What is the fastest Julia version now?  
(because of machine differences, a single benchmark is usually uninformative)

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 28, 2022, 4:16am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/27 "2022-12-28T04:16:51Z")

</div>

> [@Dan](#):
>
> Can you add some other baselines?

Yes. These are taken from previous posts. (I added `@inline` annotations to the one i posted)

```julia
julia> @btime solve($s)
  197.365 μs (2263 allocations: 141.36 KiB)
75501

julia> @btime solve2($s)
  148.081 μs (4520 allocations: 335.45 KiB)
75501

# The following reported 12.6 μs on one btime (!)
julia> @btime hilox($s)
  20.572 μs (0 allocations: 0 bytes)
75501

julia> @btime max_num($s)
  22.548 μs (0 allocations: 0 bytes)
75501

julia> @btime solvelox($s)
  16.187 μs (0 allocations: 0 bytes)
75501

```

---

<div class="post-metadata">

**Author:** ![AMJ](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amj/32/214096_2.png) [@AMJ](https://discourse.julialang.org/u/AMJ)\
**Post date:** [December 28, 2022, 9:35am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/28 "2022-12-28T09:35:02Z")

</div>

Why didn’t you use `codeunits()` instead of `transcode()` ? I am not sure but when I benchmarked the two functions `codeunits()` were faster (few tens/hundreds ns faster, wouldn’t make tangible difference here).

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 28, 2022, 10:03am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/29 "2022-12-28T10:03:04Z")

</div>

because looking for some function that transforms characters into numbers, I found transcode first.  
I saw that it took less than 1us to do the operation and therefore I concentrated on finding an algorithm that did everything needed in one step.  
Indeed codeunits is 10 times faster than transcode, but this weighs little on the final result:

```julia
julia> @btime transcode(UInt8,bigdata);
  222.391 ns (1 allocation: 16 bytes)

julia> 

julia> @btime codeunits(bigdata);
  21.910 ns (1 allocation: 16 bytes)

julia> function solvelox(data)
           v=codeunits(data)
           largest = total = totpar= 0
           pre_nl=true
           for e in v
               if (e == 0x0a) && pre_nl
                   largest = max(largest, total)
                   total = totpar= 0
               elseif (e == 0x0a) && !pre_nl
                   total=total+totpar
                   pre_nl=true
                   totpar=0
               else
                   totpar = totpar*10+e-0x30
                   pre_nl=false
               end

           end
           return max(largest, total+totpar)
       end
solvelox (generic function with 1 method)

julia> @btime solvelox($bigdata)
  15.100 μs (0 allocations: 0 bytes)
75501

julia> @btime max_num($bigdata)
  28.700 μs (0 allocations: 0 bytes)
75501

```

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [December 29, 2022, 3:22am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/30 "2022-12-29T03:22:20Z")

</div>

> [@giordano](#):
>
> I believe so, you’re missing `evals=1` to the benchmark settings:

You’re right. I had forgotten that BenchmarkTools runs multiple evaluations per sample.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [December 29, 2022, 7:56am UTC](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161/31 "2022-12-29T07:56:05Z")

</div>

I also tried to use the functions of IterTools, but tests on pieces of code did not encourage me to continue

```julia
v = transcode(UInt8,data)

using IterTools

p=partition(v,2,1)

ps=Iterators.Stateful(p)

Iterators.reset!(ps)

@time reduce((s,c)->s*10+first(c)-0x30 ,takewhile(t->first(t)!=0x0a,takewhile(!=((0x0a, 0x0a)), ps)),init=0)

@time takewhile(t->first(t)!=0x0a,takewhile(!=((0x0a, 0x0a)), ps))

```

although I enjoyed doing something like this …

```julia
bigdata=codeunits(data)
blk=groupby(!=[0x0a,0x0a]),groupby(!=(0x0a),bigdata))
dig2num=sum(vofv)sum(v->reduce((s,c)->c==0x0a ? s : s*10+c-0x30, v, init=0), vofv)
maximum(dig2num,blk)

```

[Previous page](https://discourse.julialang.org/t/performance-of-splitting-string-and-parsing-numbers/92161.md?page=1)
