# The One Billion Row Challenge

**URL:** <https://discourse.julialang.org/t/the-one-billion-row-challenge/109534>\
**Category:** Performance\
**Created:** [January 31, 2024, 10:00pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534 "2024-01-31T22:00:53Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 1, 2024, 4:25pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/21 "2024-02-01T16:25:32Z")

</div>

Oh the file itself is loaded into RAM? That will definitely change my timings.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [February 1, 2024, 4:51pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/22 "2024-02-01T16:51:50Z")

</div>

> [@3j5a](#):
>
> ```julia
> > Abéché1️⃣🐝🏎️;27.3;27.3;27.3
> > Almaty1️⃣🐝🏎️;15.3;15.3;15.3
> 
> ```

yep, standard string manipulation error. `@view` not to blame.  
Fix by using:

```julia
station = @view(measurement[1:prevind(measurement,pos)])

```

Using this `@view` approach should improve speed.

---

<div class="post-metadata">

**Author:** ![3j5a](https://avatars.discourse-cdn.com/v4/letter/3/ec9cab/32.png) [@3j5a](https://discourse.julialang.org/u/3j5a)\
**Post date:** [February 1, 2024, 9:06pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/23 "2024-02-01T21:06:52Z")

</div>

_prevind_ 🤦 I should have figure it out. thnx.  
now, don’t think it can go any “baseline” than this then and it runs in about 4min 47sec on my laptop.

```julia
ndé=-24.3/23.8/76.3, Yellowknife=-54.2/-4.3/43.7, Yerevan=-43.4/12.4/61.4, Yinchuan=-41.3/9.0/61.1, Zagreb=-38.3/10.7/62.9, Zanzibar City=-21.8/26.0/75.3, Zürich=-38.6/9.3/61.5, Ürümqi=-43.0/7.4/58.1, İzmir=-34.3/17.9/70.2}
./calculate_average_3j5a_julia.sh 281.46s user 5.44s system 100% cpu 4:46.84 total

```

below is full updated version of it:

```julia
mutable struct MeasurementsStats
    min::Float32; max::Float32; sum::Float64; count::Int64
end

roundjava(it, digits) = round(it, RoundNearestTiesUp; digits = digits)
roundjava(it) = roundjava(it, 1)

function outformat((station, stats))
     average = roundjava(10*stats.sum/stats.count)/10
    "$station=$(roundjava(stats.min))/$(roundjava(average))/$(roundjava(stats.max))"
end

function calculate_average(measurements)
    station_measurements = Dict{String,MeasurementsStats}()
    open(measurements, "r") do file
        for measurement in eachline(file)
            pos = findfirst(';', measurement)
            station = @view(measurement[1:prevind(measurement, pos)])
            temperature = parse(Float32, @view(measurement[pos+1:end]))
            stats = get!(station_measurements, station) do
                MeasurementsStats(temperature, temperature, 0, 0)
            end
            stats.min = min(stats.min, temperature)
            stats.max = max(stats.max, temperature)
            stats.sum += temperature
            stats.count += 1
        end
    end
    results = collect(station_measurements)
    sort!(results, by = it -> it[1])
    print("{")
    join(stdout, (outformat(result) for result in results), ", ")
    println("}")
end

calculate_average(isempty(ARGS) ? "./measurements.txt" : ARGS[1])

```

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [February 1, 2024, 11:15pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/24 "2024-02-01T23:15:27Z")

</div>

From the prior thread on this, the baseline could be the [duckdb entry](https://rmoff.net/2024/01/03/1%EF%B8%8F%E2%83%A3%EF%B8%8F-1brc-in-sql-with-duckdb/#optimisations) loaded into a DataFrame. Preferably with the latest dev version of duckdb because the version on the registry had an issue previously.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [February 2, 2024, 12:20am UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/25 "2024-02-02T00:20:32Z")

</div>

The `mutable struct` might cause some allocation slowdown in the big loop. Replacing it with a `Tuple` might help. The following is untested, but should be fixable:

```julia
function calculate_average(measurements)
    station_measurements = Tuple{Float32, Float32, Float64, Int}[]
    # use sizehint! on station_measurements to avoid some allocations?
    station_index = Dict{String,Int}()
    station_count = 0
    open(measurements, "r") do file
        for measurement in eachline(file)
            pos = findfirst(';', measurement)
            station = @view(measurement[1:prevind(measurement, pos)])
            temperature = parse(Float32, @view(measurement[pos+1:end]))
            idx = get!(station_index, station) do
                push!(station_measurements, (temperature, temperature, 0.0, 0))
                station_count += 1
            end
            stats = station_measurements[idx]
            station_measurements[idx] = (min(stats.min, temperature),
                                         max(stats.max, temperature),
                                         stats.sum + temperature,
                                         stats.count + 1)
        end
    end
    print("{")
    for (i,p) in enumerate(sort(stations_measurements))
        print(i > 1 ? ", " : "", outformat(p))
    end
    println("}")
end

```

`outformat` becomes:

```julia
function outformat((station, stats))
    average = roundjava(10*stats[3]/stats[4])/10
    "$station=$(roundjava(stats[1]))/$(roundjava(average))/$(roundjava(stats[2]))"
end

```

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [February 2, 2024, 9:09am UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/26 "2024-02-02T09:09:53Z")

</div>

> [@jules](#):
>
> Oh the file itself is loaded into RAM? That will definitely change my timings.

They explain that in the challenge somewhere. Official measurement is from cold-start of the program, but the actual data file is already in page-cache.

Ideally no IO should hit the bus and the data is processed zero-copy (just mapped).

For julia, you should probably modify the requirements, i.e. report the following performance numbers:

1. Naive (script runs, i.e. needs compilation)
2. The standard julia way. You run the code on a tiny sample datafile in order to get everything compiled, then you time running it on the actual data.
3. With precompilation (i.e. you make a package and precompile, then run the script)
4. With prepared sysimage.

Of these, (1) and (2) are low-effort, and (1) will have the worst timings (startup + compilation included) and (2) will have the best timings (startup + compilation excluded).

The way I understand it, the 1brc benchmark ensures that stuff is in the page-cache by running the entire thing multiple times on a linux machine with ample memory, without any flushing. But afaiu you can also just put your data file into `/tmp` on a machine with swap disabled.

I’d like to note that 1brc invites a complete anti-pattern: If the file is not already loaded into page-cache, then a naive `mmap` on SSD will be worse that `read`, because you can’t feed the ssd read queue unless you massively multi-thread (page-faults block!). Consider MAP\_POPULATE.

An excellent example is [1brc/src/main/java/dev/morling/onebrc/CalculateAverage\_shipilev.java at main · gunnarmorling/1brc · GitHub](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_shipilev.java)

This is by far not the fastest submission to 1brc, but it is well-documented, careful and idiomatic. On modern java, it should work around that issue (it spawns enough tasks to fill the io queue if you are in a post-loom world), and it also takes care to work in settings where the data doesn’t even fit into main memory.

That being said, I have not verified that the shipilev submission actually works well in that setting 😉

---

<div class="post-metadata">

**Author:** ![3j5a](https://avatars.discourse-cdn.com/v4/letter/3/ec9cab/32.png) [@3j5a](https://discourse.julialang.org/u/3j5a)\
**Post date:** [February 2, 2024, 9:14am UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/27 "2024-02-02T09:14:53Z")

</div>

Right, I see what you mean. Mutable struct is an attempt for naive but reasonable first approach to the problem… but maybe nobody in Julia land would start with it anyway? I was looking at the baseline implementation of the challenge, where results aggregator is a mutable struct (well kind of obviously, it’s a class with mutable fields), and results get added to a kind of standard map/dictionary.  
It goes beyond a naive implementation very quickly, so it’s not a real “baseline” to the problem as it’s ridiculously easy to beat it. But I’m already learning things by just starting from it. Thnx!

---

<div class="post-metadata">

**Author:** ![vnegi10](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vnegi10/32/205882_2.png) [@vnegi10](https://discourse.julialang.org/u/vnegi10)\
**Post date:** [May 12, 2024, 3:46pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/28 "2024-05-12T15:46:12Z")

</div>

Hi, I was about to make a post on 1BRC in Julia, and then found this discussion. Have there been any benchmarks shared already?

My [implementation](https://github.com/vnegi10/1brc_julia) makes use of external packages with a best time of 89.459 s. I am sure there’s a lot more room for improvement. It would therefore be great to share ideas/solutions [here.](https://github.com/gunnarmorling/1brc/discussions/748)

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 30, 2024, 10:09am UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/29 "2024-10-30T10:09:04Z")

</div>

> [@kristoffer.carlsson](#):
>
> [https://twitter.com/thomaswue/status/1752817903483724013](https://twitter.com/thomaswue/status/1752817903483724013) suggests they got it down to 300ms in Java.

how fast is their disk… at that point should the benchmark starts with data in RAM?

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [October 30, 2024, 12:58pm UTC](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534/30 "2024-10-30T12:58:21Z")

</div>

> [@jling](#):
>
> how fast is their disk… at that point should the benchmark starts with data in RAM?

The file starts in RAM. I think they don’t force it (e.g. by disabling swap and putting it on `/tmp/`), but it is a natural consequence of methodology, kernel and machine sizing, and it is known to them.

[Previous page](https://discourse.julialang.org/t/the-one-billion-row-challenge/109534.md?page=1)