There’s been a challenge in Java community that generated a considerable amount of interest I’d say: The One Billion Row Challenge “A fun exploration of how quickly 1B rows from a text file can be aggregated with Java”. While evaluation part of the challenge is not available to other languages, ther…

I feel like these io-bound problems always end up boiling down to OS wizardry because defaults for page sizes or whatever are suboptimal and can be tweaked for the problem at hand.

it actually stops being an io-bound problem pretty soon as you dig into it imho

Yeah, that’s the part confusing me 13GB in 300ms would mean ~40GB/s read speeds, which does not sound plausible. Can someone explain what is the trick here? I looked through the top two solutions yesterday, but since in Java and super hackish, I did not understand one bit.

Nice. I did tweak your code to run it with 8 threads on my laptop… and while obviously tests don’t pass (e.g. expected formatting is off), it ran in about 29sec for me: Zanzibar City=26.002252093912084/-25.8/78.0, Zürich=9.305815037868426/-38.6/59.6, Ürümqi=7.394862869786727/-41.2/54.3, İzmir=17.…

I think the main thing, besides all the bit-masking-jitsu, is direct memory access usng things like java.lang.foreign.Arena sun.misc.Unsafe you can see in the top submission. The first one is part of a preview/new API that controls the “lifecycle of native memory segments, providing both flexib…

I think there was another thread on this challenge but I can’t seem to find it?

What does direct memory access mean in this context? The file is located on an SSD so you can’t surpass SSD read speed, even if you memory map the SSD into your memory address space as if it was RAM.

Just to note that for temperatures -Inf =-273,15 [°C] :smile: More seriously to ask how to recover billion row file

ah, well it may indeed be something Java specific, where you don’t normally go out and grab some native memory, hence names like “Unsafe” and “foreign”. And challenge evaluation rules actually state: “Programs are run from a RAM disk (i.o. the IO overhead for loading the file from disk is not releva…

If you want to generate 1B file, take a look at 1brc/prepare_3j5a_julia.sh at main · 3j5a/1brc · GitHub it should (:crossed_fingers: ) help installing Java and building the project, then ./create_measurements.sh 1000000000

The One Billion Row Challenge

General Usage Performance

rdavis120 February 1, 2024, 11:15pm 24

From the prior thread on this, the baseline could be the duckdb entry loaded into a DataFrame. Preferably with the latest dev version of duckdb because the version on the registry had an issue previously.

Topic		Replies	Views
How to optimise and be faster than Java? Performance	24	2198	September 4, 2018
Julia programs now shown on benchmarks game website Community announcement	144	14673	December 3, 2019
Julia needs work at "Benchmark Game": numbers seem off up to 100x slower, maybe "Julia AOT" entries needed? Community	10	1915	November 23, 2018
At a glance, why don't I have better performances between Julia and PHP? Performance	24	1923	July 30, 2020
Help to get my slow Julia code to run as fast as Rust/Java/Lisp Performance	100	5317	August 6, 2021

The One Billion Row Challenge

Related topics