# \[ANN\] StreamSampling.jl - Sampling methods for data streams

**URL:** <https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779>\
**Category:** Package Announcements\
**Tags:** package, announcement, sampling\
**Created:** [January 13, 2024, 7:40pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779 "2024-01-13T19:40:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [January 13, 2024, 7:40pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/1 "2024-01-13T19:40:02Z")

</div>

Hi all!

I’m happy to announce the new package [IteratorSampling.jl](https://github.com/Tortar/IteratorSampling.jl), which can help to sample from arbitrary iterables in a single pass through the data. It is mainly based on the theory of reservoir sampling methods.

Since reservoir sampling doesn’t need to collect the iterable in memory, it is faster than using `StatsBase.sample` in some cases as you can see in the (contrived) example in the [ReadMe](https://github.com/Tortar/IteratorSampling.jl/blob/main/README.md) of the package; this means it can also be useful in some dynamic simulation scenarios where one can’t assume much about the population to sample and so scanning all the iterable is needed.

I plan to add some more features in the future such as weighted sampling methods and the possibility to resume the sampling preserving the unbiasedness of the sample. If you have suggestions on some more features, they are really welcomed!

For those interested, the package is already available in the general registry 🙂

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [January 15, 2024, 9:08am UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/2 "2024-01-15T09:08:21Z")

</div>

Worth noting that OnlineStats.jl also has [reservoir sampling](https://joshday.github.io/OnlineStats.jl/latest/api/#OnlineStats.ReservoirSample).

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [January 15, 2024, 11:12am UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/3 "2024-01-15T11:12:58Z")

</div>

Thanks @cydoris! didn’t find it when I searched in the Julia ecosystem!

I did a little benchmark to compare the implementation performance:

```julia
julia> using IteratorSampling, OnlineStats, BenchmarkTools

julia> iter = 1:10^7; # length is known

julia> @btime fit!(ReservoirSample(10^4, Int), $iter);
  29.043 ms (3 allocations: 78.28 KiB)

julia> @btime itsample($iter, 10^4);
  2.760 ms (14 allocations: 381.29 KiB)

julia> iter = Iterators.filter(x -> x != 10, 1:10^7); # length is not known

julia> @btime fit!(ReservoirSample(10^4, Int), $iter);
  33.474 ms (3 allocations: 78.28 KiB)

julia> @btime itsample($iter, 10^4);
  7.931 ms (2 allocations: 78.17 KiB)

```

the speed difference is due to the usage of a different algorithm when the length is known and the use of an optimized implementation for reservoir sampling (Vitter’s algorithm L) when it is not, see the docs [API · IteratorSampling.jl](https://tortar.github.io/IteratorSampling.jl/dev/) for a little more details.

I noticed also that the implementation there covers only unweighted sampling without replacement, in IteratorSampling you have a “general” function `itsample([rng], iter, n::Int; replace = false, ordered = false)` which mimics `StatsBase.sample` and (hopefully in the future) weighted sampling available and some more features useful in stream sampling.

I will ask the authors if they would like to have IteratorSampling.jl as a dependency in the future 🙂

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [January 15, 2024, 2:59pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/4 "2024-01-15T14:59:49Z")

</div>

> [@Tortar](#):
>
> If you have suggestions on some more features, they are really welcomed!

A thing I have repeatedly needed and never found a good library for is multi-threading support, i.e. the items are produced / handled in multiple threads (which can be created and destroyed during the lifetime of the reservoir).

One would need support for splitting off from a reservoir, and support for explicitly rejoining a split-off reservoir, i.e. no synchronization at all on consuming a single sample. The meat of code for that feature would be the code for joining reservoirs.

An example use would be allocation profiling: Typical allocations only pay a thread/core-local increment and well-predicted branch, and spinning up or down threads involves locks anyways. This is cheap enough to be basically free.

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [April 20, 2024, 9:22pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/5 "2024-04-20T21:22:10Z")

</div>

I’m happy to announce version 0.3 of the package!

Now the package has all the basic building blocks I wanted to add when I conceived it.

Apart from the already available `itsample`, it now allows to control the sampling process from the “outside” by instantiating a `ReservoirSample` which then can be updated with the `update!` function:

```julia
julia> using StreamSampling

julia> rs = ReservoirSample(Int, 5);

julia> for x in 1:100
           update!(rs, x)
       end

julia> value(rs)
5-element Vector{Int64}:
  7
  9
 20
 49
 74

```

Also, new weighted sampling algorithms were added both for sampling with and without replacement, which means that all classical sampling procedures are now implemented for arbitrary data streams.

It would be even possible to integrate `itsample` in `StatsBase` for discoverability purposes, the `StatsBase.sample` function could be then used on any kind of iterable, not only on `AbstractArray`, but this would mean that the user needs to be a bit more careful because collecting the iterators is usually more performant if one needs multiple samples from it. If anyone has thoughts on this please let me know!

I also made a little benchmark comparison with `StatsBase.sample` on a simple iterator:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/6/d6a78f9fec4004d9dc37ff2a53da697944478eaf.png)

(where “collection-based with setup” means that collecting the iterator in memory is considered part of the benchmark)

* * *

Importantly, I **renamed** the package since I thought that StreamSampling.jl was more appropriate. I can’t anymore edit comments referring to the previous name, hopefully it shouldn’t cause too much confusion.

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [October 7, 2024, 3:38pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/6 "2024-10-07T15:38:07Z")

</div>

Some updates for v0.5.0:

- The library now supports multithreaded merging of reservoirs as @foobar_lv2 suggested with `merge`/`merge!` and it allows to creates a single sample from multiple iterables in parallel with the `itsample` function. It works only with (weighted) sampling with replacement though.
- A little illustrative example is now present in the docs: [An Illustrative Example · StreamSampling.jl](https://juliadynamics.github.io/StreamSampling.jl/stable/) and some more benchmarks of the sampling procedures are at [StreamSampling.jl/benchmark at main · JuliaDynamics/StreamSampling.jl · GitHub](https://github.com/JuliaDynamics/StreamSampling.jl/tree/main/benchmark).
- The API is now in line with the one of OnlineStatsBase.jl and the internals are much more polished.

Let me know if you have some suggestions for improvements, I was mostly focused on reservoir sampling techniques, but I think I could maybe enlarge a bit more the scope of the library.

Interestingly it is actually one of the few open source libraries which tries to implement these techniques, I only found [GitHub - bigmlcom/sampling: Random Sampling in Clojure](https://github.com/bigmlcom/sampling) as a similar attempt.

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [October 11, 2024, 3:38pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/7 "2024-10-11T15:38:48Z")

</div>

I released a new version which tweaks a bit the API (v0.6), at the same time I implemented two new (but old 😃) algorithms for stream sampling without a reservoir, so that they require O(1) memory for unweighted sampling with and without replacement. They are algorithm D by Vitter and algorithm 4 by Bentley, all brought back from research in the eighties.

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [August 14, 2025, 11:32am UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/8 "2025-08-14T11:32:05Z")

</div>

## StreamSampling.jl v0.7

New major version landed with some important updates:

- The merging API is almost complete, it is now possible to merge multiple samplers together to obtain a unique summary of multiple streams for almost all algorithms. This is very helpful for parallel sampling.
- A new example about sampling from on-disk data. See [StreamSampling.jl/stable/example/#Sampling-from-Data-on-Disk](https://juliadynamics.github.io/StreamSampling.jl/stable/example/#Sampling-from-Data-on-Disk). We achieve with ease a sampling performance of 500MB/s with 4 threads with HDF5.jl and 2GB/s with Arrow.jl! The most interesting aspect is that the sampling algorithm becomes the bottleneck in the sampling process, more than loading data from disk (at least with an SSD).
- Many general performance and interface improvements.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 15, 2025, 2:56am UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/9 "2025-08-15T02:56:20Z")

</div>

> [@Tortar](#):
>
> 2GB/s with [Arrow.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/Arrow)!

given this is sequential read from disk and Arrow in theory is very performant, I wonder what does this represent in terms of the % of max disk I/O?

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [August 15, 2025, 1:26pm UTC](https://discourse.julialang.org/t/ann-streamsampling-jl-sampling-methods-for-data-streams/108779/10 "2025-08-15T13:26:20Z")

</div>

My SSD should has a peak sequential read performance of 3.18GB/s based on specs, I now tried sampling a 100GB arrow file with 6 threads and it took around 40 seconds, so 2.5GB/s (which is also similar to the read speed I see in the System Monitor during sampling). So the % of max disk I/O is around 80%.