# Speeding up \`fit(Histogram)\`
**URL:** https://discourse.julialang.org/t/speeding-up-fit-histogram/106355
**Category:** Performance
**Created:** [November 17, 2023, 12:13pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355 "2023-11-17T12:13:58Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Dan](https://discourse.julialang.org/u/Dan)
#### Post date: [November 17, 2023, 12:13pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/1 "2023-11-17T12:13:58Z")
As an offshoot of a thread about Makie histograms, I’ve tried to benchmark StatsBase `fit(Histogtam)` like so:
```julia
julia> using StatsBase
julia> @btime fit(Histogram, v, 0.0:0.1:1.0) setup=(Random.seed!(1234); v = rand(1_000_000))
31.720 ms (2 allocations: 224 bytes)
Histogram{Int64, 1, Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}}
edges:
0.0:0.1:1.0
weights: [99542, 99942, 99836, 100537, 99777, 99271, 99582, 100479, 100476, 100558]
closed: left
isdensity: false
```
As a comparison point, here is a `for` loop simple implementation:
```julia
julia> function fastest_hist(v, r)
hist = zeros(Int, length(r)-1)
f, s, l = first(r), step(r), last(r)
for i in eachindex(v)
a = v[i]
a < f && (hist[1] += 1; continue)
a >= l && (hist[end] += 1; continue)
hist[1+floor(Int, (a - f)/s)] += 1
end
hist
end
fastest_hist (generic function with 1 method)
julia> @btime fastest_hist(v, 0.0:0.1:1.0) setup=(Random.seed!(1234); v = rand(1_000_000))
8.933 ms (1 allocation: 144 bytes)
10-element Vector{Int64}:
99542
99942
99836
100537
99777
99271
99582
100479
100476
100558
```
The gap in performance on my machine is about 3x. Is there a way to improve performance of `fit(Histogram)` or the way I’ve used it?
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [November 17, 2023, 2:16pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/2 "2023-11-17T14:16:07Z")
Maybe check out this?
> **[GitHub - Moelf/FHist.jl: A pure Julia 1D/2D histogram package that focus on...](https://github.com/Moelf/FHist.jl)**
>
> A pure Julia 1D/2D histogram package that focus on speed and thread-safe. - GitHub - Moelf/FHist.jl: A pure Julia 1D/2D histogram package that focus on speed and thread-safe.
---
### Author:  [@Dan](https://discourse.julialang.org/u/Dan)
#### Post date: [November 17, 2023, 2:21pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/3 "2023-11-17T14:21:22Z")
Yep, it’s even faster than the `for` loop version:
```julia
julia> @btime Hist1D(v, 0.0:0.1:1.0) setup=(Random.seed!(1234); v = rand(1_000_000))
6.645 ms (6 allocations: 544 bytes)
edges: 0.0:0.1:1.0
bin counts: [99542, 99942, 99836, 100537, 99777, 99271, 99582, 100479, 100476, 100558]
total count: 1000000
```
That’s really good. Maybe it should replace the StatsBase implementation.
---
### Author:  [@viralbshah](https://discourse.julialang.org/u/viralbshah)
#### Post date: [November 17, 2023, 2:28pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/4 "2023-11-17T14:28:49Z")
Slightly off-topic - I think the JuliaStats ecosystem has matured nicely, but also needs some new contributors to forge the way forward. The challenge has always been that existing owners are too busy to review everything and they do need to keep the ecosystem working.
This would be exactly the kind of thing that can help bring new owners for these core packages. I’ll be happy to help make that happen.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 17, 2023, 2:53pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/5 "2023-11-17T14:53:29Z")
gotta go fast: [Inline and optimize `push!()` when one-shot histogramming by Moelf · Pull Request #87 · Moelf/FHist.jl · GitHub](https://github.com/Moelf/FHist.jl/pull/87)
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 17, 2023, 2:54pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/6 "2023-11-17T14:54:04Z")
I tried and failed:
- [Pull requests · JuliaStats/StatsBase.jl · GitHub](https://github.com/JuliaStats/StatsBase.jl/pulls?q=is%3Apr+author%3AMoelf+is%3Aclosed)
- [Issues · JuliaStats/StatsBase.jl · GitHub](https://github.com/JuliaStats/StatsBase.jl/issues?q=is%3Aissue+author%3AMoelf+is%3Aclosed)
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 17, 2023, 2:57pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/7 "2023-11-17T14:57:17Z")
> [@Dan](#):
>
> That’s really good. Maybe it should replace the StatsBase implementation.
`convert()` should already work, so you can take my histogram and disard some information and go back to `StatsBase.Histogram`.
My understanding is that #hep use cases are very fringe in the grand scheme of things, for example, in addition to the bin count (can be weighted) we also keep track of:
- sum of wgt^2 in each bin
- total number of times this histogram has been filled
---
### Author:  [@Dan](https://discourse.julialang.org/u/Dan)
#### Post date: [November 17, 2023, 3:08pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/8 "2023-11-17T15:08:16Z")
> [@jling](#):
>
> #hep use cases are very fringe in the grand scheme of things
Don’t think it is so fringe. HEP practitioners might be more fringe, but the statistics/computation needs of every field seem to converge quite surprisingly.
I agree that keeping the square weights is also important in a lot of use cases. Also choosing the right bins is interesting and takes a complete pass on the data and even partial sorting in some cases (i.e. calculating inter-quartile range).
Maybe we should delineate the algorithms according to memory complexity:
- on-line algos (dynamic histogram)
- 1-pass algos
- 2-pass algos
- random access…
And have a histogram version for each of these complexities.
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [November 17, 2023, 3:16pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/9 "2023-11-17T15:16:37Z")
That’s really a bummer, seeing all the efforts you have put in.
I also work in a field where histograms are breakfast, lunch and dinner (astroparticle physics) and I do everything related with `FHist.jl`, it’s awesome. I wish JuliaStats/StatsBase would appreciate the need for a proper and fast implementation.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 17, 2023, 4:45pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/10 "2023-11-17T16:45:20Z")
> [@Dan](#):
>
> Maybe we should delineate the algorithms according to memory complexity:
>
> - on-line algos (dynamic histogram)
> - 1-pass algos
> - 2-pass algos
> - random access…
> And have a histogram version for each of these complexities.
The first exists in: [GitHub - joshday/OnlineStats.jl: ⚡ Single-pass algorithms for statistics](https://github.com/joshday/OnlineStats.jl)
and for other binning, we actually have a cool Bayesian block: [https://github.com/Moelf/FHist.jl/blob/fd65d4b93f4d4bfca07f4f3bec481c7650c9f1db/src/hist1d.jl#L333](https://github.com/Moelf/FHist.jl/blob/fd65d4b93f4d4bfca07f4f3bec481c7650c9f1db/src/hist1d.jl#L333)
anyway, but in #hep actually most of the time you make a histogram and `push!()` into it event by event… which is why initially I missed this optimization, and even now I don’t have this optimization for 2D and 3D histogram.
* * *
In an ideal world I’d love to have the best histogram ecosystem in Julia, there are many interesting things going on:
- [GitHub - scikit-hep/hist: Histogramming for analysis powered by boost-histogram](https://github.com/scikit-hep/hist)
- [GitHub - scikit-hep/uhi: Universal Histogram Interface](https://github.com/scikit-hep/uhi)
- [superhistogram](https://indico.cern.ch/event/1028381/contributions/4340429/attachments/2238725/3795262/main.pdf) with many histogram into one and bunch of meta data
- how do we persistify histogram onto disk for long term and language-agnostic usage?
---
### Author:  [@Dan](https://discourse.julialang.org/u/Dan)
#### Post date: [November 17, 2023, 4:48pm UTC](https://discourse.julialang.org/t/speeding-up-fit-histogram/106355/11 "2023-11-17T16:48:19Z")
> [@jling](#):
>
> In an ideal world I’d love to have the best histogram ecosystem in Julia
Another niche important to be best in is the `countmap` area, also in StatsBase. Investing in StatsBase is important to draw/keep power users.