# Kullback-Leibler divergence for vector and normal distribution

**URL:** <https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246>\
**Category:** Statistics\
**Created:** [March 30, 2021, 3:51pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246 "2021-03-30T15:51:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jamblejoe](https://avatars.discourse-cdn.com/v4/letter/j/ee7513/32.png) [@jamblejoe](https://discourse.julialang.org/u/jamblejoe)\
**Post date:** [March 30, 2021, 3:51pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/1 "2021-03-30T15:51:23Z")

</div>

I want to calculate the [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) of data I collected in a vector `x`, which I interpret as samples from an unknown distribution, and the standard normal distribution. The maths behind the KL divergence are straightforward. My naive approach would be to

- choose a number of bins
- make a histogram of `x`
- discretize the density of the normal distribution according to the bins
- calculate the KL divergence of two vectors using for example `kldivergence` from `StatsBase`

I wonder how good of an approach that is (conceptually and implementation wise). Is there a Julia package with more refined methods? What about the sensitivity with respect to the number of bins?

Thanks for all the answers!

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [March 30, 2021, 5:33pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/2 "2021-03-30T17:33:22Z")

</div>

> I want to calculate the [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)

It’s implemented in Distances.jl (not the other cool package, just used there): ~~this~~ package (one of the metrics): [https://github.com/turingtest37/SequencerJ.jl/blob/master/docs/src/index.md](https://github.com/turingtest37/SequencerJ.jl/blob/master/docs/src/index.md)

and in the original Python version. See: [sequencer.org](http://sequencer.org)

---

<div class="post-metadata">

**Author:** ![jamblejoe](https://avatars.discourse-cdn.com/v4/letter/j/ee7513/32.png) [@jamblejoe](https://discourse.julialang.org/u/jamblejoe)\
**Post date:** [March 30, 2021, 9:30pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/3 "2021-03-30T21:30:02Z")

</div>

@Palli thanks for your quick answer. If I understood correctly, the KL-divergence in `Distances.jl` is calculating the distance between two vectors. So conceptually it is doing the same as `kldivergence` from `StatsBase`.

I did not understand the use of `Sequencer.jl` package. What problem does it solve and how would I use it for my case?

---

<div class="post-metadata">

**Author:** ![Adriel](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@Adriel](https://discourse.julialang.org/u/Adriel)\
**Post date:** [March 30, 2021, 11:24pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/4 "2021-03-30T23:24:10Z")

</div>

Discretizing the normal is the right thing to do.  
I’m sure you know this but you want to normalize it as a discrete distribution, not as a density (i.e. ignore the bin widths). Also, you can add a very small constant to everything to avoid numerical issues if there are any zero bins.  
I’ve heard it said that a good number of bins is the square root of the number of samples.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [March 31, 2021, 10:06am UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/5 "2021-03-31T10:06:57Z")

</div>

If you need to compare two densities, specially in high-dimensions, consider using a density ratio: [GitHub - JuliaML/DensityRatioEstimation.jl: Density ratio estimation in Julia](https://github.com/JuliaEarth/DensityRatioEstimation.jl)

You can express the KL-divergence in terms of the estimated ratio and that is usually more robust. All you need are samples from the two densities, no need to create bins.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [April 1, 2021, 1:22am UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/6 "2021-04-01T01:22:14Z")

</div>

> I did not understand the use of `Sequencer.jl` package.

It’s off-topic but cool. I didn’t read your question too carefully, and thought you were looking for an implementation, and I remembered (used) it there, but then I realized only in a dependency and edit my answer.

---

<div class="post-metadata">

**Author:** ![Rehana\_popy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rehana_popy/32/207810_2.png) [@Rehana\_popy](https://discourse.julialang.org/u/Rehana_popy)\
**Post date:** [June 6, 2024, 8:38pm UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/7 "2024-06-06T20:38:23Z")

</div>

Hi, I have a question regarding KL divergence calculation between two bivariate distributions P(x,y) and Q(x,y) where x is discrete but y is continuous. My strategy is to discretize the continuous variable using bins and then calculate their joint probability distributions. Does this sound okay?

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [June 8, 2024, 6:22am UTC](https://discourse.julialang.org/t/kullback-leibler-divergence-for-vector-and-normal-distribution/58246/8 "2024-06-08T06:22:10Z")

</div>

Hi @Rehana_popy welcome to the discourse ! Maybe you should open a new thread instead of resurecting a 2 years old one ? Also have a look at [Please read: make it easier to help you](https://discourse.julialang.org/t/please-read-make-it-easier-to-help-you/14757) 🙂

Otherwise yes, your strategy looks _globally_ OK but without a MWE I cannot judge much more.