Dear Community,

I am about to calculate the **mutual information** of two PMFs according to (1) in mathworld

which i would normalize to get the redundancy, R, in Wiki. A particular feature of the problem is that the **samples “behind” PMFs are not available**. It occured, that two PMFs, PX and PY, contributed to negative mutual information, which is suspicious to me.

```
# My suspicious MWE:
# bincenters = (1:1:length(PX))' # bin-centers are simply integers
PX = [0.010656 0.0130878 0.0291189 0.0262176 0.0350016 0.0604959 0.0620084 0.0695799 0.0813843 0.0831733 0.0807244 0.0763465 0.0981173 0.0632269 0.0648809 0.0661584 0.0316574 0.0189747 0.0194769 0.00971281]; # PMF of x
PY = [0.0172311 0.0162825 0.0379995 0.0257677 0.0390367 0.0609703 0.0624129 0.0723287 0.0810149 0.0842886 0.0910396 0.0840303 0.0704959 0.0664023 0.0486554 0.0366271 0.0321917 0.0234354 0.0151906 0.0345987]; # PMF of y
#=
import Pkg
Pkg.add("Plots")
using Plots
plot(PX; line=:stem, marker=:circle, color=:black, legend =:none)
plot!(PY; line=:stem, marker=:circle, color=:gray)
=#
PXY = PX.*PY' # joint probability if PX and PY are independent ?
MI = sum(PXY.*log2.(PXY./(PX.*PY))) # mutual information
HX = -sum(PX.*log2.(PX)) # entropy X
HY = -sum(PY.*log2.(PY)) # entropy Y
R = MI/(HX+HY) # normalized mutual information
```

I have a hard time to figure out, why MI<0 in this case, and the followings concerned me:

- If PX and PY are
**independent**, PXY can be used, as I did? - If PX and PY are
**dependent**, how shall I calculate the conditional probabilities? - Shall I use
**samplers**to produce x and y underlying realizations? (I tried to make friends with samplers but that is a new field of study for me, and thus, it is hard to familiarize with the Documentation of Turing.jl despite its values.)

Ultimately, how can happen that MI is negative? It is possible, that I miss concepts from probability, so I am grateful for your suggestions

Thank you in advance and in the meantime I study further the Docs.

Kind regards,

ladislaus