Question about mutual information calculation

adannenberg
Fri, 17 Jan 2025 20:09:09 +0000
Specific Domains

I came across this article which points out that
A(x,y) = sqrt(1 - exp(-2 * MI(x,y)))
(where MI(x,y) is the mutual information shared by data samples x and y) is a very nice measure of association between x and y.

One of the nice features that A(x,y) has is that A(x,y) = pearson correlation(x,y) when x and y are truly linearly related, e.g. if

n = 10_0000;
rho = .3;
x = randn(n);
z = randn(n);
y = rho * x + sqrt(1-rho^2)*z;

then, supposedly A(x,y) is approximately equal to rho.

The article uses R and the package FNN which bins the data using the method described in A. Kraskov, H. Stogbauer and P.Grassberger
(2004). Unfortunately the source is a mix of C and R and looks pretty impenetrable at first glance.

I tried to implement A(x,y) in Julia and my implementation failed its very first sanity check: I failed to reproduce the A(x,y) = rho feature for data that’s truly linearly correlated.

I first tried computing MI(x,y) with TransferEntropy.mutualinfo(x,y,Kraskov1(2)) and variations (different cluster size inputs to Kraskov1 and Kraskov2, etc.). I thought this should be the same method and result as the R package, but it doesn’t seem to be close.

I’ve also tried InformationMeasures.get_mutual_information(x,y) but it also fails to reproduce the result.

Can anyone provide guidance here? Am I calling the mutual info functions from TransferEntropy.jl and InformationMeasures.jl wrong? Is there a different package I should be using? It seems like A(x,y) is a better tool than Pearson correlation whenever there’s genuine non-linearity and a decent S/N ratio so it would be nice to have a working Julia implementation…

6 posts - 3 participants