Beta Distribution - full precision may not have been achieved

Specific Domains Statistics
1

Hi,

I’m upgrading my code to use Julia 0.6 and all of the libraries along with and working through the code to fix any deprecation warnings / errors.

However, since I update the Distributions library (v0.15.0) I seem to get the following warning output:

full precision may not have been achieved in 'qbeta'

qbeta(a, *) =: x0 with |pbeta(x0,*) - alpha| = 0.41325 is not accurateqbeta(a, *) =: x0 with |pbeta(x0,*) - alpha| = 0.49556 is not accurateqbeta(a, *) =: x0 with |pbeta(x0,*) - alpha| = 0.13113 is not accuratefull precision may not have been achieved in 'qbeta'
	From worker 25:	qbeta(a, *) =: x0 with |pbeta(x0,*) - alpha| = 0.23072 is not accuratefull precision may

This ultimately clogs up all my logs in my worker processes and I think causes it to fail with the following error:

Excessive output truncated after 524446 bytes.

Is there anyway to suppress this warning?

Regards,
Adrian.

2

It looks as if that warning is coming from the C function qbeta_raw in the Rmath package, which is a rather messy function. Can you determine what Julia function is being called with which arguments to generate one of these errors? Having |pbeta(x0,*) - alpha| = 0.41325 is a sign that something is definitely not right.

3

Looks like these parameters will cause an instance

q =  0.2702890618693184
d_suc
l = Distributions.quantile(d_suc,q)

Regards,
Adrian.

4

and this causes a slight other variation of the issue:

q =  0.4378680888898363
d_suc
l = Distributions.quantile(d_suc,q)
5

Can you be more explicit about d_suc? I assume it is a Distributions.Beta object but what are the parameter values?

6

Yes, sorry, I missed that when I pasted the issue:

a = 0.0009361372452551395
b = 0.057650470332520756
d_suc = Distributions.Beta(a,b)

Distributions.Beta{Float64}(α=0.0009361372452551395, β=0.057650470332520756)
7

The problem is that those parameter values are extreme and it is not really possible to get a positive value for which the cdf is less than 0.5

julia> quantile(Beta(0.0009361372452551395, 0.057650470332520756), 0.437868088889836)
full precision may not have been achieved in 'qbeta'
1.8941961587890447e-297

julia> realmin(Float64)
2.2250738585072014e-308

julia> cdf(Beta(0.0009361372452551395, 0.057650470332520756), realmin(Float64))
0.5070331794768044

The cumulative distribution function goes from 0. at 0. to over 0.5 at the next normalized floating point number. You really can’t define the quantile in this case.

You may want to check how the calculation comes about and see if it is possible to avoid these extreme parameter values.

1 Like
8

The data comes from a third party vendor so not too much we can do about that part of it.

The biggest issue seems to be the the underlying library isn’t throwing an error that I can wrap in a try catch block, it just prints it straight to standard out.

Ideally, it be nice to to omit the message as it seems to mess with the standard out causing the following:

Excessive output truncated after 524446 bytes.