Rationale for the `@pure`ity of `Rational{T}(x)`

[tkf]

Looking at some example usages of @pure in Base, I cannot understand why Rational{T}(x::AbstractIrrational) is considered pure.

JuliaLang/julia/blob/851ae465c45c9db88f0a854646f28eb0589883e5/base/irrationals.jl#L32-L45

setprecision(BigFloat, _) mutates a global state and also it may throw. I thought Base.@pure docstring prohibits any global mutable states. Is it OK to mutate a global state if the function rollbacks the change?

[chakravala]

Looks like the precision value is not a regular global variable,

precision(::Type{BigFloat}) = DEFAULT_PRECISION[] # precision of the type BigFloat itself

and have a look at this

julia> Base.MPFR.DEFAULT_PRECISION
Base.RefValue{Int64}(256)

So there is definitely something different about it, although Base.RefValue does not have any doc string.

JuliaLang/julia/blob/master/base/refvalue.jl

[tkf]

The docstring Essentials · The Julia Language says

@pure function cannot use any global mutable state, including generic functions

So it applies to any state, not just global vairable. That’s why it extends to a generic function (the mutable state there is the method table). I’m almost sure that it includes a RefValue (but it would be very interesting if not).

[chakravala]

@tkf

I’m almost sure that it includes a RefValue (but it would be very interesting if not).

Looks like it is subtype of Ref

help?> Base.Ref
  Ref{T}

  An object that safely references data of type T. This type
  is guaranteed to point to valid, Julia-allocated memory of
  the correct type. The underlying data is protected from
  freeing by the garbage collector as long as the Ref itself
  is referenced.

  In Julia, Ref objects are dereferenced (loaded or stored)
  with [].

  Creation of a Ref to a value x of type T is usually written
  Ref(x). Additionally, for creating interior pointers to
  containers (such as Array or Ptr), it can be written Ref(a,
  i) for creating a reference to the i-th element of a.

  When passed as a ccall argument (either as a Ptr or Ref
  type), a Ref object will be converted to a native pointer
  to the data it references.

  There is no invalid (NULL) Ref in Julia, but a C_NULL
  instance of Ptr can be passed to a ccall Ref argument.

[JeffreySarnoff]

It does extend to Ref and RefValue –
If one may alter any aspect of or connection with φ then φ cannot participate @purely.
Purity is not restricted to Type info.

[ the symbol \varphi above stands for whatever happens to be relevant to the discussion ]

[tkf]

Can you look at my original question? Is it OK to be a @pure function if the mutation in the global state is rolled back? Maybe the real version of the spec is “@pure function cannot use any global mutable state in an observable manner” or something like that?

[tkf]

@JeffreySarnoff

If one may alter any aspect of or connection with φ then φ cannot participate @pure ly.

Actually, maybe you explained in this sentence. But it’s hard to understand it for me. (I can guess φ is the phi in LLVM IR but I’m not super familiar with it.)

[JeffreySarnoff]

afaik, the basic reason for @pure is that in very restricted and tightly established circumstances … where there can be no influence outside of the context directly given as that which follows @pure function and concludes with the matching end … and there can be no influence from within this context that extends outside of it … certain very strong presumptive compilation based runtime accelerators are available to be applied.

So any attempt to work-around purity or extend a pure function into a nonpure context is, by definition, inviting very hard to find bugs and errors in computational resolution. It does not mean they will occur; it does mean they might and there is no a priori way to know. (The manner of acceleration may change or new actions may be taken with the same pure context in a revision of Julia’s handling of @pure things.)

and this from Slack

Simon Danisch [11:32]
I think this is mainly about what @pure promises the compiler… since lots of theoretical compiler optimizations are not implemented, in practise you can use @pure quite freely…
but it will be like when people did unsafe stuff in Julia 0.6, which worked fine because we didn’t have agressive optimizations…and then suddenly it started segfaulting on 1.0, when we got more aggressive gc elimination passes

[tkf]

Thanks, my question here is rather to understand why such strict restrictions you listed do not apply to the method Rational{T}(x::AbstractIrrational) exists in Base. (Rather than to discuss, as in Extending methods of pure function, how to workaround such restriction, if possible at all).

If you think Rational{T}(x::AbstractIrrational) violates the assumptions of purity, I think we should make a PR.

The reason I can think of why Rational{T} is @pure is that:

• It rolls back the global state. Thus, executing the body of it or not does not change the semantics of the code following it.
• The second argument p in setprecision(BigFloat, p) never hits the branch in setprecision which throws.

[chakravala]

It would seem that Rational{Int}(π) returns the same value as rationalize does, and if you change the precision globally, it does not affect the results of the calculation.

julia> rationalize(Int,π,tol=0)
2646693125139304345//842468587426513207

julia> Rational{Int}(π)
2646693125139304345//842468587426513207

julia> setprecision(BigFloat,512)
512

julia> Rational{Int}(π)
2646693125139304345//842468587426513207

Also, I am trying to understand why the inequality here is > eps(bx) but maybe I just don’t understand how this algorithm is supposed to work…

@tkf

if abs(BigFloat(r) - bx) > eps(bx)

Is it supposed to increase the precision at each iteration? This algorithm doesn’t quite make sense.

[ChrisRackauckas]

@tkf

Thanks, my question here is rather to understand why such strict restrictions you listed do not apply to the method Rational{T}(x::AbstractIrrational) exists in Base . (Rather than to discuss, as in Extending methods of pure function, how to workaround such restriction, if possible at all).

If you think Rational{T}(x::AbstractIrrational) violates the assumptions of purity, I think we should make a PR.

It will cause issues. Specifically, @pure allows the compiler to compile one version that basically just spits out a pre-computed answer based on the input types here. That makes it so it won’t ever re-check the global after the first time it’s compiled, which is what @chakravala’s example is then showing. That is a feature of @pure, and why it shouldn’t be used in this case. I think it was added assuming BigFloat had one precision, which was incorrect and a PR should correct it.

[tkoolen]

Yeah, I think this is a bug.

https://github.com/JuliaLang/julia/blame/6c4429b2b547b3a3ae3087a2245055d044ceca4d/base/irrationals.jl#L18-L21

got changed into

https://github.com/JuliaLang/julia/blame/a02c7312d2e5ef0f047765e4df7bfd38c084573d/base/irrationals.jl#L18-L31

in Fix conversion Irrational => Rational by simonbyrne · Pull Request #16527 · JuliaLang/julia · GitHub, and it’s just been carried over ever since.

[chakravala]

@tkoolen

in https://github.com/JuliaLang/julia/pull/16527 , and it’s just been carried over ever since.

If T is a AbstractFloat or Rational type, then it will return the closest value to x representable by T .

The aim of the @generated is to get it to perform the calculation at compile time, as the result will always be constant. This is exactly the same as the usage in the function directly below.

Technically, an @pure annotation would be sufficient , once #14324 is sorted out.

This might be correct after all, since the result depends on the representation by T, which is invariant under changes with respect to the BigFloat precision, i.e. constant.

Therefore, the @pure macro was used. So this is why this function doesn’t really depend on a global state, because the result shouldn’t really change anyway, because the representation in T won’t change.

So the result doesn’t really depend on the global state of BigFloat precision, also it seems that rationalize would also be sufficient for irrationals, and the rest of the algorithm is unnecessary?

EDIT: ahh now I see how it works: in case you changed the BigFloat precision before calling this function, this algorithm ensures that it uses the correct precision value and resets it after, but overall, it does not depend on the global state of the precision. The result does not and should not change.

So it seems the @pure macro was used here in a very special use case, where a global variable is used in the algorithm, which only needs to be used once at compile time and is otherwise irrelevant. The use of the global variable is only to ensure that the computation uses the standard precision at compilation, the state of the global variable does not and should not matter otherwise. Now it all makes sense.

I believe this would be a sufficient algorithm

@pure function Rational{T}(x::AbstractIrrational) where T<:Integer
    o = precision(BigFloat)
    setprecision(BigFloat, 256)
    r = rationalize(T, x, tol=0)
    setprecision(BigFloat, o)
    return r
end

[tkf]

Thanks for detailed explanation. Probably I should have summarized that in the first post (I thought quoting the code was enough). To clarify, my question has been: operationally Rational{T}(x::AbstractIrrational) looks like a pure function but does it really avoid restriction of @pure? Is it OK to mutate global states in @pure if it is not observable? If so, what is the definition of the observability?

For example, for Rational{T}(x::AbstractIrrational) to be safe when multithreading becomes a normal style in Julia, I think DEFAULT_PRECISION has to be guarded by a lock. An interesting question is that “does accruing a lock mutate the global state?” It certainly is “observable” in the sense that not accruing a lock can turn dead-locking program to a non-dead-locking program. But then maybe things are not “observable” when it does not alter “correct” programs?

@chakravala

I believe this would be a sufficient algorithm

But how do you know 256 is enough? Also, if you want to discuss this aspect of Rational{T}, it’s maybe a good idea to open a new topic (or a github issue).

[chakravala]

@tkf

But how do you know 256 is enough? Also, if you want to discuss this aspect of Rational{T} , it’s maybe a good idea to open a new topic (or a github issue).

It would only be enough for native Julia types, such as Int, UInt128, etc. More generally, it would not be enough. I wasn’t planning on making a Pull Request for it, just analyzing the usage of setprecision and the @pure macro in this particular situation. Could someone else comment on this?

@tkf

To clarify, my question has been: operationally Rational{T}(x::AbstractIrrational) looks like a pure function but does it really avoid restriction of @pure ? Is it OK to mutate global states in @pure if it is not observable ? If so, what is the definition of the observability ?

I can’t answer this, because I am not knowledgeable enough, but based on my findings, it looks like in this particular case it is valid.

[tkf]

As I said, I’ve already noticed what you found. I also explained why that’s not enough.