Should Generators finally be given eltype

[Tamas_Papp]

@yuyichao

AFAICT most of the code in there is actually dealing with iterator, i.e. the customized and non-reusable part, rather than dealing with widening.

The interface I am thinking about is for collections, ie

container2 = store!_or_widen(container, elt)

defined for various types of container, complete with a empty_container(::Type{T}) and a finalize_container(container). When container2 !=== container, branch to recursion.

Eg the obvious one is Vector{T}, created by Base.Bottom[] (which I think is what should be returned for empty collections into vectors), and finalized as itself.

You are right that the application of this interface is fairly trivial, I was thinking of standardizing the container part. I am experimenting with this in FunctionalTables.jl, but it is heavily WIP.

[yuyichao]

@cstjean

Maybe I just need to assume that I can’t rely on the type of f.(v) at all, and code accordingly.

Yes. Without any context or without a well defined inference rule, we just can’t give any guarantee on the return type of f.(v) other than the current behavior. I’ll even say that in some sense a well understood dependency on the input is better to account for or debug than something that only happens when the inference decide to give you a bad day…

In this case, the right way to give it the context is to manually give it the type since you know what you are expecting (you have code that want to throw in a NaN of type Float64 after all).

@Tamas_Papp

You are right that the application of this interface is fairly trivial, I was thinking of standardizing the container part. I am experimenting with this in FunctionalTables.jl, but it is heavily WIP.

Right. Having some wrappers around the basic building block is certainly fine, especially in a package. (I wouldn’t personally call those building block any more since they’ll make more assumption than the “building blocks” in Base). My warning would be that such an interface might encourage people to match that interface when they shouldn’t, e.g. for different array types or even container type, like Dict/Set. This warning would not apply as much anymore once you’ve got a more complete collection of utilities written for different situations/container types and such a collection is certainly useful (at that point I’ll almost certainly not call it building blocks anymore but that’s not important…)

[ndinsmore]

@cstjean

I’m saying that the fewer branches to test, the better. f.(::Vector{Union{Missing, Float64}}) gives me four possible outcomes to worry about.

I can only figure out what three of the four branches you are talking about are.

But for what it is worth I use the signature AbstractArray{Union{Float64,T}} where T<:Missing to match two cases but not the third:

Vector{Float64}                   <: AbstractArray{Union{Float64,T}} where T<:Missing   == true 
Vector{Union{Float64,Missing}}    <: AbstractArray{Union{Float64,T}} where T<:Missing   == true
Vector{Missing}                   <: AbstractArray{Union{Float64,T}} where T<:Missing   == false

[cstjean]

I’ve been wondering how hard it would be to implement 0.4-style comprehensions and broadcasting, as a package-provided macro. Something like @stable f.(x), and @stable (f(x) for x in v), that returns a StableGenerator{Base.return_type(...)}, so it has an eltype

[tkf]

BroadcastArray does that https://github.com/JuliaArrays/LazyArrays.jl

[tkf]

@cstjean

function kalman_filter(initial_state, observations, theta, N)
   filtered = []
   likelihoods = []
   state = initial_state
   for i in 1:N
       state, ll = kernel(state, observations[i], theta)
       push!(filtered,  state)
       push!(likelihoods, ll)
   end
   return filtered, likelihoods
end

How should I pick the eltype of filtered ?

It would be nice if we can just write

function kalman_filter(initial_state, observations, theta, N)
   filtered = Union{}[]
   likelihoods = Union{}[]
   state = initial_state
   for i in 1:N
       state, ll = kernel(state, observations[i], theta)
       filtered = push!!(filtered,  state)
       likelihoods = push!!(likelihoods, ll)
   end
   return filtered, likelihoods
end

where push!! is a hypothetical push!-or-widen API. Is it crazy to imagine that compiler can do the “union-splitting” the for loop and lower it to a type stable code (when kernel is well behaved) in near future?