OnlineStats can’t handle Missing?

Am I mistaken in my experience that the OnlineStats package can’t handle Missing? Is there anything from Julia Computing that actually works?

Not only are these packages out there, they’re presenting them as being good, and even trying to market “Pro” services. The low quality is going to make Julia itself disparaged by association.

Shouldn’t there be some kind of package moderation to avoid this abuse?

Perhaps wrap your iterator in skipmissing? This works for any iterator that may have Missing in it.

julia> OnlineStats.fit!(Mean(), [1, missing, 3])
ERROR: MethodError: no method matching iterate(::Missing)
Closest candidates are:
  iterate(::Core.SimpleVector) at essentials.jl:604
  iterate(::Core.SimpleVector, ::Any) at essentials.jl:604
  iterate(::ExponentialBackOff) at error.jl:214
  ...
Stacktrace:
 [1] fit!(::Mean{Float64,EqualWeight}, ::Missing) at /Users/sebastianrollen/.julia/packages/OnlineStatsBase/L6i9N/src/OnlineStatsBase.jl:109
 [2] fit!(::Mean{Float64,EqualWeight}, ::Array{Union{Missing, Int64},1}) at /Users/sebastianrollen/.julia/packages/OnlineStatsBase/L6i9N/src/OnlineStatsBase.jl:110
 [3] top-level scope at /Users/sebastianrollen/.julia/packages/JuliaInterpreter/MXq3U/src/construct.jl:0

julia> OnlineStats.fit!(Mean(), skipmissing([1, missing, 3]))
Mean: n=2 | value=2.0

Side-note, but what’s the deal with the Julia Computing beef? There’s two topics from the front page currently where you place blame on them for issues that seem to stem completely from your own lack of familiarity with the Julia ecosystem.

4 Likes

As you’ve been told on the other thread, this kind of attack isn’t welcome here. If you’re a customer of Julia Computing, you can contact support and you will receive professional support in the appropriate forum, and though we’d prefer a calm approach there as well, we’ll make sure you get the help you need regardless of tone. However, in this forum both you and the authors of the various packages you’re attacking (whether they be employed by Julia Computing or not) are peers within the Julia community in a forum run by the julia community and in this community we don’t do things like that - who employs whom doesn’t change that.

11 Likes

Adding skipmissing didn’t fix it for me. I’m trying to run this on a CSV.Rows object, so skipmissing might not work with the iterator?

My frustration comes from JuliaDB loadndsparse: many errors - #15 by xiaodai
While I’m definitely a beginner with the language, my first impulse is to blame myself and try to find what I did wrong. Having people tell me that JuliaDB is essentially unmaintained and in the same thread tell me that it’s what to use for larger-than-memory data, well, we’re past my first impulse, and I’m running late due to these misrepresentations.

Although, I can sympathise with the frustration, I think there are more constructive ways of communicating.

I believe that juliadb didn’t get much commercial traction and hence is not in active development. Given that it is available for free to non paying customers I think it’s not reasonable to treat like any other open source project i.e. without entitlement.

The question i have for anyone who complains about open source is: have u contributed to open source in any way? Or have you always just benefited from open source?

1 Like

Am I mistaken in my experience that the OnlineStats package can’t handle Missing?

Yes, you are mistaken.

Is there anything from Julia Computing that actually works?

OnlineStats is not a product of Julia Computing. It is a product of me that I built during my PhD and continue to maintain for free while I work at Julia Computing.

If you search the documentation for “missing”, the only item that appears is FTSeries, which provides several examples of how to handle missing data. So firstly, read the docs.

Please remember that you are talking to human beings that mostly maintain their open source packages because it’s fun. Please don’t make it less fun for us.

23 Likes

Thanks, I’ll try to be less of an A’s, too. (Hopefully it works, lol)