By adding support for missing data and NamedTuples Julia is offering very solid foundations to build a data ecosystem upon. While I’m extremely happy that these two features have been added (especially given the extensive discussion and work that they required), I’d like to show that there are some …

My PR has been merged, so now map, collect, broadcast and similar functions use Union{T, Missing} rather than Any, including with Tuple and NamedTuple. This should satisfy all our needs in terms of user-visible behavior, right @davidanthoff ? Now the next step is to find a way to handle missing valu…

Bravo! And in time for 1.0 too! About the next steps, how tough do you think it will be to restore performance for this sort of union type inference? julia> @btime $[1, 2] .+ $[1, missing] 26.469 μs (40 allocations: 1.34 KiB) 2-element Array{Union{Missing, Int64},1}: 2 missing julia> …

I have no idea how hard it will be to improve performance in general. There are probably some low-hanging fruits. In the broadcast example you give, it appears that broadcast_nonleaf is used, and it’s a lot slower than the method used for concrete types. The approach is similar to map: it starts wit…

I’ve filed another issue to discuss a possible approach based on improving inference.

@davidanthoff When filing issue #25925 , I realized that inference is currently quite good with tuples, but not with NamedTuples (details in the issue). Do you think Query would work well with Missing if it used tuples rather than NamedTuples? Of course I’m not suggesting to use tuples, I’d just like…

Alright, I finally managed to catch up with this thread and some of the issues that it links to. Sorry for being slow. My high level comment is that I think we need to get a clear feedback from the core devs whether a strategy that relies on inference is all of a sudden blessed. The message I got f…

[image] davidanthoff: My high level comment is that I think we need to get a clear feedback from the core devs whether a strategy that relies on inference is all of a sudden blessed. The message I got from pretty much all the core devs was really clear: don’t use inference to pick container type…

[image] nalimilan: Once again, you need to distinguish between choosing the return type using inference and optimizing the implementation behind the scenes using inference. The former is indeed strongly discouraged, but I don’t think core devs are opposed to the latter. That’s a key point: In…

[image] piever: That’s a key point: IndexedTables is in the same situation: in a few cases the output type depends on the outcome of inference, which it shouldn’t, whereas I think it is fine to use a faster implementation if one can infer a concrete type. There was some discussions here and here…

[image] nalimilan: Interesting. Could you provide an example where this is the case? How does the current implementation manage to be efficient in these cases? @shashi is for sure much more qualified than I am to comment on this. My understanding is that now there are two different strategies…

Missing data and NamedTuple compatibility

Internals & Design

Topic		Replies	Views
Representing Nullable Values Internals & Design	39	7591	January 20, 2018
Announcement: An Update on DataFrames Future Plans Data announcement	41	9487	December 27, 2017
Missing or NaN General Usage	26	12630	August 1, 2018
Compatibility of Query and Union{T, Missing} Data	3	1767	November 28, 2017
Aliases for Union{T, Nothing} and Union{T, Missing}? New to Julia	40	7562	May 10, 2019

Missing data and NamedTuple compatibility

Related topics