This roadmap should be characterized as the DataFrames.jl / DataTables.jl / DataStreams.jl roadmap, but not the roadmap for the data ecosystem in general. There might be other packages that are on board, but at least some folks (e.g. me) are not yet sold on the Union{T,Null} approach and have plans/road…

From talking with various parties about the Union{T, Null} approach, the overall plan is the following: Represent nullable data with T? defined as Union{T, Null}. The representation of small unions is optimized to keep data inline like C unions with extra type indicator bits. Represent arrays of n…

+1 to all that. One thing we do lose is that Nullable forced client code to acknowledge that a value could be null since it would have to unwrap it. eg y = ... x = f(y) return x+1 If f returns a Nullable{Int}, then this code will fail when tested with any value of y and the author will be remind…

That would be addressed by the Value{T}? type where Value wraps an inner value and thereby distinguishes null meaning “there is no value” from Value(null) meaning “there is a value and it is null”. This single type addresses the nullable value pattern as well as the result type pattern since a funct…

Ah I see, great. Maybe having a github tracking issue that lists all the issues and PRs associated with making the plan you discussed a reality would be useful.

Sure. Just pulling that summary together has taken me a week :joy: :sob:, but yes, it should be collected on GitHub with PR references somewhere. As far as I know the Value idea is not posted anywhere, I got that from a direct conversation with @jameson . If he’s posted that on GitHub somewhere, hope…

[image] Announcement: An Update on DataFrames Future Plans Sure. Just pulling that summary together has taken me a week :joy: :sob:, but yes, it should be collected on GitHub with PR references somewhere. As far as I know the Value idea is not posted anywhere, I got that from a direct conversatio…

[image] Announcement: An Update on DataFrames Future Plans From talking with various parties about the Union{T, Null} approach, the overall plan is the following: I assume I was one of those “parties”… I would like to clarify that I think this is the correct plan for return values only, but t…

Despite the fear, uncertainty and doubt this reply likely evokes, none of this negates the fundamental plan – everything I wrote is still exactly what we plan to do. There are technical debates to be had (e.g. I vehemently disagree with the need for having a NullableArray{T} type separate from Array…

[image] davidanthoff: In my mind the high level strategy for the data ecosystem broadly should be that we try to push both the Union{T,Null} and the DataValue approach as far as we can, and once we have a better understanding of the trade-offs decide between those two approaches. +1 Someone …

[image] StefanKarpinski: I vehemently disagree with the need for having a NullableArray{T} type separate from Array{T?} … Our current undef is not a real value – but you also can’t return it from a function My intention was to observe the connection between nullable data and nullable fields…

Representing Nullable Values

Internals & Design

quinnj July 7, 2017, 10:05pm 9

Jameson posted some comments about there here

Topic		Replies	Views
Announcement: An Update on DataFrames Future Plans Data announcement	41	9488	December 27, 2017
Missing data and NamedTuple compatibility Internals & Design	92	11113	April 2, 2018
Getting our act together in the data ecosystem Data	4	1832	July 4, 2017
DataTables or DataFrames? Data question	32	15642	November 19, 2018
Missing or NaN General Usage	26	12630	August 1, 2018

Representing Nullable Values

Related topics