This post summarizes the current roadmap for DataFrames.jl and the data ecosystem in general for the Julia 1.0 milestone. Plans have changed significantly since the previous post . Indeed, it has appeared that representing missing values via the special Nullable type is not the only solution to attai…

Yes. Just to be explicit, f(a[1]) will optimize to something like if a[1] == null f(a[1]::Null) # Static dispatch else f(a[1]::Float64) # Static dispatch end

With the future Array{Union{T, NAType/Null/Whatever}}, could a BitArray be used instead of the Array{UInt8} to reduce memory used for the NA flags?

I thought this optimization already existed for unions with a small number of elements.

I’m not sure really, I know people were talking about it at some point. If we already have this, then we are much closer to replacing Nullable with the union representation than I thought.

[image] Ward9250: With the future Array{Union{T, NAType/Null/Whatever}}, could a BitArray be used instead of the Array{UInt8} to reduce memory used for the NA flags? That’s one of the options, it’s been discussed in this issue .

Potentially, but there are tradeoffs with performance. For now, it will be one extra byte per array element. Also note that the current compiler improvements underway generalize to more than just Vector{Union{T, Null}}; it supports Arrays of arbitrary dimensions, Unions with more than 2 unioned type…

There are codegen Union optimizations that have already been merged to do the call-site method splitting @malmaud mentioned above; unfortunately, there have been issues inlining after the method splitting, so the optimizations aren’t fully realized (I believe this is fully fixed in master though). T…

Oh great. Might you have any example of using @code_llvm or what have you that shows call-side method splitting in action?

NM, I think I have one: f(x::Int)=x+1 f(::Void) =nothing g(x) = x>0 ? f(x) : f(nothing) @code_llvm g(0) define { i8**, i8 } @julia_g_61178([8 x i8]* noalias nocapture, i64) #0 !dbg !5 { top: %2 = icmp slt i64 %1, 1 br i1 %2, label %L3, label %if if: …

@nalimilan , is there an update on this? I’ve noticed that all of the DataTables commits have been merged with DataFrames and that DataFrames now uses Nulls instead of DataArrays. This is definitely a welcome update, I’d like to start migrating my packages from DataTables to DataFrames in anticipat…

Announcement: An Update on DataFrames Future Plans

Specific Domains Data

Topic		Replies	Views
Representing Nullable Values Internals & Design	39	7606	January 20, 2018
Is there light at the end of the DataFrames tunnel? Data question	36	4595	November 24, 2017
DataTables or DataFrames? Data question	32	15675	November 19, 2018
Getting our act together in the data ecosystem Data	4	1834	July 4, 2017
Missing data and NamedTuple compatibility Internals & Design	92	11137	April 2, 2018

Announcement: An Update on DataFrames Future Plans

Related topics