I’ve been advised to ask this here vs stackoverflow

I want to compute the median values of all rows in a dataframe. Some columns contain NaN values. Some rows even have all NaN values. The problem with `median`

is

- if there’s any NaN values in a vector it returns NaN. In this case I would like to skip NaNs (like in Pandas).
- it is undefined for empty vectors (throws an error). In this case I want to return a NaN (like in Pandas)

I came up with the following solution:

`df = DataFrame(rand(100, 10), :auto) df[1, :x3] = NaN df[20, [:x3, :x6]] .= NaN df[5, :] .= NaN safemedian(y) = all(isnan.(y)) ? NaN : median(filter(!isnan, y)) x = select(df, AsTable(:) => ByRow(safemedian∘collect) => "median")`

This works however it’s rather slow.

**Question 1)** Is there a way to speed this up?

I think the collect method is causing the sluggish performance. But I need to use the collect method otherwise I get an error:

`safemedian(y) = all(isnan.(y)) ? NaN : median(filter(!isnan, y)) x = select(df, AsTable(:) => ByRow(safemedian) => "median") # results in ArgumentError: broadcasting over dictionaries and `NamedTuple`s is reserved`

This is because `AsTable(:)`

passes each row a named tuple.

**Question 2)** Is there a way to pass rows as vectors instead?

This way I could pass the row to any function that expects a vector (for example the `nanmedian`

function from the NaNStatistics.jl Package). Note I would not need to use the `collect`

method if the `AsVector(:)`

method was implemented (see here). Unfortunately it didn’t get the go ahead and I’m not sure what the alternative is.

**Question 3)** This one is more philisophical. Coming from Python/Pandas some operations in Julia are hard to figure out. Pandas for example handles NaNs/None values seemlessly (for better or worse). In Julia I artificially replace the missing values in my dataframe using `mapcols!(x -> coalesce.(x, NaN), df)`

. This is because many package functions (and functions I’ve written) are implemented for `AbstractArray{T} where {T<:Real}`

and not `AbstractArray{Union{T, Missing}} where {T<:Real}`

(ie. they don’t propagate missings). But since there is no `skipnan`

yet and there is a `skipmissing`

function in Julia, I’m thinking I’ve got it all wrong. Is the idiomatic way to keep missing values in Julia and handle them where appropriate? Or is it ok to use NaN’s (and keep the type fixed as say `Float64`

)?