which doesn’t give me totals for rows with missing values. I have tried various permutations of skipmissing without success. Also I can’t quite work out an expression to count the non-missing values in each row.
Thanks in advance.
EDIT: @pdeffebach provides an efficient solution to the :score problem. I also found a possibly less efficient solution which I can also apply to the :attempts problem.
Thanks again @pdeffebach
It’s clear I have to get my head around the src => fun => dest syntax.
Your solution worked, though I did just find an alternative solution which also worked for me. Not sure if it is as efficient but for me it is more comprehensible.
A nice to have would be to have the sum of all missings also be a missing rather than a zero. This would only affect row 5 in this example. Is this possible?
I propose a different solution. However, I would like to understand why the use of collect is necessary.
I note that a filtered missing vector produces an empty vector of type Missing.
Why doesn’t the same thing happen for the missing tuple?
julia> v=[missing,missing,missing]
3-element Vector{Missing}:
missing
missing
missing
julia> typeof(filter(!ismissing,v))
Vector{Missing} (alias for Array{Missing, 1})
julia> t=(missing,missing,missing)
(missing, missing, missing)
julia> typeof(filter(!ismissing,t))
Tuple{}
julia> sum(filter(!ismissing,v))
missing
julia> sum(filter(!ismissing,t))
ERROR: ArgumentError: reducing with add_sum over an empty collection of element type Union{} is not allowed.
Missings.jl has a new function called emptymissing which can help in this case. But it was only just merged and a new version hasn’t been released yet.
As you say, with collect you make a vector such as [missing, missing] of type Vector{Missing}. After filtering this becomes an empty vector of type Vector{Missing}. The sum of an empty vector is simply the “zero value” for the element type of the vector. In this case the zero value is defined as missing:
julia> zero(Missing) # Zero value for type Missing
missing
julia> sum(Missing[]) # Sum of empty vector of type Missing
missing
Without collect, what you have is a tuple such as (missing, missing). This has type Tuple{Missing, Missing}. When you filter-out one value you get a tuple (missing,) of type Tuple{Missing}. When you filter-out the last value you get a tuple () of type Tuple{}. The sum function should return the zero value for the element type, but there is no element type left, so sum doesn’t work:
julia> sum(())
ERROR: ArgumentError: reducing with add_sum over an empty collection of element type Union{} is not allowed.
A tuple is an immutable list of values, where each value can have any type. (1, 2.0, "three") has type Tuple{Int64, Float64, String}. If there is no value, there is no value type!
By the way you also cannot compute the sum of an empty vector of “unspecified” type Any because there is no “zero value” defined for Any:
julia> sum([])
ERROR: MethodError: no method matching zero(::Type{Any})
Thanks all for your help. Julia is indeed a rich language, especially when coupled with experts that frequent discourse!
Here is what I ended up using, where the column to sum starts with the letter “q”:
# sum, count & rank the quizzers and write the results
dfquizzersum.Total=[sum(filter(!ismissing, values(r))) for r in collect.(eachrow(dfquizzersum[:,r"q"]))]
dfquizzersum.Count=[length(filter(!ismissing, values(r))) for r in collect.(eachrow(dfquizzersum[:,r"q"]))]
dfquizzersum.Rank = competerank(-1 .* dfquizzersum.Total) # use tiedrank for "1 2.5 2.5 4" ranking, competerank for "1 2 2 4"
PS
But if you have the two columns of the total and the count, why do you complicate your life by also using the missing (instead of the 0) in the total column?
What additional information does the use of missing give you?