# DataFramesMeta computing weighted median within @by

**URL:** <https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [August 27, 2021, 3:58pm UTC](https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111 "2021-08-27T15:58:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [August 27, 2021, 3:58pm UTC](https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111/1 "2021-08-27T15:58:30Z")

</div>

I’m struggling to figure this out. I have some data that look like this:

```julia
using DataFrames
using DataFramesMeta
using StatsBase

df = DataFrame(a=[1,1,2,2,3,3,4,4,5,5,6,6], b=rand(vcat(1:3, missing), 12), c=rand(10:50, 12))

```

![image](https://global.discourse-cdn.com/julialang/original/3X/9/d/9dd96c536dac084e4da84c993dbe44b76e0151e0.png)

and I need to compute a weighted median of `:b` for each `:a` in the presence of `missing` values. I would compute the non-weighted version like so:

```julia
@chain df begin
	@by(:a,
		median_b = StatsBase.median(skipmissing(:b))
	)
end

```

I’ve tried to several ways to compute the weighted median but without any success. Here are a couple of examples:

```julia
@chain df begin
	@by(:a,
		median_b = StatsBase.median(
			skipmissing(:b),
			pweights(Array{Int64,1}(getindex(:c, map(x -> !ismissing(x), :b))))
		)
	)
end

# produces a MethodError

@chain df begin
	@by(:a,
		median_b = StatsBase.median(
			skipmissing(:b),
			pweights(:c[:b .!== missing])
		)
	)
end

# also produces a MethodError

```

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [August 27, 2021, 4:06pm UTC](https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111/2 "2021-08-27T16:06:36Z")

</div>

This always happens. I banged my head against the wall for half an hour trying to figure this out, and then 5 minutes after posting here I figured it out:

```julia
@chain df begin
	@by(:a,
		median_b = StatsBase.median(
			collect(skipmissing(:b)),
			pweights(:c[:b .!== missing])
		)
	)
end

```

Just didn’t think long enough about _why_ I was getting a `MethodError` (or take the time to carefully read the error message 😌). The call to `skipmissing` results in `::Base.SkipMissing{SubArray{Union{Missing, Int64}` so you just have to `collect` it to get `Array{Int64,1}`

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 27, 2021, 8:18pm UTC](https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111/3 "2021-08-27T20:18:14Z")

</div>

This is something we want to make easier! See a stale PR [here](https://github.com/JuliaData/DataFramesMeta.jl/pull/276).

As an aside, note that with more recent versions of DataFramesMeta, you can use `begin ... end` blocks, with transformations on separate lines, to avoid ugly parentheses and commas (as well as use macro-flags like `@byrow` and `@passmissing` easier).

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 27, 2021, 8:39pm UTC](https://discourse.julialang.org/t/dataframesmeta-computing-weighted-median-within-by/67111/4 "2021-08-27T20:39:12Z")

</div>

In general alternatively you can add `dropmissing(:b, view=true)` (view to avoid allocations alternatively just `dropmissing(:b)`) as a first step in the chain and things should simplify.