# Data Cleaning: Split, Combine, Apply?

**URL:** <https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [October 24, 2021, 5:08pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296 "2021-10-24T17:08:26Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![YummyPampers2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yummypampers2/32/27328_2.png) [@YummyPampers2](https://discourse.julialang.org/u/YummyPampers2)\
**Post date:** [October 24, 2021, 5:08pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/1 "2021-10-24T17:08:27Z")

</div>

Greetings Everyone:

I have a DF:

```julia
Grade = ["A", "B","C","D"]
Price = rand(1:.001:500)
Name = ["John", "Jim", "Paula", "Ruby"]
Date = ["13/06/2008","17/05/2009", "21/04/2008"]

data = DataFrame(Name,Date, Grade,Price)

```

Excluding missing data types, I am attempting to calculate the mean price of each group with dropmissing() and by().

```julia
grPr = by(dropmissing(data), :Grade, :Price => x -> 
	AvgPrice = round(mean(x), digits=-3))

```

Returning the follow error:

```julia
ArgumentError: by function was removed from DataFrames.jl. Use the `combine(groupby(...), ...)` or `combine(f, groupby(...))` instead.

```

What adaptation would you use to convert the block above into one that includes combine(f, groupby(…)) since the by() method has been discontinued by Julia.

I applied:

```julia
grGr = groupby(dropmissing(data), [:Grade])

```

Then:

```julia
aggPr = combine(grGr, [:Price] .=> mean)

```

How would you combine these blocks? Or is it good practice to keep them separate?

Thanks,

---

<div class="post-metadata">

**Author:** ![Jakob](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jakob](https://discourse.julialang.org/u/Jakob)\
**Post date:** [October 24, 2021, 5:28pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/2 "2021-10-24T17:28:40Z")

</div>

Your example doesn’t work, possibly because `rand(1:.001:500)` doesn’t do what you think it does.

However, something like this works:

```julia
using DataFrames, Statistics

df = DataFrame(A=rand(100), B=rand(["a", "b", "c"], 100))
combine(groupby(df, :B), :A => mean)

```

---

<div class="post-metadata">

**Author:** ![YummyPampers2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yummypampers2/32/27328_2.png) [@YummyPampers2](https://discourse.julialang.org/u/YummyPampers2)\
**Post date:** [October 24, 2021, 5:30pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/3 "2021-10-24T17:30:34Z")

</div>

@Jakob - Thank you!

How would you apply the step of 0.001?

---

<div class="post-metadata">

**Author:** ![Jakob](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jakob](https://discourse.julialang.org/u/Jakob)\
**Post date:** [October 24, 2021, 5:35pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/4 "2021-10-24T17:35:24Z")

</div>

I’m not exactly sure what you wanted to do originally, but what that line was doing doing is defining the sequence from 1 to 500 with stepsize .001 (checkout `collect(1:.001:500)`) and then drawing one random sample from that.

---

<div class="post-metadata">

**Author:** ![Jakob](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jakob](https://discourse.julialang.org/u/Jakob)\
**Post date:** [October 24, 2021, 5:38pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/6 "2021-10-24T17:38:35Z")

</div>

> [@YummyPampers2](#):
>
> `DataFrame(Name,Date, Grade,Price)`

That is also not a valid way to create a `DataFrame`. You can either specify the names or use the keyword syntax, like so:

```julia
DataFrame(;Grade, Price)

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 24, 2021, 5:40pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/7 "2021-10-24T17:40:29Z")

</div>

While `by` was deprecated from DataFrames.jl, it lives on in the macro `@by` in DataFramesMeta.jl.

DataFramesMeta.jl provides a more convenient syntax for helping new users use Split-Apply-Combine concepts, as well as more basic transformations, in DataFrames. I recommend you check it out.

---

<div class="post-metadata">

**Author:** ![YummyPampers2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yummypampers2/32/27328_2.png) [@YummyPampers2](https://discourse.julialang.org/u/YummyPampers2)\
**Post date:** [October 24, 2021, 5:48pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/8 "2021-10-24T17:48:08Z")

</div>

thank you,

```julia
using DataFramesMeta

grPr = @by(dropmissing(Data), :Grade, :Price=> x -> 
	AvgPrice = round(mean(x), digits=-3))

```

Returns

```julia
LoadError: ArgumentError: Malformed expression in DataFramesMeta.jl macro

```

Would you make any adjustments to the code block above to accomodate the @by expression?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 24, 2021, 5:49pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/9 "2021-10-24T17:49:28Z")

</div>

Yes. Please read the documentation [here](https://juliadata.github.io/DataFramesMeta.jl/stable/). You want

```julia
@by(dropmissing(Data), :Grade, :AvgPrice = round(mean(:Price), digits = 3))

```

---

<div class="post-metadata">

**Author:** ![YummyPampers2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yummypampers2/32/27328_2.png) [@YummyPampers2](https://discourse.julialang.org/u/YummyPampers2)\
**Post date:** [October 24, 2021, 5:53pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/10 "2021-10-24T17:53:26Z")

</div>

Thanks for resource. Will access in the future!

---

<div class="post-metadata">

**Author:** ![YummyPampers2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yummypampers2/32/27328_2.png) [@YummyPampers2](https://discourse.julialang.org/u/YummyPampers2)\
**Post date:** [October 28, 2021, 10:22pm UTC](https://discourse.julialang.org/t/data-cleaning-split-combine-apply/70296/11 "2021-10-28T22:22:24Z")

</div>

Simply had to reset the kernels, everything worked well. Thank you!