# Frustrated using DataFrames

**URL:** <https://discourse.julialang.org/t/frustrated-using-dataframes/67833>\
**Category:** New to Julia\
**Tags:** dataframes, data\_structures\
**Created:** [September 7, 2021, 9:38pm UTC](https://discourse.julialang.org/t/frustrated-using-dataframes/67833 "2021-09-07T21:38:10Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![Nathan\_Boyer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nathan_boyer/32/14825_2.png) [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)\
**Post date:** [September 7, 2021, 9:38pm UTC](https://discourse.julialang.org/t/frustrated-using-dataframes/67833/1 "2021-09-07T21:38:10Z")

</div>

# Overview

`DataFrames.jl` seems to be the preferred way to work with tabular data in Julia, but it never does what I want the first time. The syntax I need to contrive to do things I think would be simple is mind-boggling, especially when I try to explain what my code is doing to non-coders. It makes me wonder if there is a different package I should be using for my typical workflow, since the things I want to do often are so hard. I am usually joining CSVs together, filtering, sorting, and computing some new columns. These all sound like functions provided by DataFrames, but I don’t think I do it as explained in [The Split-Apply-Combine Strategy](https://dataframes.juliadata.org/latest/man/split_apply_combine/).

I have read the [documentation](https://dataframes.juliadata.org/latest/) and done the [tutorial on Github](https://github.com/bkamins/Julia-DataFrames-Tutorial). I nodded along as I went through, but using `DataFrames` on my own projects has been a different experience. I always think “No problem, I will whip this up in 10 minutes.” but an hour later am still sifting through error messages and documentation.

# Examples

1. [What rows are missing data?](https://stackoverflow.com/questions/64823551/return-copy-of-a-dataframe-that-contains-only-rows-with-missing-data-in-julia)

2. [Convert temperature data to Celsius.](https://stackoverflow.com/questions/65241599/what-is-the-right-way-to-transform-multiple-columns-of-a-dataframe-in-place-in-j)

3. Filter out rows which are all zeros.

> **Code for Example 3**
>
> ```julia-auto
> julia> using DataFrames
> 
> julia> df = DataFrame(node=1:4, x=[1,0,9,5], y=[0,0,12,8])
> 4×3 DataFrame
> Row │ node x y     
> │ Int64 Int64 Int64 
> ─────┼─────────────────────
> 1 │ 1 1 0
> 2 │ 2 0 0
> 3 │ 3 9 12
> 4 │ 4 5 8
> 
> ```
> 
> Trying to use the previous StackOverflow answers as a guide:
> 
> ```julia-auto
> julia> filter!(names(df, Not(:node)) .=> ByRow(row -> any(x -> x>0, row)), df)
> ERROR: MethodError: no method matching !(::Vector{Pair{String, ByRow{var"#204#206"}}})
> Closest candidates are:
> !(::Bool) at bool.jl:33
> !(::Function) at operators.jl:968
> !(::Missing) at missing.jl:101
> Stacktrace:
> [1] filter!(f::Vector{Pair{String, ByRow{var"#204#206"}}}, df::DataFrame)
> @ DataFrames C:\Users\nboyer.AIP\.julia\packages\DataFrames\vuMM8\src\abstractdataframe\abstractdataframe.jl:1127
> [2] top-level scope
> @ REPL[106]:1
> 
> ```
> 
> This works but destroys `df`:
> 
> ```julia-auto
> julia> filter!(row -> any(x -> x>0, row), df[!,Not(:node)])
> 3×2 DataFrame
> Row │ x y     
> │ Int64 Int64 
> ─────┼──────────────
> 1 │ 1 0
> 2 │ 9 12
> 3 │ 5 8
> 
> julia> df
> Error showing value of type DataFrame:
> ERROR: AssertionError: Data frame is corrupt: length of column :x (3) does not match length of column 1 (4). The column vector has likely been resized unintentionally (either directly or because it is shared with another data frame).
> Stacktrace:
> [1] _check_consistency(df::DataFrame)
> @ DataFrames C:\Users\nboyer.AIP\.julia\packages\DataFrames\vuMM8\src\dataframe\dataframe.jl:447
> 
> ```
> 
> Finally, after much frustration, this works as expected (nested anonymous functions with a tuple splat that is part of a pair using no broadcasting dots):
> 
> ```julia-auto
> julia> filter!(Not(:node) => (row...) -> any(x -> x>0, row), df)
> 3×3 DataFrame
> Row │ node x y     
> │ Int64 Int64 Int64 
> ─────┼─────────────────────
> 1 │ 1 1 0
> 2 │ 3 9 12
> 3 │ 4 5 8
> 
> ```

# Final Thoughts

Perhaps the most frustrating part is that I am unable to reason my way through the error messages. It is usually impossible to pare the syntax problem down to something smaller and make sure the pieces that make up `args` are passing the types and data structures I expect (row, column, vector, scalar, etc.). You have to get it all right in one go, or you will just get a random method error somewhere.

I am not trying to complain that the package is bad. It seems very powerful and obviously many people use it successfully. I’m just not sure what I am doing wrong. After troubleshooting syntax for hours, I usually ask myself why I didn’t just use Excel (and I hate Excel). Do you all have any tips?

---

_[View the full topic](https://discourse.julialang.org/t/frustrated-using-dataframes/67833)._