Getting a strange error when trying to join list of DataFrames, what could this be?

Hi how’s it going?

I’m trying to join a list of DataFrames on a unique column, but it won’t even let me set a single one of those DataFrames to a variable and instead gives me this error.

“AbstractDataFrame is not iterable. Use eachrow(df) to get a row iterator or eachcol(df) to get a column iterator”

I’ve never had this issue in the past with Julia and the same code has worked for months.

Thanks

The most recent update to DataFrames.jl changed the join API. Did you upgrade DataFrames.jl to 0.21?

If so, this page shows the new joining functions.

Can you post a MWE?

1 Like
    cols = []
    for val in list_of_stuff
        greater_than_zero = filter(row -> row[val]>0,d)
        tx_sum = by(greater_than_zero, :identifier, Symbol(val) => sum)
        tx_mean = by(greater_than_zero, :identifier, Symbol(val) => mean)
        local_cols = [tx_sum,tx_mean]
        
        for df in local_cols
            push!(cols,df)
        end
    end
    
    total_df = cols[1]

    for i = 2:length(cols)
        total_df = join(total_df,cols[i],on = :identifier,makeunique=true,kind = :outer)
    end



AbstractDataFrame is not iterable. Use eachrow(df) to get a row iterator or eachcol(df) to get a column iterator

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] iterate(::DataFrame) at /home/shonbutani/.julia/packages/DataFrames/kwVTY/src/abstractdataframe/iteration.jl:23
 [3] indexed_iterate(::DataFrame, ::Int64) at ./tuple.jl:84
 [4] top-level scope at In[118]:1

Note that this is not an MWE. An MWE is copy-pasteable and doesn’t contain objects defined outside the code sample.

Can you confirm that you get an error for the following?

julia> cols = [DataFrame(a = [1, 2, 3], b = rand(3)) for i in 1:10];

julia> let total_df = cols[1]
       for i in 2:length(cols)
           total_df = join(total_df, cols[i], on = :a, kind = :outer, makeunique = true)
       end
       total_df
       end
2 Likes

I don’t get an error for that code you sent. Strange. Something to do with the by() method changing the type of DataFrame?

No, that is most likely not the issue.

You should try and make a MWE that we can copy and paste that reproduces the error.

Note that I can get the error by doing this:

julia> for i in DataFrame(rand(5,5))
       @show i
       end
ERROR: AbstractDataFrame is not iterable. Use eachrow(df) to get a row iterator or eachcol(df) to get a column iterator

So look for code that does something like that.

1 Like