Want to add new column from string but df[!,field_in] = "test" not working

New to Julia
1

thought I’d try out adding a new column to a database from a string but something I thought should work isn’t?

using DataFrames
df = DataFrame()
field_in = "top_level"
payload = "new_one"

df[!,field_in] =  payload

I get this error

df[!,field_in] = payload

ERROR: MethodError: no method matching setindex!(::DataFrame, ::String, ::typeof(!), ::String)
Closest candidates are:
  setindex!(::DataFrame, ::Any, ::Integer, ::Union{AbstractString, Signed, Symbol, Unsigned}) at ~/.julia/packages/DataFrames/zqFGs/src/dataframe/dataframe.jl:658
  setindex!(::DataFrame, ::AbstractVector, ::typeof(!), ::Union{AbstractString, Signed, Symbol, Unsigned}) at ~/.julia/packages/DataFrames/zqFGs/src/dataframe/dataframe.jl:639
  setindex!(::AbstractDataFrame, ::Any, ::CartesianIndex{2}) at ~/.julia/packages/DataFrames/zqFGs/src/other/broadcasting.jl:5
  ...
Stacktrace:
 [1] top-level scope
   @ REPL[7]:1

what I want to achieve is

1×1 DataFrame
 Row │ top_level 
     │ String    
─────┼───────────
   1 │ new_one

and carry on adding ie

field_in = “middle_level”
payload = “another_one”

yielding

1×2 DataFrame
 Row │ top_level  middle_level 
     │ String     String       
─────┼─────────────────────────
   1 │ new_one    another_one

and carry on building it out.

what am I missing please?

2

Make it a vector.

julia> df = DataFrame()
0×0 DataFrame

julia> df[!,field_in] =  [payload]
1-element Vector{String}:
 "new_one"

julia> df
1×1 DataFrame
 Row │ top_level 
     │ String    
─────┼───────────
   1 │ new_one
1 Like
3

OUTSTANDING!!! thank you for such a swift solution. I can see why that works but I wouldn’t have spotted it. Thanks again for adding to my toolkit.

4

or df[!,field_in] .= payload

1 Like
5

Fyi, in DataFrames v1.3.4 that creates an empty dataframe with 1 column:

julia> df[!,field_in] .= payload
String[]

julia> df
0×1 DataFrame
1 Like
6

Hi there @bkamins as @rafael.guerra points out this doesn’t do what I wanted in the OP. I’m using DataFrames v1.3.4 and

df[!,field_in] .= payload

creates an empty dataframe with 1 column which is empty, I wanted a dataframe with 1 column that contains the value from payload.

I am watching your juliacon 2022 video which should contain a warning. I was asked to keep the noise down in a starbucks because I kept whooping everytime a new approach was unveiled. OUTSTANDING work!

7

@rafael.guerra excellent point and thank you for taking the time. I AM using DataFrames v1.3.4 and you are right but I don’t see why. Seems to me, a noob, that the broadcast transformation “.=” should have created a vector of the type String.

8

Ah - right. I did not see that your data frame had 0 rows before.
Then [payload] is correct.

Using broadcasting keeps the number of rows in a data frame unchanged, so if it had 0 rows it still has 0 rows.

3 Likes
9

Hi there @bkamins, thank you for the clarification. EXCELLENT work on the juliacon video. Nicely paced, lashings of explanation that REALLY helps noobs like me coming from python. Looking forward to your Chicago workshop in 2023 and your new book julia for data analysis

2 Likes
10

one more thing. Another way to add a column in this case is:

insertcols!(df, :field_in => payload)

The reason this works is that insertcols! uses the same syntax as DataFrame constructor so it treats scalars as length 1 if data frame is empty.

2 Likes
12

Hi there @bkamins not sure it works. Here’s the run I just did

using DataFrames
df = DataFrame()
field_in = "top_level"
payload = "new_one"
insertcols!(df, :field_in => payload)

giving a column called “field_in”

1×1 DataFrame
 Row │ field_in 
     │ String   
─────┼──────────
   1 │ new_one

which is not the functionality I was after. BUT a little noob tweek…

using DataFrames
df = DataFrame()
field_in = "top_level"
payload = "new_one"
insertcols!(df, field_in => payload)

1×1 DataFrame
 Row │ top_level 
     │ String    
─────┼───────────
   1 │ new_one

julia> field_in = "middle_level"
"middle_level"

julia> payload = "middle_add"
"middle_add"

julia> insertcols!(df, field_in => payload)
1×2 DataFrame
 Row │ top_level  middle_level 
     │ String     String       
─────┼─────────────────────────
   1 │ new_one    middle_add

julia>

and all is good in the garden… AS alway thanks so much for going the extra mile to help a struggling noob. This will allow a simple noob to build out a dataframe on the fly using a simple zmq IPM flow and have it use Stipple.jl observables to represent it as a live dashboard. WHAT FUN!! I am loving this stuff

so if I wanted to remotely send a list of symbols to the df I could do this

using DataFrames
df = DataFrame()
field_in = "sym"
payload = ["AAPL","AMZN","C","WAM","ERE"]

insertcols!(df, field_in => payload)
5×1 DataFrame
 Row │ middle_level 
     │ String       
─────┼──────────────
   1 │ AAPL
   2 │ AMZN
   3 │ C
   4 │ WAM
   5 │ ERE
13

:field_in would be a verbatim column name, while field_in would reference a variable with a column name. I understand you wanted the latter. So I hope all is now what you want.

1 Like
14

all is good in the garden, I like both the approaches. julia is a fine language and your DataFrames.jl and DataFramesMeta.jl make it a joy to explore.
thanks again.