Add incoming data to a DataFrame

New to Julia
1

Hi,

I made a DataFrame ab and i want to fill it with incoming data from aa[13].
aa[13] is incoming data which is refreshed every some time period.
if I want to push! the current row I’d do this:

julia> push!(ab,aa[13])
3×8 DataFrame
│ Row │ count │ time │ open │ close │ high │ low │ wap │ volume │
│ │ Int64 │ String │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Int64 │
├─────┼───────┼────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┤
│ 1 │ 0 │ 20190814 16:45:00 │ 11.8 │ 11.8 │ 11.8 │ 11.8 │ 11.8 │ 0 │
│ 2 │ 0 │ 20190814 16:48:00 │ 11.86 │ 11.86 │ 11.86 │ 11.86 │ 11.86 │ 0 │
│ 3 │ 0 │ 20190814 17:31:00 │ 11.9 │ 11.9 │ 11.9 │ 11.9 │ 11.9 │ 0 │

How do i push! the rows that are incoming after that to the df?

2

Welcome @uwbanjoman! Please make sure to quote your code and and provide a minimum working example–as is, we can’t tell what aa[13] is. What isn’t working for you?

You should be able to push! as many new rows as you want:

using DataFrames
df = DataFrame(a=1, b=2.0)
push!(df, (a=2, b=1.5)) # pushing a NamedTuple
push!(df, [5, 10.0]) # pushing a Vector
1 Like
3

Thank you @ElOceanografo

typeof(aa[13]) is
NamedTuple{(:count, :time, :open, :close, :high, :low, :wap, :volume),Tuple{Int64,String,Float64,Float64,Float64,Float64,Float64,Int64}}
is coming in let’s say every minute. so if i push this once it gets stored in DataFrame ab.
works perfectly.

The push! however stores only one NamedTuple: the last one
but how do i push the next minute NamedTuple, and the next etc.?

I’m thinking something like this. as long NamedTuples keep coming in: store them to the DataFrame,
unfortunately i do not know how to do that

Jan

4

You will need to push! each new NamedTuple as it comes in. Where are they coming from/how are you getting the new data?

Note you can surround your code in triple backticks ( ```) to make it appear in fixed-width format, which makes it much easier to read.

1 Like
5 
d = Dict{Symbol,Any}()
w = Wrapper(
historicalData= function(reqId::Int, bar::DataFrame)
                      d[:history] = bar
                      println("HistoricalData: $reqId $(size(bar))")
                    end,

historicalDataUpdate= function(reqId::Int, bar::NamedTuple)
                     d[:historyupdate] = bar
                     println("HistoricalDataUpdate: $reqId $bar")
                    end,
)
d, w
end

So now i’m trying to get the returned bars from d[:historyupdate] into d[:history]
by opening a channel.

historicalDataUpdate= function(reqId::Int, bar::NamedTuple)
                    chnl = Channel(c->println("HistoricalDataUpdate: $reqId $bar"))
                    cc = take!(chnl)
                    push!(d[:history], cc)
                    println("HistoricalDataUpdate: $bar")
                  end,

but above method gives me a method error at the take!() function.
is this a correct solution direction or am i thinking the wrong way, do not understand the matter?

please help

6

I’m still not sure exactly what the data update process is here–can you provide a self-contained example (i.e., one that we can paste into a REPL and have it run, including any using statements)? What error are you getting from the take! call?

Just reading the code above, it seems likely there’s a simpler way to do what you want. Maybe you could describe more generally what you’re trying to accomplish? Where do the data come from, how often do they update, what do want to do with the DataFrame once it’s been updated, etc.?