could we use genie to represent real time ( 1 sec) updates of a data structure.
We tried stipple.jl but it freezes on the web page but the dataframe is still being updated.
This definitely sounds like a use case for Stipple - 1s is reasonable and we’ve built a couple of apps that performed OK. However, I can’t comment without more details.
1/ obviously, the most important aspect is the amount of data being transferred and how this impacts the lifecycle of the app (ie does it trigger a lot of other reactive events, does it trigger a lot of UI rendering, etc)?
2/ what does the app do and what freezes (is it the UI or the backend)? If it’s the UI, switching the backend won’t help and other solutions would be needed (like maybe moving rendering over the GPU if possible).
3/ what version of Stipple have you used - output from Pkg.status would help.
4/ Would you be able to provide a MWE illustrating the issue? Or the whole app?
I would suggest moving this to an issue in Stipple.jl to dive into it.
FYI @hhaensel
How do you update the datastructure? Could you paste that bit of code?
Hey @hhaensel
GREAT to hear from you again. Happy to paste code. Should be VERY familiar to you as most of it is yours 
The webpage data freezes but the dataframe update continues.
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "/home/dave/tontine_2022/data/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
@reactive mutable struct TontineModel <: ReactiveModel
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
function handlers(model)
on(df_table) do _
notify(model.tontine_data)
end
model
end
function price_calcs( )
#println( "entering price_calcs " )
row_found = filter(:sym => ==(sym_in),df_table[])
expected_move = round((row_found.iv[1] / 19.896) ,digits = 2)
change = round( (((row_found.price[1] - row_found.close[1] ) / row_found.close[1] ) * 100 ) , digits = 2)
sdmove = round(( change / expected_move ), digits = 2)
isinf(sdmove) && (sdmove = 999.99
try
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = 999.00
end
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
(model = init(TontineModel)) |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) ===>>> starting IN Socket 5555"
flush(io)
timeout = 10 # or whatever your startup process needs
t0 = now()
while ! model.isready[] && now() - t0 < timeout
sleep(200)
end
now() - t0 < timeout && model.df[] = my_brand_new_dataframe
or just do
model.isready[] && model.df[] = my_brand_new_dataframe
=#
while true
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "=================================================================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
break
catch zmq_error
@info "*********************************************************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
global in_source, sym_in, field_in , value_in = split( message , "~")
global value_fl = parse(Float64, value_in) # convert to Float64
try
global field_out = zmq_dash[ field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[][!,:sym] || push!(df_table[], (sym_in,0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==(field_out), names(df_table[]))] = value_fl
if field_out == "price"
price_calcs()
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
end
hi there @essenciary
congrats on the new release. exciting times. We’ve done a bunch of startups having learnt from the best (tom perkins was our first chairman and ceo).
here are your answers
1 amount of data
Nope about 10 messages a second and nothing too spectacular. The database is updated correctly it’s just that the web page is frozen. Right now the ONLY UI rendering is two columns of 66 rows.
2 The app:
opens a ZMQ feed and gets a message
the message is parsed into a dataframe update single row single column.
the dataframe is posted to a webpage with stipple.jl
what happens
the UI freezes
not a big fan of moving anything to the GPU unless there is a REALLY good reason.
3 versions.
st
Status `~/.julia/environments/v1.8/Project.toml`
[c52e3926] Atom v0.12.38
[336ed68f] CSV v0.10.9
⌃ [1b08a953] Dash v1.1.2
⌃ [a93c6f00] DataFrames v1.3.6
[1313f7d8] DataFramesMeta v0.14.0
[31a5f54b] Debugger v0.7.8
⌅ [c43c736e] Genie v4.18.1
⌅ [cd3eb016] HTTP v0.9.17
[f310f2d2] Jib v0.20.0 `https://github.com/lbilli/Jib.jl#master`
[e5e0dc1b] Juno v0.8.4
[f0f68f2c] PlotlyJS v0.18.10
⌃ [c3e4b0f8] Pluto v0.19.9
[0ff47ea0] PlutoHooks v0.0.5
[7f904dfe] PlutoUI v0.7.51
⌅ [4acbeb90] Stipple v0.24.5
⌃ [ec984513] StipplePlotly v0.12.4
⌃ [a3c5d34a] StippleUI v0.19.4
[9e3dc215] TimeSeries v0.23.1
[c2297ded] ZMQ v1.2.2
[ade2ca70] Dates
[56ddb016] Logging
[10745b16] Statistics
4 happy to provide test data for this. It has to use ZMQ as using a randomizers inside the julia environment isn’t useful to us.
I have an outstanding stipple.jl issue on this from 9/22/22 old stipple.jl issue no one got back to us so we just moved on to use another solution outside julia.
We did a little digging and tried this out in Plotly Dash - Real time candlestick dashboards (Python only) worked as advertised so it’s possible with pycall.jl BUT we want to give genie.jl the old college try.
We think that handling real time data is an ideal usecase for genie. Not only for financial applications but also for lab machine feeds and IOT devices.
thanks for looking into this
1 Like
There’s lot’s of strange stuff in your code.
- I don’t see where you are updating the field tontine_data
- is_ready shouldn’t be waited for in a loop as you do
You probably want a different handlers function:
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
Currently, there might be a problem with this setup, because models with disconnected clients are still triggered by the update of df_table. And in return they will try to reach their client which brings up an error message.
@essenciary We will have to setup a finalizer that kills the binding to df_table.
Alternatively you could define your model as a static global variable, if it’s ok that all users see the same content.
model = init(TontineModel)
route("/") do
global model
model |> handlers |> ui |> html
end
P.S.: This is all untested, maybe there are some typos …
hi @hhaensel
thank you for taking a look as always on @essenciary suggestion I reopened 9/22/23 original problem where we addressed this issue and maybe we should continue this?
to your points
1 this is your code
from the 9/22/23 github discussion where we tried many of your suggestion in this thread.
2 I believe we tried
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
and it didn’t work.
3 static content
we need our users to see the updating dataframe and use widgets to sort,select, filter based on columns and rows. Each will need to see their results and none of the other participants.
SO if you mean that everyone will see the results of actions by the others ( say selection results) then we can’t work with that.
conclusion
It seems to me that stipple.jl doesn’t currently handle streaming updates of data.
When we first raised this issue in Sept of 2022 you said that this was a good usecase for GENIE,.jl and thus would be considered in the core product. We revisited this issue because of the @essenciary release launch announcement with the hope that it was addressed in the months between 9/22/23 and 5/4/23.
We have been in many startups and understand the priorities so we understand completely. We would suggest that the ability to handle a constantly updated datastructure would be something to consider.
with the new launch and prepping for a wonderful juliaconn 2023 I would imagine that’s the area to concentrate on and we can go in another route ( python,dash,pandas) to achieve this fairly quickly. We use ZMQ for that reason. We can use julia for the data analysis and pipe the results to python/dash/pandas with a change in port address.
Try using the new API for model generation:
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
Will test tonight…
please let us know how your test goes. We are trying to get to this today. Just to be clear the change is
OLD CODE
@reactive mutable struct TontineModel <: ReactiveModel
tontine_data::R{DataTable} = DataTable(df_table[])
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading Tontine2 8/21/22")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",)
])
])
]
)
end
function handlers(model)
on(df_table) do _
notify(model.tontine_data)
end
model
end
REPLACED WITH
#@reactive mutable struct TontineModel <: ReactiveModel
# tontine_data::R{DataTable} = DataTable(df_table[])
#end
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100)
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading Tontine2 8/21/22")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",)
])
])
]
)
end
function handlers(model)
on(df_table) do _
notify(model.tontine_data)
end
model
end
Is this what you want us to try?
ERROR: LoadError: UndefVarError: @vars not defined
in expression starting at /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:43
in expression starting at /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:43
You have to update Stipple and StippleUI to the latest versions!
And you have to use the handlers function with the one I proposed
if you could run your test against the github data we put at zmq test harness that would help.
We don’t want to try something that isn’t possible. Please remember that we are using ZMQ as the feed as making random data within the script isn’t a fair use case.
We understand that real time no code web representation is a niche case in julia but it is becoming more important.
We can easily change our ZMQ stream to feed to python/dash/pandas ( took 5 minutes) and host the python dash app whilst having a BLAST with @bkamins DataFramesMeta.jl which is a pleasure to use.
if you tell us that you got the test code working with your approach then we’ll set up an new project, move all the existing code into it. Load up the latest versions of requirements and try again.
Just to be clear.
the NEW code would be
#@reactive mutable struct TontineModel <: ReactiveModel
# tontine_data::R{DataTable} = DataTable(df_table[])
#end
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100)
end
model = init(TontineModel)
route("/") do
global model
model |> handlers |> ui |> html
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading Tontine2 8/21/22")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",)
])
])
]
)
end
#function handlers(model)
# on(df_table) do _
# notify(model.tontine_data)
# end
# model
# end
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
Is this correct?.
we made the mods, created a new project, added only the packages we need and made sure they were up to date. julia is also the current version. we ran the code from the cli making sure to use the right project and it “seemed to work” then it just stopped updating the ui as before. ALSO if we hit refresh on the page we get
Info: message : 2023-05-04T16:18:22.788 :
│ message = STK~AAPL~LAST~167.78
└ @ Main /home/dave/Desktop/tontine_2023/stipple_2023/stp_2023/src/stp_2023.jl:198
┌ Error: Error attempting to invoke handler 2 for field Reactive{Bool}(Observable(true), 1, false, false, "") with value true
└ @ Stipple /home/dave/.julia/packages/Stipple/ivn7W/src/stipple/mutators.jl:36
┌ Error:
│ exception = (MethodError(convert, (DataTable, 60×11 DataFrame
Row │ sym close price sdmove hv20 hv10 hv5 iv iv%ile prc%ile ern_days
│ String Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────
1 │ AAPL 167.45 167.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 │ XLP 76.66 76.47 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 │ META 237.03 233.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 │ XBI 83.21 84.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 │ GS 328.65 320.9 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
6 │ TSLA 160.61 161.24 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 │ QCOM 112.83 106.38 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 │ NVDA 278.02 275.0 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 │ IWM 172.33 170.2 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 │ MSFT 304.4 305.42 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
11 │ QQQ 317.29 316.16 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
12 │ DD 64.4 64.06 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
13 │ MA 374.9 375.01 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14 │ SBUX 103.96 104.76 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
15 │ JPM 135.98 134.12 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
16 │ AMZN 103.65 103.98 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
17 │ GOOGL 105.41 104.5 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
18 │ GOOG 106.12 105.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
19 │ EBAY 44.77 44.38 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
20 │ SPY 408.02 405.0 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
21 │ IYR 83.07 83.74 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
22 │ NFLX 319.3 320.99 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
23 │ WFC 38.35 36.75 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
24 │ V 225.98 225.2 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
25 │ XLV 133.46 132.43 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
26 │ UNG 6.31 6.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
27 │ WDAY 183.32 181.92 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
28 │ C 45.67 44.82 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
29 │ IEFA 68.19 68.05 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
30 │ UPS 175.83 173.27 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
31 │ SLV 23.43 23.93 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32 │ IEX 210.1 207.49 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
33 │ XRT 60.33 59.48 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
34 │ VLO 107.06 104.31 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
35 │ XOP 118.05 116.23 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
36 │ XLK 148.25 147.6 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
37 │ LEN 114.13 112.35 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
38 │ XLRE 36.53 36.81 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
39 │ SLB 45.27 45.01 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
40 │ STZ 225.5 222.41 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
41 │ WDC 33.05 33.04 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
42 │ WBA 31.96 31.55 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
43 │ YUM 137.31 136.32 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
44 │ MCD 295.22 296.21 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
45 │ XLF 31.96 31.47 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
46 │ HD 293.08 285.5 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
47 │ GILD 79.45 78.4 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
48 │ TLT 106.29 105.43 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
49 │ VZ 37.98 37.32 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
50 │ WMT 150.05 150.24 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
51 │ JNK 91.45 91.18 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
52 │ HYG 74.58 74.36 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
53 │ XLU 68.07 68.49 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
54 │ DIA 334.11 331.07 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
55 │ GSK 36.84 36.71 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
56 │ GME 18.9 19.42 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
57 │ USO 60.31 60.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
58 │ VWO 39.68 40.02 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
59 │ UCO 21.79 21.95 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
60 │ INDA 40.92 41.25 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0), 0x0000000000007f2d), Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x00007fb8569f4c26, Ptr{Nothing} @0x00007fb8569fb535, Ptr{Nothing} @0x00007fb8569fd4ed, Ptr{Nothing} @0x00007fb70015253f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700151f3c, Ptr{Nothing} @0x00007fb70015244a, Ptr{Nothing} @0x00007fb700152485, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb700150ee7, Ptr{Nothing} @0x00007fb700151c86, Ptr{Nothing} @0x00007fb700151cb0, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700150b7a, Ptr{Nothing} @0x00007fb700150bc4, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb70014db83, Ptr{Nothing} @0x00007fb70014dc9f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700148c5b, Ptr{Nothing} @0x00007fb70014908b, Ptr{Nothing} @0x00007fb7001490ec, Ptr{Nothing} @0x00007fb700149161, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb7001355f3, Ptr{Nothing} @0x00007fb70013562f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb7134ed744, Ptr{Nothing} @0x00007fb7134ede20, Ptr{Nothing} @0x00007fb7134efcb7, Ptr{Nothing} @0x00007fb7134f4d7f, Ptr{Nothing} @0x00007fb7134f4e22, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb7134e09b9, Ptr{Nothing} @0x00007fb7134e303a, Ptr{Nothing} @0x00007fb7134e30ef, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a1f4ff])
└ @ Stipple /home/dave/.julia/packages/Stipple/ivn7W/src/stipple/mutators.jl:37
┌ Info: message : 2023-05-04T16:18:23.539 :
│ message = STK~TSLA~LAST~161.25
and the UI updates then freezes.
here’s what we did ( we only had 15 minutes so if you see where we screwed up please tell us)
we created a new project added only what we needed to run our code
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.5 (2023-01-08)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.8) pkg>
(@v1.8) pkg> activate .
Activating project at `~/Desktop/tontine_2023/stipple_2023`
(stipple_2023) pkg> st
Status `~/Desktop/tontine_2023/stipple_2023/Project.toml`
[336ed68f] CSV v0.10.9
[a93c6f00] DataFrames v1.5.0
[4acbeb90] Stipple v0.26.5
[a3c5d34a] StippleUI v0.22.3
[c2297ded] ZMQ v1.2.2
[ade2ca70] Dates
[56ddb016] Logging
made the code updates you suggested
#https://github.com/GenieFramework/Stipple.jl/discussions/126
# change LOG
# 9/2/22 added LOGGING and SDMOVE calcs TO 8_21_22_stpl_pull.jl to give 9_3_22_stpl_pull.jl
# 9/6/22 enabled LOGGING for test.
# 9/8/22 turned off sd logging
# 9/10/22 added global to model https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/3
# 9/11/22 added pagination
# 9/12/22 added model.isready[] functionality https://github.com/GenieFramework/Stipple.jl/issues/142
# 5/4/23 mkdir "/home/dave/Desktop/tontine_2023/stipple_2023/" pkg generate stp_2023 pkg activate stp_2023 copied 9_11_22_stpl_pull.jl to become this one.
# 5/4/23 added @hhaensel code https://discourse.julialang.org/t/stipple-reactive-data-dashboards-with-julia-wip/37291/105
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "/home/dave/tontine_2022/data/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
#= old 9_11_22_stpl_pull.jl code
@reactive mutable struct TontineModel <: ReactiveModel
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
=#
# NEW code 5/4/23
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title stp_2023 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading stp_2023 5/4/23")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",
pagination=:tontine_data_pagination)
])
])
]
)
end
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
function price_calcs( )
#println( "entering price_calcs " )
row_found = filter(:sym => ==(sym_in),df_table[])
expected_move = round((row_found.iv[1] / 19.896) ,digits = 2)
change = round( (((row_found.price[1] - row_found.close[1] ) / row_found.close[1] ) * 100 ) , digits = 2)
sdmove = round(( change / expected_move ), digits = 2)
isinf(sdmove) && (sdmove = 999.99) # https://discourse.julialang.org/t/float-is-an-inf-and-i-cant-use-to-assign-it-a-new-value/86773/3
try
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = 999.00
end
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
(model = init(TontineModel)) |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) =============================================>>> starting IN Socket 5555"
flush(io)
println(" $(now()) entering main, waiting for message")
#= 9/12/22 https://github.com/GenieFramework/Stipple.jl/issues/142
timeout = 10 # or whatever your startup process needs
t0 = now()
while ! model.isready[] && now() - t0 < timeout
sleep(200)
end
now() - t0 < timeout && model.df[] = my_brand_new_dataframe
or just do
model.isready[] && model.df[] = my_brand_new_dataframe
=#
while true
#@info "============================================================= > entering while loop waiting for message"
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "===========================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
break
catch zmq_error
@info "************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
global in_source, sym_in, field_in , value_in = split( message , "~")
global value_fl = parse(Float64, value_in) # convert to Float64
try
global field_out = zmq_dash[ field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[][!,:sym] || push!(df_table[], (sym_in,0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==(field_out), names(df_table[]))] = value_fl
if field_out == "price"
price_calcs()
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
end
@info "=========================> $(now()) exiting while loop got END "
flush(io)
Your changes are almost correct, except that I didn’t make the module static.
On my computer it runs without problems. Difficult to tell, where your app freezes.
Maybe you open the dev tools and show your console whether there are some errors?
Hi there
this is MY fault. I didn’t point out that the test data I sent has many phases to it. The issue comes when LAST messages are processed. They start at record 449
STK~DIA~LAST~320.16
and carry on from there to the end of the csv. What you show in your clip is the dataframe set up we are using to put in static information.’ You would have to run the whole csv to MAYBE see the problem. I say maybe because it’s erratic. The machine has 128gb ram and about 72 cores running linux mint 20.3 I’d say this isn’t a maching/os problem.
**how do we open dev tools please? if you are referring to firefox dev tools then we have NO skills in that area. the whole reason we are looking at GENIE is to NOT do that."
I don’t know what you mean “I didn’t make the module static”
We don’t know when the app freezes, that’s the issue. There doesn’t seem to be any error in the logs, none raised in the code which keeps running. The web page just freezes until we hit the refresh ( I sent you the log in the last post which shows what happens when we refresh. It carries on for a while ( indeterminant) tnen just stops updating.
Right now this seems to be the same as before so the issue, from our perspective is no better.
static model means
model = init(TontineModel)
route("/") do
global model
model |> handlers |> ui |> html
end
because the model does not change.
Previously it was
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
model = init(TontineModel)
model |> handlers |> ui |> html # = has lowest precedence
end
I never saw any freezing, it’s just that the changes are rather small.
Thank you we’ll swot up on static models.
There is a possibility that this is a problem that builds up over time and using 3000 messages might not cause the problem. Maybe that is why you didn’t see it. BUT we understand what you are saying and will try to build a vanilla linux machine and hook it up to the webpage to see if that is the problem.
if you wanted you could cut and paste the rows from 500 onwards a few times to get more test data. The app would just treat them as messages the values don’t matter so they can be repeated. We could build you more test data if that would help.
We made the changes small to ensure that we stood the best chance of success. Because we are using ZMQ it’s a clockwork mechanism and so less prone to failure.
We ARE seeing freezing but we don’t know when as there are no error messages anywhere. The logs are showing that the data is going into the app and when we run the app in the cli we can see BOTH the ZMQ client and server are working properly thoroughout. So the dataframe is being updated but that activity is not being shown on the screen. We can hit refresh and the screen refreshes for a time but then just stops updating.
We’ve put the changes you asked us to into the cron process and will see what happens. Right now it seems to us that we are seeing the same problem that we had on 9/22/22.
we can switch pretty quickly to pynecone but, as we said, we want to give GENIE the old college try. Thank you for helping us with this.
I refactored your code a bit. Please try it out. I’m on Julia 1.90-rc3.
With this setup I passed up to now more than 6000 messages and the UI is continuously updating.
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "c:/temp/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
function handlers(model)
on(model.isready) do isready
isready || return
model.tontine_data[] = DataTable(df_table[])
end
on(df_table) do new_table
model.isready[] || return
model.tontine_data[] = DataTable(new_table)
end
model
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
model = init(TontineModel)
model |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
# it is good practice to append a '!' to the function name if that function changes the content of one or more variables
function price_calcs!(df, sym_in)
#println( "entering price_calcs " )
row = df[findfirst(==(sym_in), df.sym), :]
expected_move = round(row.iv / 19.896 , digits = 2)
change = round((row.price - row.close) / row.close * 100 , digits = 2)
sdmove = round(change / expected_move, digits = 2)
isinf(sdmove) && (sdmove = 999.99)
try
# row is only a view on the DataFrame, so this updates the DataFrame!
row.sdmove = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
row.sd_move = 999.00
end
end
function receive!(socket, df_table)
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "=================================================================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
return 1
catch zmq_error
@info "*********************************************************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
in_source, sym_in, field_in, value_in = split( message , "~")
value_fl = parse(Float64, value_in) # convert to Float64
field_out = zmq_dash[field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
try
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[].sym || push!(df_table[], (sym_in, 0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), Symbol(field_out)] = value_fl
if field_out == "price"
price_calcs!(df_table[], sym_in)
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
return 0
end
# ---------- Main ---------
empty!(df_table[])
notify(df_table)
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) ===>>> starting IN Socket 5555"
flush(io)
n = 1
while true
print(lpad("$n", 5, '0'), "> ")
receive!(socket, df_table) == 0 || break
n += 1
end
The test server is:
s2 = Socket(PUSH);
connect(s2, "tcp://localhost:5555");
msgs = readlines("C:/temp/test_3000_msg.csv")[2:end];
for n = 1:length(msgs)
send(s2, msgs[n])
sleep(0.05)
end
send(s2, "END")
1 Like
Hi there
sorry we can’t use that refactored code. We used to use sockets and moved onto ZMQ so we can add more complexity later on. We don’t think it will be a fair test.
If we can’t get this working with ZMQ then we can just move on.
We can see your point, if you can’t replicate it then the issue is ours.
We’ll run the code we modded tomorrow and put the results in this thread.
The issue, to us, is that the dataframe is updating ( we can see it happening) , the messages are flowing and being processed ( we can run this in cli and see it )
BUT the part that ISN’T working for us is the bit we are avoiding like the plague.
The web functionality
It’s why we hoped we could use GENIE and Stipple.
Perhaps the best way to solve this, and it’s the one we would take, is to put up a MWE of a ZMQ feed updating a dataframe in the GENIE examples page. Just pick any data feed pump it into a simple ZMQ pipe and represent it in a dataframe you define.