ERROR: LoadError: UndefVarError: @vars not defined
in expression starting at /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:43
in expression starting at /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:43
You have to update Stipple and StippleUI to the latest versions!
And you have to use the handlers function with the one I proposed
if you could run your test against the github data we put at zmq test harness that would help.
We don’t want to try something that isn’t possible. Please remember that we are using ZMQ as the feed as making random data within the script isn’t a fair use case.
We understand that real time no code web representation is a niche case in julia but it is becoming more important.
We can easily change our ZMQ stream to feed to python/dash/pandas ( took 5 minutes) and host the python dash app whilst having a BLAST with @bkamins DataFramesMeta.jl which is a pleasure to use.
if you tell us that you got the test code working with your approach then we’ll set up an new project, move all the existing code into it. Load up the latest versions of requirements and try again.
Just to be clear.
the NEW code would be
#@reactive mutable struct TontineModel <: ReactiveModel
# tontine_data::R{DataTable} = DataTable(df_table[])
#end
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100)
end
model = init(TontineModel)
route("/") do
global model
model |> handlers |> ui |> html
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading Tontine2 8/21/22")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",)
])
])
]
)
end
#function handlers(model)
# on(df_table) do _
# notify(model.tontine_data)
# end
# model
# end
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
Is this correct?.
we made the mods, created a new project, added only the packages we need and made sure they were up to date. julia is also the current version. we ran the code from the cli making sure to use the right project and it “seemed to work” then it just stopped updating the ui as before. ALSO if we hit refresh on the page we get
Info: message : 2023-05-04T16:18:22.788 :
│ message = STK~AAPL~LAST~167.78
└ @ Main /home/dave/Desktop/tontine_2023/stipple_2023/stp_2023/src/stp_2023.jl:198
┌ Error: Error attempting to invoke handler 2 for field Reactive{Bool}(Observable(true), 1, false, false, "") with value true
└ @ Stipple /home/dave/.julia/packages/Stipple/ivn7W/src/stipple/mutators.jl:36
┌ Error:
│ exception = (MethodError(convert, (DataTable, 60×11 DataFrame
Row │ sym close price sdmove hv20 hv10 hv5 iv iv%ile prc%ile ern_days
│ String Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────
1 │ AAPL 167.45 167.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 │ XLP 76.66 76.47 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 │ META 237.03 233.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 │ XBI 83.21 84.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
5 │ GS 328.65 320.9 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
6 │ TSLA 160.61 161.24 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7 │ QCOM 112.83 106.38 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
8 │ NVDA 278.02 275.0 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
9 │ IWM 172.33 170.2 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
10 │ MSFT 304.4 305.42 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
11 │ QQQ 317.29 316.16 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
12 │ DD 64.4 64.06 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
13 │ MA 374.9 375.01 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
14 │ SBUX 103.96 104.76 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
15 │ JPM 135.98 134.12 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
16 │ AMZN 103.65 103.98 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
17 │ GOOGL 105.41 104.5 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
18 │ GOOG 106.12 105.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
19 │ EBAY 44.77 44.38 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
20 │ SPY 408.02 405.0 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
21 │ IYR 83.07 83.74 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
22 │ NFLX 319.3 320.99 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
23 │ WFC 38.35 36.75 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
24 │ V 225.98 225.2 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
25 │ XLV 133.46 132.43 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
26 │ UNG 6.31 6.1 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
27 │ WDAY 183.32 181.92 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
28 │ C 45.67 44.82 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
29 │ IEFA 68.19 68.05 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
30 │ UPS 175.83 173.27 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
31 │ SLV 23.43 23.93 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32 │ IEX 210.1 207.49 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
33 │ XRT 60.33 59.48 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
34 │ VLO 107.06 104.31 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
35 │ XOP 118.05 116.23 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
36 │ XLK 148.25 147.6 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
37 │ LEN 114.13 112.35 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
38 │ XLRE 36.53 36.81 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
39 │ SLB 45.27 45.01 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
40 │ STZ 225.5 222.41 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
41 │ WDC 33.05 33.04 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
42 │ WBA 31.96 31.55 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
43 │ YUM 137.31 136.32 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
44 │ MCD 295.22 296.21 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
45 │ XLF 31.96 31.47 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
46 │ HD 293.08 285.5 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
47 │ GILD 79.45 78.4 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
48 │ TLT 106.29 105.43 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
49 │ VZ 37.98 37.32 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
50 │ WMT 150.05 150.24 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
51 │ JNK 91.45 91.18 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
52 │ HYG 74.58 74.36 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
53 │ XLU 68.07 68.49 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
54 │ DIA 334.11 331.07 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
55 │ GSK 36.84 36.71 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
56 │ GME 18.9 19.42 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
57 │ USO 60.31 60.78 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
58 │ VWO 39.68 40.02 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
59 │ UCO 21.79 21.95 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0
60 │ INDA 40.92 41.25 999.99 0.0 0.0 0.0 0.0 0.0 0.0 0.0), 0x0000000000007f2d), Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x00007fb8569f4c26, Ptr{Nothing} @0x00007fb8569fb535, Ptr{Nothing} @0x00007fb8569fd4ed, Ptr{Nothing} @0x00007fb70015253f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700151f3c, Ptr{Nothing} @0x00007fb70015244a, Ptr{Nothing} @0x00007fb700152485, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb700150ee7, Ptr{Nothing} @0x00007fb700151c86, Ptr{Nothing} @0x00007fb700151cb0, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700150b7a, Ptr{Nothing} @0x00007fb700150bc4, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb70014db83, Ptr{Nothing} @0x00007fb70014dc9f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb700148c5b, Ptr{Nothing} @0x00007fb70014908b, Ptr{Nothing} @0x00007fb7001490ec, Ptr{Nothing} @0x00007fb700149161, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb7001355f3, Ptr{Nothing} @0x00007fb70013562f, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb7134ed744, Ptr{Nothing} @0x00007fb7134ede20, Ptr{Nothing} @0x00007fb7134efcb7, Ptr{Nothing} @0x00007fb7134f4d7f, Ptr{Nothing} @0x00007fb7134f4e22, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a0bc78, Ptr{Nothing} @0x00007fb7134e09b9, Ptr{Nothing} @0x00007fb7134e303a, Ptr{Nothing} @0x00007fb7134e30ef, Ptr{Nothing} @0x00007fb8569fcdbd, Ptr{Nothing} @0x00007fb856a1f4ff])
└ @ Stipple /home/dave/.julia/packages/Stipple/ivn7W/src/stipple/mutators.jl:37
┌ Info: message : 2023-05-04T16:18:23.539 :
│ message = STK~TSLA~LAST~161.25
and the UI updates then freezes.
here’s what we did ( we only had 15 minutes so if you see where we screwed up please tell us)
we created a new project added only what we needed to run our code
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.5 (2023-01-08)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.8) pkg>
(@v1.8) pkg> activate .
Activating project at `~/Desktop/tontine_2023/stipple_2023`
(stipple_2023) pkg> st
Status `~/Desktop/tontine_2023/stipple_2023/Project.toml`
[336ed68f] CSV v0.10.9
[a93c6f00] DataFrames v1.5.0
[4acbeb90] Stipple v0.26.5
[a3c5d34a] StippleUI v0.22.3
[c2297ded] ZMQ v1.2.2
[ade2ca70] Dates
[56ddb016] Logging
made the code updates you suggested
#https://github.com/GenieFramework/Stipple.jl/discussions/126
# change LOG
# 9/2/22 added LOGGING and SDMOVE calcs TO 8_21_22_stpl_pull.jl to give 9_3_22_stpl_pull.jl
# 9/6/22 enabled LOGGING for test.
# 9/8/22 turned off sd logging
# 9/10/22 added global to model https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/3
# 9/11/22 added pagination
# 9/12/22 added model.isready[] functionality https://github.com/GenieFramework/Stipple.jl/issues/142
# 5/4/23 mkdir "/home/dave/Desktop/tontine_2023/stipple_2023/" pkg generate stp_2023 pkg activate stp_2023 copied 9_11_22_stpl_pull.jl to become this one.
# 5/4/23 added @hhaensel code https://discourse.julialang.org/t/stipple-reactive-data-dashboards-with-julia-wip/37291/105
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "/home/dave/tontine_2022/data/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
#= old 9_11_22_stpl_pull.jl code
@reactive mutable struct TontineModel <: ReactiveModel
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
=#
# NEW code 5/4/23
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title stp_2023 ", head_content=Genie.Assets.favicon_support(),
[
heading("heading stp_2023 5/4/23")
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 2500px;",
pagination=:tontine_data_pagination)
])
])
]
)
end
function handlers(model)
on(model.isready) do ready
ready || return
model.tontine_data[] = df_table[]
end
on(df_table) do df_table
model.tontine_data[] = DataTable(df_table)
end
model
end
function price_calcs( )
#println( "entering price_calcs " )
row_found = filter(:sym => ==(sym_in),df_table[])
expected_move = round((row_found.iv[1] / 19.896) ,digits = 2)
change = round( (((row_found.price[1] - row_found.close[1] ) / row_found.close[1] ) * 100 ) , digits = 2)
sdmove = round(( change / expected_move ), digits = 2)
isinf(sdmove) && (sdmove = 999.99) # https://discourse.julialang.org/t/float-is-an-inf-and-i-cant-use-to-assign-it-a-new-value/86773/3
try
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==("sdmove"), names(df_table[]))] = 999.00
end
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
(model = init(TontineModel)) |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) =============================================>>> starting IN Socket 5555"
flush(io)
println(" $(now()) entering main, waiting for message")
#= 9/12/22 https://github.com/GenieFramework/Stipple.jl/issues/142
timeout = 10 # or whatever your startup process needs
t0 = now()
while ! model.isready[] && now() - t0 < timeout
sleep(200)
end
now() - t0 < timeout && model.df[] = my_brand_new_dataframe
or just do
model.isready[] && model.df[] = my_brand_new_dataframe
=#
while true
#@info "============================================================= > entering while loop waiting for message"
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "===========================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
break
catch zmq_error
@info "************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
global in_source, sym_in, field_in , value_in = split( message , "~")
global value_fl = parse(Float64, value_in) # convert to Float64
try
global field_out = zmq_dash[ field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[][!,:sym] || push!(df_table[], (sym_in,0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), findfirst(==(field_out), names(df_table[]))] = value_fl
if field_out == "price"
price_calcs()
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
end
@info "=========================> $(now()) exiting while loop got END "
flush(io)
Your changes are almost correct, except that I didn’t make the module static.
On my computer it runs without problems. Difficult to tell, where your app freezes.
Maybe you open the dev tools and show your console whether there are some errors?
Hi there
this is MY fault. I didn’t point out that the test data I sent has many phases to it. The issue comes when LAST messages are processed. They start at record 449
STK~DIA~LAST~320.16
and carry on from there to the end of the csv. What you show in your clip is the dataframe set up we are using to put in static information.’ You would have to run the whole csv to MAYBE see the problem. I say maybe because it’s erratic. The machine has 128gb ram and about 72 cores running linux mint 20.3 I’d say this isn’t a maching/os problem.
**how do we open dev tools please? if you are referring to firefox dev tools then we have NO skills in that area. the whole reason we are looking at GENIE is to NOT do that."
I don’t know what you mean “I didn’t make the module static”
We don’t know when the app freezes, that’s the issue. There doesn’t seem to be any error in the logs, none raised in the code which keeps running. The web page just freezes until we hit the refresh ( I sent you the log in the last post which shows what happens when we refresh. It carries on for a while ( indeterminant) tnen just stops updating.
Right now this seems to be the same as before so the issue, from our perspective is no better.
static model means
model = init(TontineModel)
route("/") do
global model
model |> handlers |> ui |> html
end
because the model does not change.
Previously it was
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
model = init(TontineModel)
model |> handlers |> ui |> html # = has lowest precedence
end
I never saw any freezing, it’s just that the changes are rather small.
Thank you we’ll swot up on static models.
There is a possibility that this is a problem that builds up over time and using 3000 messages might not cause the problem. Maybe that is why you didn’t see it. BUT we understand what you are saying and will try to build a vanilla linux machine and hook it up to the webpage to see if that is the problem.
if you wanted you could cut and paste the rows from 500 onwards a few times to get more test data. The app would just treat them as messages the values don’t matter so they can be repeated. We could build you more test data if that would help.
We made the changes small to ensure that we stood the best chance of success. Because we are using ZMQ it’s a clockwork mechanism and so less prone to failure.
We ARE seeing freezing but we don’t know when as there are no error messages anywhere. The logs are showing that the data is going into the app and when we run the app in the cli we can see BOTH the ZMQ client and server are working properly thoroughout. So the dataframe is being updated but that activity is not being shown on the screen. We can hit refresh and the screen refreshes for a time but then just stops updating.
We’ve put the changes you asked us to into the cron process and will see what happens. Right now it seems to us that we are seeing the same problem that we had on 9/22/22.
we can switch pretty quickly to pynecone but, as we said, we want to give GENIE the old college try. Thank you for helping us with this.
I refactored your code a bit. Please try it out. I’m on Julia 1.90-rc3.
With this setup I passed up to now more than 6000 messages and the UI is continuously updating.
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "c:/temp/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
function handlers(model)
on(model.isready) do isready
isready || return
model.tontine_data[] = DataTable(df_table[])
end
on(df_table) do new_table
model.isready[] || return
model.tontine_data[] = DataTable(new_table)
end
model
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
model = init(TontineModel)
model |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
# it is good practice to append a '!' to the function name if that function changes the content of one or more variables
function price_calcs!(df, sym_in)
#println( "entering price_calcs " )
row = df[findfirst(==(sym_in), df.sym), :]
expected_move = round(row.iv / 19.896 , digits = 2)
change = round((row.price - row.close) / row.close * 100 , digits = 2)
sdmove = round(change / expected_move, digits = 2)
isinf(sdmove) && (sdmove = 999.99)
try
# row is only a view on the DataFrame, so this updates the DataFrame!
row.sdmove = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
row.sd_move = 999.00
end
end
function receive!(socket, df_table)
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "=================================================================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
return 1
catch zmq_error
@info "*********************************************************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
in_source, sym_in, field_in, value_in = split( message , "~")
value_fl = parse(Float64, value_in) # convert to Float64
field_out = zmq_dash[field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
try
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[].sym || push!(df_table[], (sym_in, 0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), Symbol(field_out)] = value_fl
if field_out == "price"
price_calcs!(df_table[], sym_in)
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
return 0
end
# ---------- Main ---------
empty!(df_table[])
notify(df_table)
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) ===>>> starting IN Socket 5555"
flush(io)
n = 1
while true
print(lpad("$n", 5, '0'), "> ")
receive!(socket, df_table) == 0 || break
n += 1
end
The test server is:
s2 = Socket(PUSH);
connect(s2, "tcp://localhost:5555");
msgs = readlines("C:/temp/test_3000_msg.csv")[2:end];
for n = 1:length(msgs)
send(s2, msgs[n])
sleep(0.05)
end
send(s2, "END")
1 Like
Hi there
sorry we can’t use that refactored code. We used to use sockets and moved onto ZMQ so we can add more complexity later on. We don’t think it will be a fair test.
If we can’t get this working with ZMQ then we can just move on.
We can see your point, if you can’t replicate it then the issue is ours.
We’ll run the code we modded tomorrow and put the results in this thread.
The issue, to us, is that the dataframe is updating ( we can see it happening) , the messages are flowing and being processed ( we can run this in cli and see it )
BUT the part that ISN’T working for us is the bit we are avoiding like the plague.
The web functionality
It’s why we hoped we could use GENIE and Stipple.
Perhaps the best way to solve this, and it’s the one we would take, is to put up a MWE of a ZMQ feed updating a dataframe in the GENIE examples page. Just pick any data feed pump it into a simple ZMQ pipe and represent it in a dataframe you define.
Not sure whether I got it right. But I’m still using the ZMQ socket.
I just feed the PULL socket fron another terminal where I set up a PUSH socket.
The refactoring is in price calc and removal of global vars
Hi there
thanks for the refactoring, a nice approach.
did you run the code? I just ran it ( remember I’m in a meeting working on phone tmux :-)))
got error
ERROR: LoadError: UndefVarError: n not defined
so I just bodged it using a global so as to maintain most of your code.
n = 1
while true
print(lpad("$n", 5, '0'), "> ")
receive!(socket, df_table) == 0 || break
**global n += 1**
end
We can try to put it in place next week as we’re in meetings from now until Monday. We’ll see what happens with the mods we made tomorrow. We’ve got someone to keep an eye on it.
We think the best course, especially for your sanity, is for GENIE to come up with a MWE that accepts ZMQ data and uses it to update a datastructure real time as proof of concept. How about something from yahoo finance? That way you can add to your offering AND we can just mod it.
sound good?
using ZMQ would help in the sale to sensor equipment ( labs), IOT ( manufacturing) and anyone else that wants to have a reliable and easy to use data transport.
hi there @hhaensel
just got msg from person watching screen. On screen froze after 10 minutes. We’ll get your refactored code in place for Monday 5/11/23.
have an excellent weekend.
If the app freezes, please open the DevTools (right-click browser window → Inspect), chose “console” in the top tab and make a screen shot like this:
There’s probably some information on the root cause.
It might be that you have errors in the websocket. I experienced this when I chose a higher update rate.
Is it possible that two events follow very closely one after another?
In that case we might need to throttle the events.
@essenciary Would be great if we could queue the websocket messages.
Hi there
UPDATE. Decided to run the live system from CLI and caught the freeze event. Got the inspect snapshot, hit browser refresh which woke it up took another inspect snapshot, waited a few seconds to see it running and took a running properly inspect snapshot. Hope this is useful.
all snapshots in next message as they didn’t seem to want to upload into this message
can’t do it today BUT we’ll install the code you wrote into the cron process for 5/8/23.Please note I modified your code slightly as the n variable was outside the while loop. I made it global ( I know I know bad bad ) I didn’t want to mess up your code.
Thanks for the instructions about the inspect, we keep ourselves ignorant about all things web browserish, ironic really as our boss funded sun microsystems and formed the first java fund 
In answer to your question. YEP the event timing can alter dramatically. For this functionality we can introduce a delay in the ZMQ if we have to. you might have already introduced a delay with the print statement in your loop. We’ll have to see what happens when we remove it but for now we’ll leave it in.
It’s not a problem as 1 sec for this data isn’t a problem. I think it would be useful to track this down for YOUR purposes so we’ll just let the data flow at the normal rate. We know that the data is flowing correctly, that the dataframe ( in our code and yours) updates effectively. The only thing not working consistently is the model.
have an excellent one
ps here is the code we will be using on 5/8/23 for the run. Please note the global bodge for n in the loop. We know it’s a no no but we didn’t want to mess your code up. Let us know if you have a newer version you want us to try.
using Stipple
using StippleUI
using StipplePlotly
using CSV, DataFrames, Dates , Logging
using ZMQ
log_date = Dates.format(now(),"yyyy_mm_dd_HH_MM")
log_name = "/home/dave/tontine_2022/data/logs/tontine2_log_" * log_date * ".log"
io = open( log_name, "w+")
logger = SimpleLogger(io)
global_logger(logger)
dash_columns = ["sym","close","price","sdmove","hv20","hv10","hv5","iv","iv%ile","prc%ile","ern_days"]
df = DataFrame([col => (col == "sym" ? String : Float64)[] for col in dash_columns])
df_table = Observable(df)
zmq_dash = Dict("LAST" => "price","CLOSE" => "close","OPTION_IMPLIED_VOL" => "iv","OPTION_HISTORICAL_VOL" => "iv",
"VOLUME" => "volume","IV" => "iv","IV_PERCENTILE" => "iv%ile" ,"HV20" => "hv20",
"HV10" => "hv10","HV5" => "hv5" ,"PRICE_PERCENTILE" => "prc%ile","EARNDAYS" => "ern_days")
@vars TontineModel begin
tontine_data::R{DataTable} = DataTable(df_table[])
tontine_data_pagination::DataTablePagination = DataTablePagination(rows_per_page=100) #9/11/22
end
function ui(model::TontineModel)
page(
model, class="container", title="title TONTINE2 ", head_content=Genie.Assets.favicon_support(),
[
heading( "heading Tontine2 5555 9/11/22 from 9_11_22_stpl_pull.jl pag = 100 px = 3000" )
row([
cell(class="st-module", [
h5("h5 tontine data")
table(:tontine_data;
style="height: 3000px;",
pagination=:tontine_data_pagination) # 9/11/22 added pagination
])
])
]
)
end
function handlers(model)
on(model.isready) do isready
isready || return
model.tontine_data[] = DataTable(df_table[])
end
on(df_table) do new_table
model.isready[] || return
model.tontine_data[] = DataTable(new_table)
end
model
end
route("/") do #9/12/22 https://discourse.julialang.org/t/noob-needs-help-debugging-and-understanding-reactive-models/87038/12
global model
model = init(TontineModel)
model |> handlers |> ui |> html # = has lowest precedence
end
up() # up(9000; async = true, server = Stipple.bootstrap())
# it is good practice to append a '!' to the function name if that function changes the content of one or more variables
function price_calcs!(df, sym_in)
#println( "entering price_calcs " )
row = df[findfirst(==(sym_in), df.sym), :]
expected_move = round(row.iv / 19.896 , digits = 2)
change = round((row.price - row.close) / row.close * 100 , digits = 2)
sdmove = round(change / expected_move, digits = 2)
isinf(sdmove) && (sdmove = 999.99)
try
# row is only a view on the DataFrame, so this updates the DataFrame!
row.sdmove = sdmove
catch y
# println("ADDING SDMOVE CATCH something went wrong with sdmove into df_table : ", y)
@info "adding SDMOVE something went wrong error : " y
row.sd_move = 999.00
end
end
function receive!(socket, df_table)
message = String(ZMQ.recv(socket))
@info "message : $(now()) : " message # 9/11/22 https://julialogging.github.io/tutorials/logging-basics/
flush(io)
println("Received request: $message")
if message == "END"
println("dying")
println(df_table)
@info "=================================================================================================================> $(now()) got end dying"
@info df_table
try
ZMQ.close(socket)
ZMQ.close(context)
return 1
catch zmq_error
@info "*********************************************************==> something went wrong with closing zmq sockets => " zmq_error ZMQ.zmq_errno()
end
end
in_source, sym_in, field_in, value_in = split( message , "~")
value_fl = parse(Float64, value_in) # convert to Float64
field_out = zmq_dash[field_in] # ie field_in "OPTION_IMPLIED_VOL" => field_out "iv"
try
#println("message to add : ", in_source," ",sym_in," ",field_out," " ,value_in)
sym_in in df_table[].sym || push!(df_table[], (sym_in, 0.0, 0.0, 0.0, 0.0 , 0.0, 0.0 , 0.0 , 0.0 , 0.0 , 0.0))
df_table[][findfirst(==(sym_in), df_table[].sym), Symbol(field_out)] = value_fl
if field_out == "price"
price_calcs!(df_table[], sym_in)
end
try
notify(df_table)
catch f
#println(" NOTIFY CATCH something went wrong with df_table : ", f)
#println("sym_in :", sym_in," field_out : ", field_out)
@info " $(now()) NOTIFY CATCH something went wrong with df_table :" f
end
catch e
@error "** PROBLEM WITH DF UPDATE > " field_out e
end
return 0
end
# ---------- Main ---------
empty!(df_table[])
notify(df_table)
context = Context()
socket = Socket(context, PULL)
ZMQ.bind(socket, "tcp://*:5555")
@info " $(now()) ===>>> starting IN Socket 5555"
flush(io)
n = 1
while true
print(lpad("$n", 5, '0'), "> ")
receive!(socket, df_table) == 0 || break
global n += 1
end
```
From your UI it looks as the update frequency is not the problem.
If ever the freeze problem continues to exist, I’d put a field in the UI where you write the ZMQ messages to. And I’d add a textfield that updates another field via the server. in order to see wether the server is reactive.
Alternatively, you could add an admin console. I’ll let you know how to to that …
why do that? we can see the ZMQ messages are being processed properly in the log. They get sent and received as per plan. We don’t see ANY data loss.
The dataframe is updated in any of the code variants ( your and ours) so it’s not a logic problem. We don’t log that as it gets updated quite frequently sometimes example ( and this might be slow).
Info: message : 2023-05-01T15:20:13.122 :
│ message = STK~NVDA~LAST~289.28
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:13.622 :
│ message = STK~NVDA~LAST~289.36
so the only thing NOT happening is the representation of the update of the model in stipple.jl. Basically what we saw in 9/22/22.
we don’t want to add anything more to this ( thanks for the admin offer) as we can just switch to python/dash/pandas. We know it works and it’s easy to do. We can wait to see if GENIE puts up a MVP of ZMQ update of a datastructure in stipple and we can mod that.
We’ve done what we said we would do and put your code into the cron process to run on 5/8/23. This will be using live data ( ZMQ
PUSH PULL simplest form no throttle) and we’ve got someone to monitor it.
We are working to the principal that your code is the best shot to get stipple.jl functionality working and we are deploying your code. The ZMQ feeds are working exactly as we expect, bogumil’s DataFrames.jl works in either code base ( yours and ours) so that’s not the problem.
we’ll let you know what happened on 5/8/23 and will provide the inspect screen shot as requested should it freeze again.
here is an example of the way we log the ZMQ traffic inside the stipple script.
Info: message : 2023-05-01T15:20:07.113 :
│ message = STK~SPY~LAST~415.61
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:10.369 :
│ message = STK~SBUX~LAST~114.59
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:13.122 :
│ message = STK~NVDA~LAST~289.28
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:13.622 :
│ message = STK~NVDA~LAST~289.36
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:14.373 :
│ message = STK~IEFA~LAST~68.66
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:17.127 :
│ message = STK~QQQ~LAST~322.19
└ @ Main /home/dave/tontine_2022/2022_live/9_11_22_stpl_pull.jl:151
┌ Info: message : 2023-05-01T15:20:23.386 :
│ message = STK~IEFA~LAST~68.67