Update Genie webpage in a asynchrounous loop

yakir12 · February 22, 2021, 2:34pm

I’m trying to update an image in an asynchronously loop in a Genie (Stipple) app. I’ve tried different approaches, none successful. I’ve therefore created the following MWE (based on your solution to my last question, @hhaensel):

# ] add Genie, StippleUI#master, Stipple
using Stipple, StippleUI
using Genie.Renderer.Html

const IMGPATH = "img/demo.png" # make sure there is an image in public/img/demo.png

Base.@kwdef mutable struct Dashboard1 <: ReactiveModel
    imageurl::R{String} = IMGPATH
    restart::R{Bool} = false # restart the updating of the images
end

keep_updating = Ref(true) # a container for stopping the previous task

function restart()
    global model

    cd(@__DIR__)
    model = Stipple.init(Dashboard1(), debounce=1)

    on(model.restart) do x
        keep_updating[] = false # this stops the previous task
        sleep(2) # wait for enough time so that the previous task can iterate at least once to register the difference in the value of `keep_updating`
        keep_updating[] = true # prepare for the next task
        @async while keep_updating[] # start a new task for updating the image url
            model.imageurl[] = "/$IMGPATH#$(Base.time())"
            sleep(1) # sleep to allow the image to load at the client's end
        end
    end
end

function ui()
    dashboard(vm(model), [
        heading("Image Demo"),

        row(cell(class="st-module", [
            btn("New Image", "", @on("click", "restart=true")),
            quasar(:img, src=:imageurl, spinner__color="white", style="height: 140px; max-width: 150px")
        ]))
    ], title = "Image Demo") |> html
end

route("/", ui)

Genie.config.server_host = "127.0.0.1"

restart()
up(open_browser = true)

But this (and many of my other attempts) does not work and just report that

┌ Error: Base.IOError("stream is closed or unusable", 0)
└ @ Genie.WebChannels ~/.julia/packages/Genie/SN5kE/src/WebChannels.jl:223

So I’m obviously doing something wrong.

I feel there is an important distinction to be made here:
Normal UI updates the state of the webpage (e.g. update an image) as a result of an action by the user (e.g. pressing a button). What I’m trying to achieve is a loop of updates (albeit, that starts from pressing some “start” button), each update event (iteration) elicited by the server, not the user. Any ideas on what the Genie/Stipple way of achieving that is?

hhaensel · February 22, 2021, 3:01pm

The error message is most probably due to the fact, that you have restarted and opened a new tab or window. Genie stores all clients that have ever connected and does not delete them, if they do not disconnect themselves. You can check this by entering

Genie.WebChannels.subscriptions()

When you now update a value, Genie tries to broadcast the change to all clients that have not disconnected. But some of them may be dead, because you closed the tab and the WebChannel was not open.
There are two possible solutions.

Brute force
Kill all subscriptions, e.g. in restart():
Genie.WebChannels.pop_subscription.(keys(Genie.WebChannels.subscriptions()))
and reload the page in the browser.
Define a routine to check, which client is alive and delete only that client’s subscription:

function remove_dead_clients(timer)
  for (ch, clients) in Genie.WebChannels.subscriptions()
    for cl in clients
      try
        Genie.WebChannels.message(cl, "")
      catch ex
        @info "removing dead client: $cl from $ch"
        Genie.WebChannels.pop_subscription(cl, ch)
      end
    end
  end
end

In restart you could define a timer, which regularly checks for non-responsive clients:

# at the top
timer = Timer(1.0)

# in restart()
  global timer
  close(timer)
  timer = Timer(remove_dead_clients, 1; interval=60)

That should keep out the error message.
UPDATE: changed the definition of the timer ‘constant’

yakir12 · February 22, 2021, 8:16pm

(just found out about Genie.WebChannels.unsubscribe_disconnected_clients() which might be simpler than your suggestion)

I’ve come to realize that the way I’m thinking about it is a bit wrong (I’m very new to web dev stuff, so bare with the naivety):
Instead of a server pushing updates to all its clients (connected or not), I should have the server listen to a clients’ request for a new image, send the requested image, the client then “renders” the image (i.e. load and show the image), and after successfully displaying the image, request for another one. If the client disappears, then the server doesn’t get a request for a new image.
I think that is the best way to think about it. It will allow for multiple clients to be connected to the same server, and the server will be able to cater to each client individually.

So… I suspect my answers are in Advanced routing · Genie - The Highly Productive Julia Web Framework but correct me if I’m way off

Thanks!

yakir12 · February 22, 2021, 9:28pm

OK, I think I need a way for Vue to tell the server that the image is now finished loading on the client side, something like vue-images-loaded - npm
Is there a way to achieve that? I’m assuming that the ideal thing would be to add some model parameter to the quasar call:

Base.@kwdef mutable struct Dashboard1 <: ReactiveModel
    imageurl::R{String} = IMGPATH
    finished_loading::R{Bool} = true
end
...
on(model.finished_loading) do tf
    <push a new image>
end
...
quasar(:img, src=:imageurl, spinner__color="white", style="height: 140px; max-width: 150px", isLoaded=:finished_loading)
...

hhaensel · February 22, 2021, 10:25pm

That is correct. That is why I added the button logic iin my example.
Simply eliminate the button and place finished_loading=true after your client is ready.

In your code don’t forget to check whether the change went from false to true and to set the value back to false at the end.

hhaensel · February 22, 2021, 10:29pm

about Genie.WebChannels.unsubscribe_disconnected_clients()

Well, that’s what I also hoped for. But I realised that it can happen that clients go offline without unsubscribing. And then they stay in the list. I had this problem very often with mobile phones. I think they simly go offline and cannot be reached. In that case Genie doesn’t cut the connection but tries to update them. That’s where the error occurs.
So I decided to clean the list every 60s.

yakir12 · February 23, 2021, 10:16am

Thank you for your useful input, this is great!

If I understood you correctly, I have two main issues with this approach:

First and foremost, your suggestion is similar to my previous attempt in that it’s the server that blindly pushes updates to unwilling clients, instead of clients asking the server for the next image after and only after finishing to upload the previous one. The loop has been moved from a literal asynchronous loop to a loop of Observable events.

My second issue is that updating the image-url is asynchronous:

julia> @btime model.imageurl[] = "/$IMGPATH#$(Base.time())"
  19.168 μs (69 allocations: 4.34 KiB)

That is, the amount of time it takes to execute the above line is significantly shorter than the amount of time it takes the client to finish loading the image. And therefore I cannot reliably

because I don’t know when the client is ready for another push. If, however, I place an artificial pause after updating the image-url, then it “works”. But then the updates are both artificially throttled and are being pushed regardless of the client’s state or (lack of) request.

Here’s my attempt at implementing your suggestion:

using Stipple, StippleUI
using Genie.Renderer.Html
using Colors
using FileIO

const IMGPATH = "img/demo.png"

Base.@kwdef mutable struct Dashboard1 <: ReactiveModel
    imageurl::R{String} = IMGPATH
    finished_loading::R{Bool} = false
end

function restart()
    global model

    cd(@__DIR__)
    model = Stipple.init(Dashboard1(), debounce=1)

    on(model.finished_loading) do x
        x == false && return nothing

        # get and save the new image
        image = rand(RGB, 100, 100)
        save(joinpath(@__DIR__, "public", IMGPATH), image)

        model.finished_loading[] = false
        # add a time anchor to the imagepath in order to trigger a reload in the backend
        model.imageurl[] = "/$IMGPATH#$(Base.time())"
        sleep(1)

        model.finished_loading[] = true
    end
end

function ui()
    dashboard(vm(model), [
        heading("Image Demo"),

        row(cell(class="st-module", [
            btn("Start", "", @on("click", "finished_loading=true"))
            quasar(:img, src=:imageurl, spinner__color="white", style="height: 140px; max-width: 150px")
        ])),

        footer(class="st-footer q-pa-md", cell([
            img(class="st-logo", src="/img/st-logo.svg"),
            span(" &copy; 2020 - Happy coding ...")
        ]))
    ], title = "Image Demo") |> html
end

route("/", ui)

Genie.config.server_host = "127.0.0.1"

restart()
up(open_browser = true)

hhaensel · February 23, 2021, 10:39pm

Hi Yakir,
I missed some part of the question, sorry. You were quite close with your guessing. The correct implementation is @on(load, "newimage=true").
Note: you have to add an additional empty string argument, because the first non-keyword argument is always placed between the opening and closing bracket of the element. (The @on macro produces a regular argument, try it out at the REPL.)
So your quasar element would look like:

quasar(:img, "", src=:imageurl, spinner__color="white", @on(load, "finished_loading=true"))

yakir12 · February 23, 2021, 11:00pm

This worked!!! Awesome. Thank you so much.

Right, any way I can remove the animation that causes the image to bleach to white between every reload?

I can see how it’s helpful in indicating a reload, but the way I use it, might as well remove it.

hhaensel · February 23, 2021, 11:02pm

Just another note: Whenever you want to add an argument to a quasar or vue element, you can always do that by adding the pure js syntax from the examples in the internet in quotes, e.g.

quasar(:img, "", src=:imageurl,  "@load='finished_loading=true'")

This also comes in handy for boolean keywords, which are often added without “=true” in JS syntax. The julian syntax would then be “no-default-spinner” instead of no__default__spinner=true. (Double underscores in kwargs are replaced by a hyphen.)

If no hyphens are in the kwarg, then you can also use the Symbol notation, e.g. :position.

hhaensel · February 23, 2021, 11:06pm

add :basic to the arguments.
Such info can best be found in the quasar docs

yakir12 · February 23, 2021, 11:21pm

Did it. Oh my, thank you soooo much Helmut!!!

hhaensel · February 24, 2021, 12:18am

Welcome!
And welcome in the Stipple community. I can only encourage you to spend a demo to our demo collection once you have something nice up and running

hhaensel · February 24, 2021, 12:57am

The server doesn’t push without being asked, it only tries to keep all connected clients at the same state. So once a variable is changed by either the server or one of the clients, all clients are updated. This is what you would expect from an app, wouldn’t you? If you want two instances of the same app then there is multi-user support in Stipple.
Different instances are distributed via different channels. And many clients can be linked to the same instance. The programmer is responsible for doing this in the route() command. There you can access the headers and cookies.

yakir12 · February 24, 2021, 9:46am

I’ve discovered that the chain:

client side updates image → @on("load", "finished_loading=imageurl") triggers an update to imageurl → which causes the client to update the image again

gets broken after a few moments (i.e. the image doesn’t update any longer). It seems unlikely that the loop here is faster than two consecutive calls to Base.time() (especially the way I’ve set it up, see below). Note that reloading the page “fixes” the problem, until it stalls again.

I’ll just add that I think it might be much simpler and more robust to have the loop:

get new image from the server → display it → repeat

on the client side with some simple JS. This way, the client simply updates the image as and when it needs. The server simply has a route in place to supply the image when asked for it. I tried implementing that (with considerable help from a non-Julian but webdev friend), and will continue to try that as well.

Here is the problematic MWE. You’ll need a webcam (but I can rewrite it to avoid that). As I said, if you run this you’ll notice that the image stops updating after a minute or two:

# ] activate --temp
# ] add Genie Stipple StippleUI FFMPEG_jll
using Stipple, StippleUI
using Genie.Renderer.Html
using FFMPEG_jll # to get frames from the webcam

mkpath("public/img") # prepare the directories for the images
const IMGPATH = "img/demo.png"
const SZ = 240 # width and height of the images
const FPS = 10 # frames per second
ffmpeg() do exe # record from camera a frame, rewriting to `IMGPATH` 10 (= `FPS`) times a second
    run(`$exe -y -hide_banner -loglevel error -f v4l2 -video_size $(SZ)x$SZ -i /dev/video0 -r $FPS -update 1 public/$IMGPATH`, wait = false)
end

# this here is to guarantee that the printed timestamp is unique between each iteration
const T₀ = Base.time()
timestamp() = string(Base.time() - T₀)

Base.@kwdef mutable struct Dashboard1 <: ReactiveModel
    imageurl::R{String} = IMGPATH
    finished_loading::R{String} = IMGPATH*"random" # this is used mainly as a timestamp to trigger loading a new image
end

function restart()
    global model

    model = Stipple.init(Dashboard1(), debounce=1)

    on(model.finished_loading) do finished_loading
        sleep(1/FPS) # I added this as a means to make sure that two sequential `model.finished_loading` are different
        model.imageurl[] = "/$IMGPATH#$(timestamp())"
    end
end

function ui()
    dashboard(vm(model), [
        heading("SkyRoom"),

        row(cell(class="st-module", [
            quasar(:img, "", src=:imageurl, :basic, style="height: $(SZ)px; max-width: $(SZ)px", @on("load", "finished_loading=imageurl"))
        ]))
    ], title = "SkyRoom") |> html
end

route("/", ui)

Genie.config.server_host = "127.0.0.1"

restart()
up(open_browser = true)

Should I submit an issue?

yakir12 · February 24, 2021, 12:14pm

OK, the following gets the client to update the image on a loop:

route("frame") do
    """
        <!doctype html>
        <html>
        <head>
        
        <script type="text/javascript">
        
            setInterval(function() {
                var img = document.getElementById("frame");
                img.src = "/img/frame.png#" + new Date().getTime();
            }, 100);
        
        </script>
        
        </head>
        <body>
        
        <img id="frame" src="img/frame.png" />
        
        </body>
        </html>
    """
end

All the server needs to do is update the img/frame.png file in the background (using ffmpeg like I did in the previous post).

This has the advantage of letting the client control the loop. It seems a lot more robust than the other method.

My question is, how exactly do I use this (a route with a JS function and a simple img (note the use of an id)) in the whole Stipple SPA thing…?

yakir12 · February 24, 2021, 3:07pm

The saga continues…
So I think this is somewhat right, but it doesn’t work. I’m not sure if the id tag to img doesn’t take (due to Vue:ism) or if I’m totally off the mark:

function ui()
    dashboard(vm(model), [
        script(
               """
                       setInterval(function() {
                       var img = document.getElementById("frame");
                       img.src = "$IMGPATH#" + new Date().getTime();
                       }, 100);
               """
              ),
        heading("Image Demo"),
        row(cell(class="st-module", [
            quasar(:img, id = "frame", src = IMGPATH, style="height: 140px; max-width: 140px")
        ]))
    ], title = "Image Demo") |> html
end

hhaensel · February 26, 2021, 12:16am

Only very shortly: It is good practice to define the script in the js_methods in order to access the properties of your dashboard. You should definitely have a read through the vue.js basics and understand properties and methods. (I also did not know about it, before I jumped on Stipple after it was published.)
The JS side has a very similar object like Julia, and it is also called Dashboard1 in your case.
Accessing your Dashboard in JS would be Dashboard1.imageurl = "/img/frame.png#" + new Date().getTime(); - so very similar as in Julia - and Vue.js will make the rest, because Vue watches your changes to Dashboard1 and changes your ui accordingly.

Good luck

P.S.: I also had the problem of Julia stopping image transmission. Adding a small sleep() solved the issue.

yakir12 · February 28, 2021, 10:07am

I’d love to, but the only reference to js_methods I could find in GenieFramework was here:

github.com

GenieFramework/Stipple.jl/blob/4c158f3814eb7d6fc085d50d13e8934c980c0fad/src/Stipple.jl#L47


      
          
          const JS_SCRIPT_NAME = "stipple.js"
          const JS_DEBOUNCE_TIME = 300 #ms
          
          #===#
          
          function render end
          function update! end
          function watch end
          
          function js_methods(m::Any)
            ""
          end
          
          #===#
          
          const COMPONENTS = Dict()
          
          function register_components(model::Type{M}, keysvals::Vector{Pair{K,V}}) where {M<:ReactiveModel, K, V}
            haskey(COMPONENTS, model) || (COMPONENTS[model] = Pair{K,V}[])
            push!(COMPONENTS[model], keysvals...)

so I have no idea how to follow your suggestion.

I tried skimming some of the documentation in Introduction | Vue.js. Do you have any more specific reads in mind? How much Vue.js do I need to learn to be able to use Genie+Stipple?

This is still difficult for me to implement, but I might get there after more attempts and reading. Thank you for the tip.

Heh, weird, adding sleep(t) (with various t values) didn’t solve it for me. Let me highlight though that the drop happens only after a minute or 10, so you might have missed it. Try and see if it hangs after say 20 minutes.

Summary

At least in terms of functionality (but as you indicated above, might not be totally cosher), I did get it to work flawlessly by adding a script to the dashboard call:

function ui()
    dashboard(vm(model), [
                          script(
                                 """
                                 setInterval(function() {
                                 var img = document.getElementById("frame");
                                 img.src = "frame/" + new Date().getTime();
                                 }, $(1000 ÷ FPS));
                                 """
                                ),        
                          heading("SkyRoom"),
                          row(cell(class="st-module", [
                                                       """
                                                       <img id="frame" src="frame" style="height: $(SZ)px; max-width: $(SZ)px" />
                                                       """
                                                      ])),
                         ], title = "SkyRoom") |> html
end

Where FPS and SZ are the frames per second and size of the images (controlling also the ffmpeg generating the images).

Thank you for your constant input!