Sending a message to webpage on event

Specific Domains Web Stack
4

Hi @tlienart :wave:

It kind of bummed me out to hear what you said about Pages.jl :sweat_smile:

I admit, the documentation sucks though :sweat_smile:

It may not be exactly what you wanted, but I just pushed a new version of Pages.jl that includes your example. To get it, for now, you will need the master branch, i.e.

pkg> add Pages#master

I just noticed, it looks like tests are failing so I’ll need to sort that out, but that shouldn’t affect what I say below :pray:

At this points, Pages.jl is really just some sugar on top of HTTP.jl so I can walk through what Pages.jl is doing and show you what it is doing with HTTP.jl.

Pages.start()

First is the server. You can see what theserver is doing by looking at src/server.jl:

function start(p=8000)

    Endpoint("/pages.js",GET) do request::HTTP.Request
        read(joinpath(@__DIR__,"pages.js"),String)
    end

    HTTP.listen(ip"0.0.0.0", p, readtimeout=0) do http
        route = lowercase(HTTP.URI(http.message.target).path)
        if haskey(endpoints,route)
            if HTTP.WebSockets.is_upgrade(http.message)
                HTTP.WebSockets.upgrade(http) do client
                    while !eof(client);
                        data = String(readavailable(client))
                        msg = JSON.parse(data)
                        name = pop!(msg,"name")
                        route = pop!(msg,"route")
                        id = pop!(msg,"id")
                        if haskey(callbacks,name)
                            callbacks[name].callback(client,route,id,msg)
                        end
                    end
                end
            else
                e = endpoints[route]
                m = Symbol(uppercase(http.message.method))
                HTTP.handle(e.handlers[m],http)
            end
        else
            HTTP.handle(HTTP.Handlers.FourOhFour,http)
        end
    end
    return nothing
end

The main point of Pages.jl is to provide a simple link between Julia and a browser and it does this behind the scenes using websockets to communicate between Julia and JavaScript.

The first thing it does is create an Endpoint so the browser will know where to find pages.js, the small JavaScript library that handles communication from the browser side.

Endpoint("/pages.js",GET) do request::HTTP.Request
    read(joinpath(@__DIR__,"pages.js"),String)
end

An Endpoint is just some sugar to keep track of the url of the page and holds some handlers, which are from HTTP.RequestHandlerFunction.

struct Endpoint
    handlers::Dict{Symbol,HTTP.RequestHandlerFunction}
    route::String
end

One url (or route) can have multiple handlers corresponding to the different HTTP request types, i.e. GET, POST, HEAD, PUT, DELETE, etc. The default method is GET so most of the time, you don’t need to worry about it.

The constructor for an Endpoint is intended to be clear and intuitive so I hope it is :sweat_smile::pray:

Endpoint("/your/url") do request::HTTP.Request
    # Do whatever you want with the request here and return either an 
    # `HTTP.Response` or a `String`. Most of the time, I just return a `String`
    # and let HTTP.jl handle the rest.
end

It is not uncommon to want to handle a POST requests so to do that you need to specify the method

Endpoint("/your/url",POST) do request::HTTP.Request
    # Handle the `POST` request.
end

Next, the server is started on a specified port p:

HTTP.listen(ip"0.0.0.0", p, readtimeout=0) do http

When a request is received, it extracts the route from the request

route = lowercase(HTTP.URI(http.message.target).path)

but note that http is not actually an HTTP.Request. Rather, it is an HTTP.Stream. That is why we need to reach into it to extract the actual request, which is http.message and then from that, get the path.

Then it checks a dictionary to see if the route is valid. The Endpoint constructor will add the url to a global dictionary called Pages.endpoints. This deviates slightly from HTTP.jl. This is actually the original design of Pages.jl several years ago, but I switched to using HTTP.jl’s router a while back, but recently switched back to looking up routes myself. I like this better because we can experiment with adding and removing paths even while the server is still running.

If the requested route is a valid url, then it goes into the usual checking to see if the request is a websocket request or a normal HTTP request. This is straight out of HTTP.jl’s documentation, but notice it supports callbacks:

if haskey(callbacks,name)
    callbacks[name].callback(client,route,id,msg)
end

A callback is a way to have the browser execute pre-defined Julia code.

struct Callback
    callback::Function
    name::String

    function Callback(callback,name)
        cb = new(callback,name)
        callbacks[name] = cb
        cb
    end
end
const callbacks = Dict{String,Callback}() # name => args -> f(args...)

An example of an important callback is

# Callback to notify the Server that a new page is loaded 
# and its WebSocket is ready.
Callback("connected") do client, route, id, data
    if haskey(connections,route)
        connections[route][id] = client
    else
        connections[route] = Dict(id => client)
    end
    println("New connection established (ID: $(id)).")
    notify(conditions["connected"])
end

When your web page (that includes the pages.js library) is loaded and the websocket successfully connects to the Julia server, it will send a notification telling Julia it is ready to start communicating.The message includes a locally generated uuid so that you can communicate with specific browsers. There are two ways to send messages back and forth between Julia and the browser:

  1. Message: Requires a uuid for the specific client
  2. Broadcast: Sends the message to all open connections

Most of the time, I am just fiddling with things myself, so broadcast is fine.

The “connected” callback is handled on the JavaScript side with the following command:

sock.onopen = function() {
    callback("connected");
};

As you can guess, when the websocket is connected and “open” for communication, it send the “connected” callback to Julia.

From there, Julia records the id and the client connection in a global dict connections.

Another useful callback is

# Callback used to cleanup when the browser navigates away from the page.
Callback("unloaded") do client, route, id, data
    delete!(connections[route],id)
end

This probably needs some work, but the idea is that when the webpage is refreshed or the client navigates away, it sends a callback letting Julia know.

Then there is another kind of callback coming from

"""
Block Julia control flow until until callback["notify"](name) is called.
"""
function block(f::Function,name)
    conditions[name] = Condition()
    f()
    wait(conditions[name])
    delete!(conditions,name)
    return nothing
end

Since Julia is synchronous (and fast!) and JavaScript is asynchrous, sometimes you want Julia code execution to wait for JavaScript to complete. If you call block with a name name, Julia will wait until it receives a notification from JavaScript with the same name before continuing execution.

One example I like that illustrates this is

"""Add a JS library to the current page from a url."""
function add_library(url)
    name = basename(url)
    block(name) do
        Pages.broadcast("script","
            var script = document.createElement('script');
            script.charset = 'utf-8';
            script.type = 'text/javascript';
            script.src = '$(url)';
            script.onload = function() {
                Pages.notify('$(name)');
            };
            document.head.appendChild(script);
        ")
    end
end

This method takes the url for some JavaScript library, e.g. d3.js or plotly.js, and injects it into the page even after the page is already loaded. Julia execution is halted until the library is loaded into the browser and ready to use via the line:

script.onload = function() {
    Pages.notify('$(name)');
};

Getting back to server.jl, callbacks only come into play if the request received is a websocket request. Otherwise, we go to

e = endpoints[route]
m = Symbol(uppercase(http.message.method))
HTTP.handle(e.handlers[m],http)

This code extracts the Endpoint from the global dictionary endpoints, determines what kind of HTTP request it is, i.e. GET, POST or whatever, and then uses HTTP.handle to handle the request.

So it is really, just some sugar on top of HTTP.jl, but I think it is simple and convenient sugar :sweat_smile:

HTML Elements

Finally, there is a completely undocumented, but super useful functionality in Pages.jl, i.e.

mutable struct Element
    id::String
    tag::String
    name::String
    attributes::Dict{String,String}
    style::Dict{String,String}
    innerHTML::String
    parent_id::String
end

Note: I significantly changed this API just now because an earlier version depended on d3.js, which isn’t really necessary so I rewrote it to just use vanilla JS.

Although, not documented, it was used in an earlier Plotly example, but I recently moved the Plotly example to another package (Figures.jl).

This is a generic HTML element that contains info for creating browser elements on the fly from Julia.

The type of element, e.g. div, li, select, input, etc, is specified by the tag.

There are two main methods for modifying the browser DOM from Julia

  1. appendChild
  2. removeChild

Putting it all together to implement your example is now pretty simple.

Simply start with

julia> using Pages; Pages.examples();

This will start a local server and open a page (launching a browser if one isn’t open or creating a new tab). If you don’t want it to launch the page or open a new tab, you can run the example with

julia> using Pages; Pages.examples(launch=false);

but then you will need to manually navigate to http://localhost:8000/examples.

At the moment, there are only two simple examples there:

  1. Requests - Illustrates how to implement POST requests and GET requests with parameters in the url.
  2. Blank - Just a blank page with pages.js loaded and ready for experimenting.

For your example, click “Blank” to go to the blank page.

Here is the code for your example:

function randomping()
    count = 1
    t0 = time()
    while true
        sleep(1)
        e = Element(string(count),innerHTML = 
                "Ping $(count): $(round(time()-t0,digits=2)) seconds")
        e.style["color"] = rand(["red","green","blue"])
        rand() < .5 && Pages.appendChild(e)
        count += 1
        tstart = time()
    end
end

This is basically an infinite loop that executes every 1 second.

Each loop, creates a new div (default tag is div so you do not need to specify)

e = Element(string(count),innerHTML = 
        "Ping $(count): $(round(time()-t0,digits=2)) seconds")

The div id is set to the count number and the displayed text (innerHTML) contains the count number and the total elapsed time from when the loop was started (to show it is roughly 1 second).

Just for fun, I also have it select a random color:

e.style["color"] = rand(["red","green","blue"])

Then, like you suggested, if a random number is less than 0.5, it will add a new div to the web page:

rand() < .5 && Pages.appendChild(e)

Well, that was a long post! :sweat_smile:

I hope you find it helpful and whether or not you end up using Pages.jl, I think it should probably help with whatever you are trying to do by looking through some of the code. If you have questions, don’t hesitate to ask. I’m happy to help if I can.

Cheers!

5 Likes