Hi @tlienart 
It kind of bummed me out to hear what you said about Pages.jl 
I admit, the documentation sucks though
It may not be exactly what you wanted, but I just pushed a new version of Pages.jl that includes your example. To get it, for now, you will need the master branch, i.e.
pkg> add Pages#master
I just noticed, it looks like tests are failing so I’ll need to sort that out, but that shouldn’t affect what I say below 
At this points, Pages.jl is really just some sugar on top of HTTP.jl so I can walk through what Pages.jl is doing and show you what it is doing with HTTP.jl.
Pages.start()
First is the server. You can see what theserver is doing by looking at src/server.jl:
function start(p=8000)
Endpoint("/pages.js",GET) do request::HTTP.Request
read(joinpath(@__DIR__,"pages.js"),String)
end
HTTP.listen(ip"0.0.0.0", p, readtimeout=0) do http
route = lowercase(HTTP.URI(http.message.target).path)
if haskey(endpoints,route)
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http) do client
while !eof(client);
data = String(readavailable(client))
msg = JSON.parse(data)
name = pop!(msg,"name")
route = pop!(msg,"route")
id = pop!(msg,"id")
if haskey(callbacks,name)
callbacks[name].callback(client,route,id,msg)
end
end
end
else
e = endpoints[route]
m = Symbol(uppercase(http.message.method))
HTTP.handle(e.handlers[m],http)
end
else
HTTP.handle(HTTP.Handlers.FourOhFour,http)
end
end
return nothing
end
The main point of Pages.jl is to provide a simple link between Julia and a browser and it does this behind the scenes using websockets to communicate between Julia and JavaScript.
The first thing it does is create an Endpoint so the browser will know where to find pages.js, the small JavaScript library that handles communication from the browser side.
Endpoint("/pages.js",GET) do request::HTTP.Request
read(joinpath(@__DIR__,"pages.js"),String)
end
An Endpoint is just some sugar to keep track of the url of the page and holds some handlers, which are from HTTP.RequestHandlerFunction.
struct Endpoint
handlers::Dict{Symbol,HTTP.RequestHandlerFunction}
route::String
end
One url (or route) can have multiple handlers corresponding to the different HTTP request types, i.e. GET, POST, HEAD, PUT, DELETE, etc. The default method is GET so most of the time, you don’t need to worry about it.
The constructor for an Endpoint is intended to be clear and intuitive so I hope it is
Endpoint("/your/url") do request::HTTP.Request
# Do whatever you want with the request here and return either an
# `HTTP.Response` or a `String`. Most of the time, I just return a `String`
# and let HTTP.jl handle the rest.
end
It is not uncommon to want to handle a POST requests so to do that you need to specify the method
Endpoint("/your/url",POST) do request::HTTP.Request
# Handle the `POST` request.
end
Next, the server is started on a specified port p:
HTTP.listen(ip"0.0.0.0", p, readtimeout=0) do http
When a request is received, it extracts the route from the request
route = lowercase(HTTP.URI(http.message.target).path)
but note that http is not actually an HTTP.Request. Rather, it is an HTTP.Stream. That is why we need to reach into it to extract the actual request, which is http.message and then from that, get the path.
Then it checks a dictionary to see if the route is valid. The Endpoint constructor will add the url to a global dictionary called Pages.endpoints. This deviates slightly from HTTP.jl. This is actually the original design of Pages.jl several years ago, but I switched to using HTTP.jl’s router a while back, but recently switched back to looking up routes myself. I like this better because we can experiment with adding and removing paths even while the server is still running.
If the requested route is a valid url, then it goes into the usual checking to see if the request is a websocket request or a normal HTTP request. This is straight out of HTTP.jl’s documentation, but notice it supports callbacks:
if haskey(callbacks,name)
callbacks[name].callback(client,route,id,msg)
end
A callback is a way to have the browser execute pre-defined Julia code.
struct Callback
callback::Function
name::String
function Callback(callback,name)
cb = new(callback,name)
callbacks[name] = cb
cb
end
end
const callbacks = Dict{String,Callback}() # name => args -> f(args...)
An example of an important callback is
# Callback to notify the Server that a new page is loaded
# and its WebSocket is ready.
Callback("connected") do client, route, id, data
if haskey(connections,route)
connections[route][id] = client
else
connections[route] = Dict(id => client)
end
println("New connection established (ID: $(id)).")
notify(conditions["connected"])
end
When your web page (that includes the pages.js library) is loaded and the websocket successfully connects to the Julia server, it will send a notification telling Julia it is ready to start communicating.The message includes a locally generated uuid so that you can communicate with specific browsers. There are two ways to send messages back and forth between Julia and the browser:
- Message: Requires a uuid for the specific client
- Broadcast: Sends the message to all open connections
Most of the time, I am just fiddling with things myself, so broadcast is fine.
The “connected” callback is handled on the JavaScript side with the following command:
sock.onopen = function() {
callback("connected");
};
As you can guess, when the websocket is connected and “open” for communication, it send the “connected” callback to Julia.
From there, Julia records the id and the client connection in a global dict connections.
Another useful callback is
# Callback used to cleanup when the browser navigates away from the page.
Callback("unloaded") do client, route, id, data
delete!(connections[route],id)
end
This probably needs some work, but the idea is that when the webpage is refreshed or the client navigates away, it sends a callback letting Julia know.
Then there is another kind of callback coming from
"""
Block Julia control flow until until callback["notify"](name) is called.
"""
function block(f::Function,name)
conditions[name] = Condition()
f()
wait(conditions[name])
delete!(conditions,name)
return nothing
end
Since Julia is synchronous (and fast!) and JavaScript is asynchrous, sometimes you want Julia code execution to wait for JavaScript to complete. If you call block with a name name, Julia will wait until it receives a notification from JavaScript with the same name before continuing execution.
One example I like that illustrates this is
"""Add a JS library to the current page from a url."""
function add_library(url)
name = basename(url)
block(name) do
Pages.broadcast("script","
var script = document.createElement('script');
script.charset = 'utf-8';
script.type = 'text/javascript';
script.src = '$(url)';
script.onload = function() {
Pages.notify('$(name)');
};
document.head.appendChild(script);
")
end
end
This method takes the url for some JavaScript library, e.g. d3.js or plotly.js, and injects it into the page even after the page is already loaded. Julia execution is halted until the library is loaded into the browser and ready to use via the line:
script.onload = function() {
Pages.notify('$(name)');
};
Getting back to server.jl, callbacks only come into play if the request received is a websocket request. Otherwise, we go to
e = endpoints[route]
m = Symbol(uppercase(http.message.method))
HTTP.handle(e.handlers[m],http)
This code extracts the Endpoint from the global dictionary endpoints, determines what kind of HTTP request it is, i.e. GET, POST or whatever, and then uses HTTP.handle to handle the request.
So it is really, just some sugar on top of HTTP.jl, but I think it is simple and convenient sugar
HTML Elements
Finally, there is a completely undocumented, but super useful functionality in Pages.jl, i.e.
mutable struct Element
id::String
tag::String
name::String
attributes::Dict{String,String}
style::Dict{String,String}
innerHTML::String
parent_id::String
end
Note: I significantly changed this API just now because an earlier version depended on d3.js, which isn’t really necessary so I rewrote it to just use vanilla JS.
Although, not documented, it was used in an earlier Plotly example, but I recently moved the Plotly example to another package (Figures.jl).
This is a generic HTML element that contains info for creating browser elements on the fly from Julia.
The type of element, e.g. div, li, select, input, etc, is specified by the tag.
There are two main methods for modifying the browser DOM from Julia
- appendChild
- removeChild
Putting it all together to implement your example is now pretty simple.
Simply start with
julia> using Pages; Pages.examples();
This will start a local server and open a page (launching a browser if one isn’t open or creating a new tab). If you don’t want it to launch the page or open a new tab, you can run the example with
julia> using Pages; Pages.examples(launch=false);
but then you will need to manually navigate to http://localhost:8000/examples.
At the moment, there are only two simple examples there:
- Requests - Illustrates how to implement
POST requests and GET requests with parameters in the url.
- Blank - Just a blank page with pages.js loaded and ready for experimenting.
For your example, click “Blank” to go to the blank page.
Here is the code for your example:
function randomping()
count = 1
t0 = time()
while true
sleep(1)
e = Element(string(count),innerHTML =
"Ping $(count): $(round(time()-t0,digits=2)) seconds")
e.style["color"] = rand(["red","green","blue"])
rand() < .5 && Pages.appendChild(e)
count += 1
tstart = time()
end
end
This is basically an infinite loop that executes every 1 second.
Each loop, creates a new div (default tag is div so you do not need to specify)
e = Element(string(count),innerHTML =
"Ping $(count): $(round(time()-t0,digits=2)) seconds")
The div id is set to the count number and the displayed text (innerHTML) contains the count number and the total elapsed time from when the loop was started (to show it is roughly 1 second).
Just for fun, I also have it select a random color:
e.style["color"] = rand(["red","green","blue"])
Then, like you suggested, if a random number is less than 0.5, it will add a new div to the web page:
rand() < .5 && Pages.appendChild(e)
Well, that was a long post!
I hope you find it helpful and whether or not you end up using Pages.jl, I think it should probably help with whatever you are trying to do by looking through some of the code. If you have questions, don’t hesitate to ask. I’m happy to help if I can.
Cheers!
(which is related primarily to the fact that I know very little about this stuff and secondarily to the fact that the docs of these packages is often a bit scarce, (*)) since everything seems to build upon HTTP.jl I think I’d like to get a better idea for how that works and use that!