A harness to test daily, hourly and millisecondly data processing


I need to demonstrate that software intended to accept, record and respond to financial multi-market data (both slow, once-a-day and fast, lots-a-day) does in fact accept, record and respond properly. I know how to evince the properly and how to convey that. I need an easily understood spigot – something that automates the gating of each next set of [stored] data and lets me demonstrate over the course of a conversation what happens over deeper horizons. Is something of Julia that I have yet not used well suited to this?


It would help the discussion if you list what you already used…


I have not used channels, tasks, parallel this and that. I have not done anything with buffers or streams of information coming in “over the transom”. I have written RunningFunctions.jl and that is “the other way around” – the function is stepping over the data. I need the data to be “stepping” into the “harness”.


parts of my question

(a) Can I have some package/clock be used to determine when data should next arrive by frequency (1/week, 1/day, 1/hour) without waiting around for the rest of it to arrive?

(b) What would work as separate nexus of separate control information
© How would the code that gets given each new value respond


Something like this is appealing to me. Instead of the for loop, can stream in your raw data as the messages come in. Your derived functionality is calculated on the fly. Could be used to create nice visualizations in realtime.

using Nulls
using Reactive

bidprice = Signal(?Float64, null)
askprice = Signal(?Float64, null)
bidsize = Signal(?Int64, null)
asksize = Signal(?Int64, null)

microprice = map((a,b,c,d) -> (a*d+b*c)/(c+d), bidprice, askprice, bidsize, asksize; typ=?Float64, init=null)

# Asynchronously update the top of book and magically, watch microprice update
for i in 1:100

  push!(bidprice, randn())
  push!(askprice, randn())
  push!(bidsize, rand(1:10))
  push!(asksize, rand(1:10))

  println("micro = ", value(microprice))



@hgeorgako so sweet a solution … the best harness has no bridle.