# A harness to test daily, hourly and millisecondly data processing

**URL:** <https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223>\
**Category:** General Usage\
**Created:** [June 12, 2017, 12:32pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223 "2017-06-12T12:32:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 12, 2017, 12:32pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/1 "2017-06-12T12:32:22Z")

</div>

I need to demonstrate that software intended to accept, record and respond to financial multi-market data (both slow, once-a-day and fast, lots-a-day) does in fact accept, record and respond properly. I know how to evince the properly and how to convey that. I need an easily understood spigot – something that automates the gating of each next set of [stored] data and lets me demonstrate over the course of a conversation what happens over deeper horizons. Is something of Julia that I have yet not used well suited to this?

---

<div class="post-metadata">

**Author:** ![lobingera](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lobingera/32/211_2.png) [@lobingera](https://discourse.julialang.org/u/lobingera)\
**Post date:** [June 12, 2017, 1:13pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/2 "2017-06-12T13:13:19Z")

</div>

> [@JeffreySarnoff](#):
>
> Is something of Julia that I have yet not used well suited to this

It would help the discussion if you list what you already used…

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 12, 2017, 2:15pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/3 "2017-06-12T14:15:23Z")

</div>

I have not used channels, tasks, parallel this and that. I have not done anything with buffers or streams of information coming in “over the transom”. I have written RunningFunctions.jl and that is “the other way around” – the function is stepping over the data. I need the data to be “stepping” into the “harness”.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 12, 2017, 5:02pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/4 "2017-06-12T17:02:46Z")

</div>

parts of my question

(a) Can I have some package/clock be used to determine when data should next arrive by frequency (1/week, 1/day, 1/hour) without waiting around for the rest of it to arrive?

(b) What would work as separate nexus of separate control information  
(c) How would the code that gets given each new value respond

---

<div class="post-metadata">

**Author:** ![hgeorgako](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hgeorgako/32/5518_2.png) [@hgeorgako](https://discourse.julialang.org/u/hgeorgako)\
**Post date:** [June 13, 2017, 4:03pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/5 "2017-06-13T16:03:38Z")

</div>

Something like this is appealing to me. Instead of the for loop, can stream in your raw data as the messages come in. Your derived functionality is calculated on the fly. Could be used to create nice visualizations in realtime.

```julia
using Nulls
using Reactive

bidprice = Signal(?Float64, null)
askprice = Signal(?Float64, null)
bidsize = Signal(?Int64, null)
asksize = Signal(?Int64, null)

microprice = map((a,b,c,d) -> (a*d+b*c)/(c+d), bidprice, askprice, bidsize, asksize; typ=?Float64, init=null)

# Asynchronously update the top of book and magically, watch microprice update
for i in 1:100

  push!(bidprice, randn())
  push!(askprice, randn())
  push!(bidsize, rand(1:10))
  push!(asksize, rand(1:10))

  println("micro = ", value(microprice))

end

```

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [June 13, 2017, 7:23pm UTC](https://discourse.julialang.org/t/a-harness-to-test-daily-hourly-and-millisecondly-data-processing/4223/6 "2017-06-13T19:23:56Z")

</div>

@hgeorgako so sweet a solution … the best harness has no bridle.