Realtime streaming of tick data

question

#1

I have a black box model (written in Julia) that requires realtime tick data as an input. The output of this model needs to be pub-subbed out to clients listening on a socket or topic. I have dozens to potentially hundreds of input streams that I need to process in realtime.

Other than the Reactive, RabbitMQ and Kafka packages, are there any other libraries that I should explore for such a use case?


#2

ØMQ is pretty good. You can use ZMQ.jl to access it from Julia, although API is better to learn from Python docs.

Another option is Redis. jkaye2012/Redis.jl is quite full-featured.

Note, however, that RabbitMQ, Kafka, ØMQ and Redis all have pretty different sets of features. For example, ØMQ and Redis are the fastest, but may become a single point of failure (cluster mode being much harder to use), Kafka provides persistence for a configured period of time, but has much higher latencies, etc. In fact, I rarely see them as switchable alternatives.


#3

I was hoping for a more Julia-centric approach that doesn’t rely on ØMQ or Redis either (I have used those successfully via python and c++ clients). Something like OnlineStats.jl sounds like it could work but, frankly, I haven’t had a chance to explore other alternatives yet. Thank you for the recommendations @dfdx.


#4

If you are looking for Julia-only solution, consider channels. They aren’t really queues and lack many of their advantages, but may be good for your specific use case.