# Realtime streaming of tick data

**URL:** <https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713>\
**Category:** General Usage\
**Tags:** question\
**Created:** [September 4, 2017, 1:41pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713 "2017-09-04T13:41:41Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![hgeorgako](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hgeorgako/32/5518_2.png) [@hgeorgako](https://discourse.julialang.org/u/hgeorgako)\
**Post date:** [September 4, 2017, 1:41pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/1 "2017-09-04T13:41:41Z")

</div>

I have a black box model (written in Julia) that requires realtime tick data as an input. The output of this model needs to be pub-subbed out to clients listening on a socket or topic. I have dozens to potentially hundreds of input streams that I need to process in realtime.

Other than the Reactive, RabbitMQ and Kafka packages, are there any other libraries that I should explore for such a use case?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [September 4, 2017, 2:19pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/2 "2017-09-04T14:19:36Z")

</div>

ØMQ is pretty good. You can use [ZMQ.jl](https://github.com/JuliaInterop/ZMQ.jl) to access it from Julia, although API is better to learn from [Python docs](http://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pubsub.html).

Another option is Redis. [jkaye2012/Redis.jl](https://github.com/jkaye2012/Redis.jl) is quite full-featured.

Note, however, that RabbitMQ, Kafka, ØMQ and Redis all have pretty different sets of features. For example, ØMQ and Redis are the fastest, but may become a single point of failure (cluster mode being much harder to use), Kafka provides persistence for a configured period of time, but has much higher latencies, etc. In fact, I rarely see them as switchable alternatives.

---

<div class="post-metadata">

**Author:** ![hgeorgako](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hgeorgako/32/5518_2.png) [@hgeorgako](https://discourse.julialang.org/u/hgeorgako)\
**Post date:** [September 4, 2017, 2:39pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/3 "2017-09-04T14:39:25Z")

</div>

I was hoping for a more Julia-centric approach that doesn’t rely on ØMQ or Redis either (I have used those successfully via python and c++ clients). Something like OnlineStats.jl sounds like it could work but, frankly, I haven’t had a chance to explore other alternatives yet. Thank you for the recommendations @dfdx.

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [September 4, 2017, 3:05pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/4 "2017-09-04T15:05:53Z")

</div>

If you are looking for Julia-only solution, consider channels. They aren’t really queues and lack many of their advantages, but may be good for your specific use case.

---

<div class="post-metadata">

**Author:** ![FemtoTrader](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/femtotrader/32/309_2.png) [@FemtoTrader](https://discourse.julialang.org/u/FemtoTrader)\
**Post date:** [June 11, 2019, 9:01pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/5 "2019-06-11T21:01:52Z")

</div>

I was also looking for a solution for a similar use case and found this article interesting [Modern Open Source Messaging: NATS, RabbitMQ, Apache Kafka, hmbdc, Synapse, NSQ and Pulsar | by Philip Feng Ph.D | Medium](https://medium.com/@philipfeng/modern-open-source-messaging-apache-kafka-rabbitmq-nats-pulsar-and-nsq-ca3bf7422db5) Modern Open Source Messaging: NATS, RabbitMQ, Apache Kafka, hmbdc, Synapse, NSQ and Pulsar

Unfortunately it seems that only RabbitMQ and Apache Kafka currently have Julia client ([https://github.com/JuliaComputing/AMQPClient.jl](https://github.com/JuliaComputing/AMQPClient.jl) and [https://github.com/dfdx/RDKafka.jl](https://github.com/dfdx/RDKafka.jl) )

I was looking at NSQ … but there is no Julia client currently [NSQ Docs 1.2.1 - Client Libraries](https://nsq.io/clients/client_libraries.html)  
Here is protocol spec [NSQ Docs 1.2.1 - TCP Protocol Spec](https://nsq.io/clients/tcp_protocol_spec.html)

Julia client for NATS [https://nats.io/](https://nats.io/) could also be interesting.

I want to have a messaging system out of JVM world because of heaviness (1) nor C++ world because of complexity (1) and ideally without dependencies (and so a Go solution could be interesting)

(1) please don’t feed the troll

---

<div class="post-metadata">

**Author:** ![zsykm](https://avatars.discourse-cdn.com/v4/letter/z/97f17d/32.png) [@zsykm](https://discourse.julialang.org/u/zsykm)\
**Post date:** [June 12, 2019, 11:40am UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/6 "2019-06-12T11:40:54Z")

</div>

@hgeorgako @FemtoTrader @dfdx Hi guys, how things going now? Made any decisions? What is your final choice and why?

I am trying to build something similar and collecting building blocks for it.

---

<div class="post-metadata">

**Author:** ![hgeorgako](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hgeorgako/32/5518_2.png) [@hgeorgako](https://discourse.julialang.org/u/hgeorgako)\
**Post date:** [June 12, 2019, 11:57am UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/7 "2019-06-12T11:57:08Z")

</div>

Ended up using 0MQ for a prototype with a mix of c++ and python. The prototype ended up becoming production and eventually code stopped being maintained. Became more of a 3 language problem after a while. Interested to see how this thread plays out.

---

<div class="post-metadata">

**Author:** ![FemtoTrader](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/femtotrader/32/309_2.png) [@FemtoTrader](https://discourse.julialang.org/u/FemtoTrader)\
**Post date:** [June 12, 2019, 12:32pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/8 "2019-06-12T12:32:17Z")

</div>

I wonder if [MQTT](http://mqtt.org/) couldn’t also be a solution for this use case.

See Julia client

[https://github.com/rweilbacher/MQTT.jl](https://github.com/rweilbacher/MQTT.jl)

and a comparison of MQTT implementations (could be useful to find a MQTT broker)

> **[Comparison of MQTT implementations](https://en.wikipedia.org/wiki/Comparison_of_MQTT_implementations)**
>
> MQTT is an ISO standard (ISO/IEC PRF 20922) publish-subscribe-based messaging protocol. It works on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited. The publish-subscribe messaging pattern requires a message broker.
> All comparison categories use the stable version of each implementation listed in the overview section. The comparison is limited to features that relate to the MQTT prot...

---

<div class="post-metadata">

**Author:** ![zsykm](https://avatars.discourse-cdn.com/v4/letter/z/97f17d/32.png) [@zsykm](https://discourse.julialang.org/u/zsykm)\
**Post date:** [June 20, 2019, 2:44pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/9 "2019-06-20T14:44:57Z")

</div>

[Here](https://staysail.github.io/nng_presentation/nng_presentation.html) is something interesting to read.

Concerning to the performance and system architecture (for your HFT sys I guess), choosing a proper MQ could be important, but may not so critical at the very beginning, because you can always switch to or “plug into” a different MQ sys if you carefully designed the interface.

---

<div class="post-metadata">

**Author:** ![zsykm](https://avatars.discourse-cdn.com/v4/letter/z/97f17d/32.png) [@zsykm](https://discourse.julialang.org/u/zsykm)\
**Post date:** [June 20, 2019, 2:55pm UTC](https://discourse.julialang.org/t/realtime-streaming-of-tick-data/5713/10 "2019-06-20T14:55:34Z")

</div>

As per the [following](https://nanomsg.org/documentation-zeromq.html)

> Threading Model

> One of the big architectural blunders I’ve done in ZeroMQ is its threading model. Each individual object is managed exclusively by a single thread. That works well for async objects handled by worker threads, however, it becomes a trouble for objects managed by user threads. The thread may be used to do unrelated work for arbitrary time span, e.g. an hour, and during that time the object being managed by it is completely stuck. Some unfortunate consequences are: inability to implement request resending in REQ/REP protocol, PUB/SUB subscriptions not being applied while application is doing other work, and similar. In nanomsg the objects are not tightly bound to particular threads and thus these problems don’t exist.

> - REQ socket in ZeroMQ cannot be really used in real-world environments, as they get stuck if message is lost due to service failure or similar. Users have to use XREQ instead and implement the request re-trying themselves. With nanomsg, the re-try functionality is built into REQ socket.
> - In nanomsg, both REQ and REP support cancelling the ongoing processing. Simply send a new request without waiting for a reply (in the case of REQ socket) or grab a new request without replying to the previous one (in the case of REP socket).
> - In ZeroMQ, due to its threading model, bind-first-then-connect-second scenario doesn’t work for inproc transport. It is fixed in nanomsg.

Have you ever encountered any issue in using ZeroMQ as described above?

> [@hgeorgako](#):
>
> Became more of a 3 language problem after a while.

Quite the same, 😁 I have to deal with front-end modules written in c# calling data feeder in c++ DLL , may be more Python modules coming in.