Realtime streaming of tick data

General Usage
1

I have a black box model (written in Julia) that requires realtime tick data as an input. The output of this model needs to be pub-subbed out to clients listening on a socket or topic. I have dozens to potentially hundreds of input streams that I need to process in realtime.

Other than the Reactive, RabbitMQ and Kafka packages, are there any other libraries that I should explore for such a use case?

2

ØMQ is pretty good. You can use ZMQ.jl to access it from Julia, although API is better to learn from Python docs.

Another option is Redis. jkaye2012/Redis.jl is quite full-featured.

Note, however, that RabbitMQ, Kafka, ØMQ and Redis all have pretty different sets of features. For example, ØMQ and Redis are the fastest, but may become a single point of failure (cluster mode being much harder to use), Kafka provides persistence for a configured period of time, but has much higher latencies, etc. In fact, I rarely see them as switchable alternatives.

3

I was hoping for a more Julia-centric approach that doesn’t rely on ØMQ or Redis either (I have used those successfully via python and c++ clients). Something like OnlineStats.jl sounds like it could work but, frankly, I haven’t had a chance to explore other alternatives yet. Thank you for the recommendations @dfdx.

4

If you are looking for Julia-only solution, consider channels. They aren’t really queues and lack many of their advantages, but may be good for your specific use case.

5

I was also looking for a solution for a similar use case and found this article interesting Modern Open Source Messaging: NATS, RabbitMQ, Apache Kafka, hmbdc, Synapse, NSQ and Pulsar | by Philip Feng Ph.D | Medium Modern Open Source Messaging: NATS, RabbitMQ, Apache Kafka, hmbdc, Synapse, NSQ and Pulsar

Unfortunately it seems that only RabbitMQ and Apache Kafka currently have Julia client (https://github.com/JuliaComputing/AMQPClient.jl and https://github.com/dfdx/RDKafka.jl )

I was looking at NSQ … but there is no Julia client currently NSQ Docs 1.2.1 - Client Libraries
Here is protocol spec NSQ Docs 1.2.1 - TCP Protocol Spec

Julia client for NATS https://nats.io/ could also be interesting.

I want to have a messaging system out of JVM world because of heaviness (1) nor C++ world because of complexity (1) and ideally without dependencies (and so a Go solution could be interesting)

(1) please don’t feed the troll

1 Like
6

@hgeorgako @FemtoTrader @dfdx Hi guys, how things going now? Made any decisions? What is your final choice and why?

I am trying to build something similar and collecting building blocks for it.

1 Like
7

Ended up using 0MQ for a prototype with a mix of c++ and python. The prototype ended up becoming production and eventually code stopped being maintained. Became more of a 3 language problem after a while. Interested to see how this thread plays out.

8

I wonder if MQTT couldn’t also be a solution for this use case.

See Julia client

https://github.com/rweilbacher/MQTT.jl

and a comparison of MQTT implementations (could be useful to find a MQTT broker)

9

Here is something interesting to read.

Concerning to the performance and system architecture (for your HFT sys I guess), choosing a proper MQ could be important, but may not so critical at the very beginning, because you can always switch to or “plug into” a different MQ sys if you carefully designed the interface.

10

As per the following

Threading Model

One of the big architectural blunders I’ve done in ZeroMQ is its threading model. Each individual object is managed exclusively by a single thread. That works well for async objects handled by worker threads, however, it becomes a trouble for objects managed by user threads. The thread may be used to do unrelated work for arbitrary time span, e.g. an hour, and during that time the object being managed by it is completely stuck. Some unfortunate consequences are: inability to implement request resending in REQ/REP protocol, PUB/SUB subscriptions not being applied while application is doing other work, and similar. In nanomsg the objects are not tightly bound to particular threads and thus these problems don’t exist.

  • REQ socket in ZeroMQ cannot be really used in real-world environments, as they get stuck if message is lost due to service failure or similar. Users have to use XREQ instead and implement the request re-trying themselves. With nanomsg, the re-try functionality is built into REQ socket.
  • In nanomsg, both REQ and REP support cancelling the ongoing processing. Simply send a new request without waiting for a reply (in the case of REQ socket) or grab a new request without replying to the previous one (in the case of REP socket).
  • In ZeroMQ, due to its threading model, bind-first-then-connect-second scenario doesn’t work for inproc transport. It is fixed in nanomsg.

Have you ever encountered any issue in using ZeroMQ as described above?

Quite the same, :grin: I have to deal with front-end modules written in c# calling data feeder in c++ DLL , may be more Python modules coming in.

1 Like