Seeking Insights: Tickerplants and Complex Event Processing (CEP) in Julia

[rohanshiloh]

The functional programming language and time-series vector database q/KDB has a low-latency architecture for efficiently processing extremely large volumes of real-time financial data directly from exchanges like the NYSE called a tickerplant.

A q/KDB tickerplant architecture stores 24 hours of realtime data in-memory (RDB) and writes it to disk (HDB) at EOD while using a CEP to build statistical/iterative models on the realtime streaming data and I was wondering if anyone has explored building a similar architecture in Julia possibly using JuliaDB as the on-disk store?

An interface between q/KDB and Julia currently exists: KdbConnect.jl

But I was wondering how performant Julia would be if we were to replicate this tickerplant architecture in pure Julia code with JuliaDB?

If anyone is interested in discussing or exploring this idea please do get in touch.

[nilshg]

JuliaDB is abandoned/unmaintained. There are various alternatives but I’m not familiar enough with KDB to recommend alternatives.

There’s a database channel on Slack you can join if you want to discuss the pros and cons of various db options, my feeling is that most people these days use DuckDB, maybe via QuackIO.jl

[j_u]

@nilshg

my feeling is that most people these days use DuckDB, maybe via QuackIO.jl

May I ask, how are you finding DuckDB, particularly its in-memory functionality? I use QuestDB, which focuses on time series and open-source formats. It is not an in-memory database per se; as far as I know, it appends data to column files and memory-maps the ‘tail’ of these files into virtual memory pages.

[j_u]

@rohanshiloh

A q/KDB tickerplant architecture stores 24 hours of realtime data in-memory (RDB) and writes it to disk (HDB) at EOD while using a CEP to build statistical/iterative models on the realtime streaming data and I was wondering if anyone has explored building a similar architecture in Julia [..]

I use Libwebsockets.jl, YYJSON.jl, and RDKafka.jl as my Feed Handler (FH), and Redpanda as my Tickerplant (TP) and TP Log File. I do not yet have a Real-Time Database (RDB); however, I am currently implementing a Single Producer - Single Consumer (SPSC) Shared Memory (SM) interface between Julia and q/KDB, with the goal of using KDB-X for this purpose. This is still a work in progress, though most major functionalities already seem to work. For the On-Disk File System (OFS) I use BeeGFS, and, as mentioned above, QuestDB serves as the Historical Database (HDB). Additional components include Grafana and TimeStored Pulse for visualizations, as well as Arroyo for stream processing. In the future I plan to extend the system with Rembus.jl and potentially add FPGA support via XRT.jl. My design philosophy is focused on: (i) connectivity, (ii) flexibility, (iii) usability, and (iv) performance, in that order.

As far as I understand, CEP in Julia is usually implemented with Rocket.jl, a reactive programming framework that simplifies working with asynchronous data. In addition, among other packages, i.e OnlineStats.jl provides high-performance single-pass algorithms for statistics and data visualization.

@rohanshiloh

But I was wondering how performant Julia would be if we were to replicate this tickerplant architecture in pure Julia code with JuliaDB?

In general, I believe Julia is very performant. However, the real question may be about GC (Garbage Collection) behavior and latency predictability in concurrent, streaming, or real-time workloads, especially compared to q, which as far as I know, is also garbage-collected. Do you have any information about q/KDB GC behavior and its architecture? By the way, I recently came across the paper Reconsidering Garbage Collection in Julia – A Practitioner Report (de Souza Amorim, Lin, Blackburn, Netto, Baraldi, Daly, Hosking, Pamnany, Smith, ISMM 2025).

I’m also curious, how are you finding Julia’s DataFrames.jl as a time-series and in-memory database alternative? I recently came across Deephaven.io, a query engine and framework for live dataframes that builds data maps using a directed acyclic graph. I wonder whether it would be possible to extend DataFrames.jl with Rocket.jl, Rembus.jl, Dagger.jl, and BonitoBook.jl to create… a slightly more advanced version in Julia?

@rohanshiloh

If anyone is interested in discussing or exploring this idea please do get in touch.

May I ask, how has the response been so far?

[nilshg]

Sorry if I gave the wrong impression - I’m not actually a heavy database user myself, and if I do have to use them it’s usually client data bases so I don’t get to pick what I work with, so I have little experience with DuckDB.

[j_u]

I see. At first, I was looking into Oracle Autonomous Database, but at the time I thought it wasn’t fully supported in Julia. I also considered KDB, but if I remember right, the personal edition license didn’t allow running it on cloud servers. So in the end, I was deciding between ClickHouse and QuestDB. I’m really happy with my choice, though I do wish it supported q/KDB syntax. I believe, it would make a lot of things much easier. Lately, I’ve been reading a lot of positive things about DuckDB, and I noticed that its Julia support even includes User Defined Functions (UDFs). As far as I know, it also has an in-memory option. That’s what led me to ask my question.

[j_u]

I must have misunderstood something here, sorry.

[j_u]

@j_u

I am currently implementing a Single Producer - Single Consumer (SPSC) Shared Memory (SM) interface between Julia and q/KDB, with the goal of using KDB-X for this purpose.

As for now, I have it like in the screenshot presented below. Its not perfect. I will try to collect additional information about q/kdb and KdbConnect.jl in the near future.