Seeking Insights: Tickerplants and Complex Event Processing (CEP) in Julia

[rohanshiloh]

The functional programming language and time-series vector database q/KDB has a low-latency architecture for efficiently processing extremely large volumes of real-time financial data directly from exchanges like the NYSE called a tickerplant.

A q/KDB tickerplant architecture stores 24 hours of realtime data in-memory (RDB) and writes it to disk (HDB) at EOD while using a CEP to build statistical/iterative models on the realtime streaming data and I was wondering if anyone has explored building a similar architecture in Julia possibly using JuliaDB as the on-disk store?

An interface between q/KDB and Julia currently exists: KdbConnect.jl

But I was wondering how performant Julia would be if we were to replicate this tickerplant architecture in pure Julia code with JuliaDB?

If anyone is interested in discussing or exploring this idea please do get in touch.

[nilshg]

JuliaDB is abandoned/unmaintained. There are various alternatives but I’m not familiar enough with KDB to recommend alternatives.

There’s a database channel on Slack you can join if you want to discuss the pros and cons of various db options, my feeling is that most people these days use DuckDB, maybe via QuackIO.jl

[j_u]

@nilshg

my feeling is that most people these days use DuckDB, maybe via QuackIO.jl

May I ask, how are you finding DuckDB, particularly its in-memory functionality? I use QuestDB, which focuses on time series and open-source formats. It is not an in-memory database per se; as far as I know, it appends data to column files and memory-maps the ‘tail’ of these files into virtual memory pages.

[j_u]

@rohanshiloh

A q/KDB tickerplant architecture stores 24 hours of realtime data in-memory (RDB) and writes it to disk (HDB) at EOD while using a CEP to build statistical/iterative models on the realtime streaming data and I was wondering if anyone has explored building a similar architecture in Julia [..]

I use Libwebsockets.jl, YYJSON.jl, and RDKafka.jl as my Feed Handler (FH), and Redpanda as my Tickerplant (TP) and TP Log File. I do not yet have a Real-Time Database (RDB); however, I am currently implementing a Single Producer - Single Consumer (SPSC) Shared Memory (SM) interface between Julia and q/KDB, with the goal of using KDB-X for this purpose. This is still a work in progress, though most major functionalities already seem to work. For the On-Disk File System (OFS) I use BeeGFS, and, as mentioned above, QuestDB serves as the Historical Database (HDB). Additional components include Grafana and TimeStored Pulse for visualizations, as well as Arroyo for stream processing. In the future I plan to extend the system with Rembus.jl and potentially add FPGA support via XRT.jl. My design philosophy is focused on: (i) connectivity, (ii) flexibility, (iii) usability, and (iv) performance, in that order.

As far as I understand, CEP in Julia is usually implemented with Rocket.jl, a reactive programming framework that simplifies working with asynchronous data. In addition, among other packages, i.e OnlineStats.jl provides high-performance single-pass algorithms for statistics and data visualization.

@rohanshiloh

But I was wondering how performant Julia would be if we were to replicate this tickerplant architecture in pure Julia code with JuliaDB?

In general, I believe Julia is very performant. However, the real question may be about GC (Garbage Collection) behavior and latency predictability in concurrent, streaming, or real-time workloads, especially compared to q, which as far as I know, is also garbage-collected. Do you have any information about q/KDB GC behavior and its architecture? By the way, I recently came across the paper Reconsidering Garbage Collection in Julia – A Practitioner Report (de Souza Amorim, Lin, Blackburn, Netto, Baraldi, Daly, Hosking, Pamnany, Smith, ISMM 2025).

I’m also curious, how are you finding Julia’s DataFrames.jl as a time-series and in-memory database alternative? I recently came across Deephaven.io, a query engine and framework for live dataframes that builds data maps using a directed acyclic graph. I wonder whether it would be possible to extend DataFrames.jl with Rocket.jl, Rembus.jl, Dagger.jl, and BonitoBook.jl to create… a slightly more advanced version in Julia?

@rohanshiloh

If anyone is interested in discussing or exploring this idea please do get in touch.

May I ask, how has the response been so far?

[nilshg]

Sorry if I gave the wrong impression - I’m not actually a heavy database user myself, and if I do have to use them it’s usually client data bases so I don’t get to pick what I work with, so I have little experience with DuckDB.

[j_u]

I see. At first, I was looking into Oracle Autonomous Database, but at the time I thought it wasn’t fully supported in Julia. I also considered KDB, but if I remember right, the personal edition license didn’t allow running it on cloud servers. So in the end, I was deciding between ClickHouse and QuestDB. I’m really happy with my choice, though I do wish it supported q/KDB syntax. I believe, it would make a lot of things much easier. Lately, I’ve been reading a lot of positive things about DuckDB, and I noticed that its Julia support even includes User Defined Functions (UDFs). As far as I know, it also has an in-memory option. That’s what led me to ask my question.

[j_u]

I must have misunderstood something here, sorry.

[j_u]

@j_u

I am currently implementing a Single Producer - Single Consumer (SPSC) Shared Memory (SM) interface between Julia and q/KDB, with the goal of using KDB-X for this purpose.

As for now, I have it like in the screenshot presented below. Its not perfect. I will try to collect additional information about q/kdb and KdbConnect.jl in the near future.

EDIT: Version 0.1.1 introduces several functional, reliability, and performance oriented enhancements across the Julia producer, C shim, and q consumer components. Based on my current understanding, the concurrency model in q/kdb appears to rely primarily on message passing and inter process communication, rather than cooperative multitasking or coroutine based constructs. Consequently, I plan to deprecate the blocking mode functions showPendingState[] and flushPending[], acknowledging that messages from the final partial (unfinished) batch in blocking mode may be lost in the event of a hard exit (e.g., via Ctrl + C). Furthermore, I intend to streamline the q codebase by removing the functionality that restricts message insertion to a limited number, thereby focusing exclusively on continuous processing. I was unable to find much information on the memory management architecture of q, however, my latest findings might indicate that it does not rely on a conventional garbage collection mechanism. I hope to sustain my involvement in the ongoing discussions in this thread …

EDIT: As for now, I’ll conclude with the pre-announcement flyers for version 0.1.2. Please see the attachments below. Over the next few weeks, I’m planning additional development activities and extensive peer reviews of the code in Warsaw, Zurich, London, New York, Austin, Menlo Park, Vancouver, Mumbai, Boston, and Miami (hence the slightly lighter suite of colors in the logo which itself is inspired by the IBM 2741 APL keyboard and System/360). For various reasons, I also intend to consult legal and security advisory experts regarding this potential package in the near future. For those interested in the history of kdb and an introduction to the q language, I highly recommend the video lecture by Mr. Paul Bilokon, Ph.D., titled “What is kdb+/q”, available on YouTube under this link: (https://)youtu.be/deOWz5Pbg0M?si=baW3e-gdBghtWyDF.