Backtesting framework

Sounds great ! Yes, Rocket is mind bending at first.

TBH it looks like an anti-pattern. The whole point is to separate data and concerns among actors for composition and reusability. In Rocket/Reactive, actors shouldn’t share states. It’s not a hard line but it can save you tons of trouble with async stuffs.

I see three possible concerns here:

  1. Building a sink DataFrame. (Some kind of blotter. See the keep() operator in Rocket)
  2. Making Decision. (A Strategy)
  3. Charting stuffs. (Some kind of blotter)

Sounds like a lot of waiting. Why not making things independent ?

  1. Have each indicator have its own life and fire data as it processes them.
  2. Have a strategy that keeps whatever data it needs and no more. Have it receive notification for quotes, indicators, positions, fills or whatever you want.

That’s how Rocket and Lucky are designed anyway.

Look at sync() operator in Rocket.

I kinda thought this was coming. I know I’m sharing a lot of state, but I think it’ll be OK, because only one thing (the ChartSubject) is mutating the Charts. Everything else just reads from the Charts.

If you go over to talipp, you’ll see that their indicators have an add and update method. In OnlineTechnicalIndicators.jl, fit! is the equivalent of add, but (as far as I know), there is no equivalent of update. If I want accurate values, I have to wait for candle close before I feed data to it. I’m OK with waiting for candle close, but it’s something to be aware of when using this library.

Everything else aside, my personal preference for organizing indicators is to group them by timeframe.

  • Their source data is going to be the same.
  • They’re often used together, so it’s convenient that they’re close to each other.

I like to do multi-timeframe analysis.

I get where you’re coming from. What I’m doing may not seem to be in the Erlang spirit of actors not sharing memory and sending messages to each other. However, the only thing I’m sharing is the chart data. Everything else in the async pipeline passes little messages back and forth. Just being able to manage communication between a lot of async processes has been a huge benefit for me.

(As an aside, are actors not allowed to access databases? That’s shared state too. What’s the guiding principle in what can and cannot be shared?)

It might not be as bad as you think. Or maybe it’s worse. We’ll see in a few months. I’m still in a very experimental and exploratory stage, so I’m letting myself make mistakes.

2 Likes

Embedding raw HTML in Documentation

I learned a useful technique from the LightweightCharts.jl documentation and used it to embed an interactive chart in my documentation.

:christmas_tree: :santa:

3 Likes

Early Christmas Morning

Early Christmas morning, I was able to do something that I had wanted to be able to do for a long time, and that’s to backtest a strategy inside a Pluto.jl notebook. Running trade simulations in the REPL was fun, but what I really needed to do was find a workflow that would allow me to SIT STILL and look at the results of a simulated run, and see where it went right and more importantly see where it went wrong, and then try to improve upon it.

HMAStrategy

function should_open_long(strategy::HMAStrategy)
    # if we're neutral and
    crossed_up(strategy.rf.hma330, strategy.rf.hma440)
end

function should_close_long(strategy::HMAStrategy)
    # if we're long and
    crossed_down(strategy.rf.hma330, strategy.rf.hma440)
end

I’m a fan of the Hull Moving Average, so the second strategy I wrote after a Golden Cross strategy was a long-only HMA cross strategy. I let it run from 2023-07-01 to 2024-11-29 which was a bullish period where I thought it would do well.

It did alright and made some imaginary money buying and selling BTCUSD, but towards the end, it made 3 losing trades in a row. They’re trades 4, 5, and 6 in the image above. Trades 4 & 5 had late entries, but they were profitable for a few days before price dipped and the position was closed on a late exit signal. Those two trades looked salvageable to me. Trade 6, on the other hand, was just a bad entry, and I thought I could come up with some criteria to easily avoid it. …so I made some adjustments.

HMA2Strategy

  • I added the concept of late entry and early exit to remedy the problem I saw on trades 4 & 5.
  • I added an extra check for the slope of the 440 HMA and the position of the close price relative to the 440 HMA. If the slope is negative and the price is under the 440 HMA, skip the trade.
  • Then I let it run.

My ideas worked, and I achieved a pretty good performance improvement.

That was the first time I was able to properly iterate on a strategy in Julia.

I knew Pluto was going to be really well suited for this. Pluto was the reason I even started exploring Julia in the first place. Anyone who has had to manually control order-of-execution in Jupyter knows what a pain that can be. I didn’t have to worry about that at all in Pluto, and it made the backtesting workflow feel so much better.

I regret that this notebook may be difficult for others to run, because you’re going to have to download a big dataset using CryptoMarketData.jl to even start. However, I’ve made a static export of the notebook that’ll let you look around and play with the charts. You can also see the implementation of HMA2Strategy which I did not include in this post.

For those who are curious, I also made a static page that tries to explain how my simulation system works. I drew a few diagrams with nomnoml too, because it’s easier to explain some things visually.

There’s a lot that’s unfinished and underdeveloped in my system, but it’s starting to get somewhere. I feel like I have a viable notebook-based workflow for strategy development now. It took me a while to reach this point, and I’m thankful for all the libraries and infrastructure in the Julia community, and the all the human help I received along the way. It’s been a while since I’ve enjoyed learning and using a language this much.

9 Likes

This is really great !

Congratulations.

2 Likes

Yes, I fully agree; @g-gundam is writing top-notch posts that are very informative. By the way, do you guys understand what rbeeli is working on in his repositories? I mean, how to run a system similar to the one he seems to be designing, preferably in conjunction with Lucky (assuming full compatibility)? I hope this is not too off-topic.

EDIT 1: The rationale behind this question is that it seems there are many approaches to the topic of algorithmic trading, and this diversity makes it somewhat difficult to pursue any broader and more focused cooperation. I believe the prerequisite is to find an architecture that is appealing to a wider group of people. This would make conversation and any cooperation much easier, assuming there is interest in such cooperation at all. Otherwise, I am afraid it seems to be a discussion on a collection of rather separate packages and partial solutions, which is great in its own right but can be quite time-consuming. Again, I am assuming there is interest in discussing an institutional-grade quality, Julia-based algorithmic trading solution.

EDIT 2: To follow up, some time ago I tried to focus on the analytical side of this algorithmic trading equation. The system consisted of redpanda.com, arroyo.dev, questdb.com, and grafana.com. This was a hobby project that lasted a few months, during which I also deployed ZFS and BeeGFS on the storage side. While it was a bit heavy, the advantage was that it was easy to operate by one person, could run 24/7, and was flexible. Thanks to Redpanda and its Kafka compatibility, the system was also extendable. It was relatively easy to add additional open-source software while keeping the operations online. I liked it quite a lot; however, I have to admit that it was definitely not a high-frequency trading solution. My focus was on analytics, not on trading per se at that time. I am interested in hearing your opinions and, of course, should you find it suitable, your recommendations. I also hope that this information might be useful to somebody who reads this thread.

1 Like

I’m going to start with a partial reply to what you said in EDIT 1.

There are definitely a lot of different approaches being taken to solving the algorithmic trading puzzle, and it did seem like cooperation seemed difficult because everyone had their own framework, but I think the introduction of Rocket.jl into the picture changed that. I created a gist a few posts back that added OnlineTechnicalIndicators.jl to the goldencross example that comes with Lucky.jl just to get OTI styled indicator calculation in the same process.

Since then, I took it a step further and wrote a new trade function that uses OTI indicators to make trade decisions instead of Lucky.jl indicators. The rest of Lucky.jl is still being used for order execution, but I was able to splice in a foreign system and still have everything work. It was Rocket.jl that allowed me to reroute the data flow this way, and I think that’s huge. It is way more flexible than I anticipated, and it’ll let us mix and match disparate systems together through message passing.

1 Like

Yes, thank you. In general, I do agree with you on your points related to Rocket, OnlineTechnicalIndicators, and Lucky, and broadly speaking, your trading pipeline that currently includes HierarchicalStateMachines. However, I have the feeling that these are still, in a sense, separate processes rather than a fully unified framework. For example, it is not clear what your approach is to data ingestion and, in general, how data flows through your system. Are you separating trading pipelines from analytics and plotting, or are you mixing them together?

1 Like

Absolutely, there are lots of different approaches. Lots of folks are trying to solve a particular problem before thinking of any big framework. I believe that’s what rbeeli (Rino B.) · GitHub is doing.

I fully agree with @g-gundam that ReactiveBayes/Rocket.jl is an excellent foundation for a core trading framework. The point of oliviermilla/Lucky.jl is to create a thin layer dedicated to trading to:

  1. Leverage julia’s core proposition of not having to recode anything between test and prod (the dual implementation problem) while being able to “trade from a REPL/workbook”.
  2. being reasonably fast while doing so. There is no limit to how fast one might want to trade, but the point here is not to have a sub milliseconds system by design.
  3. Allow for kinda anything. You can use Lucky to do some Monte Carlo stuff, Optimization, portfolio tracking, place arbitrage, etc.

One could say a native ccxt for Finance (not just crypto).

Being able to provide a solid strategy development environment while using my brokers as a PMS is my first target.

Integration to wider framework and system (arroyos & stuffs) can be done in a deep (native types) or shallow way (sending messages around). So can integration to brokers, etc. Then it could be dubbed closer to a framework such as QuantTower.

Anyway, I am aware that Lucky overlaps existing julia libraries. It is fine. Heck, even basic stuffs are borrowed from JuliaFinance (while respecting the Licensing).

I do believe the right way is to bundle everything so a basic framework emerges, then spinoff repositories to reduce redundancy and ease integration. (one repo for ontologies, one for indicators, one for pricing, one for blotters, etc.) Not the other way around. That is also how julia evolved. All in one place, then standard library got spin off, then the compiler, etc.

I am personally very happy of the fact that @g-gundam could use Lucky even though the framework is still in its infancy with a half backed doc.

My current target is to integrate more fully with InteractiveBrokers to have an end to end system. It’s a painful step as

  1. The original julia InteractiveBrokers library (Jib) is messy as hell.
  2. The maintainer does not seem to be collaborate Cooperation · Issue #50 · lbilli/Jib.jl
  3. The IBKR API isn’t standard in any way. Even getting timestamped data is a mess compared to any other API.

I wish I would spend more time on Lucky than on IBKR, but it is what it is.

2 Likes

I would agree with that assessment. TradingPipeline.jl is the sum of my current experiments. It’s just starting to become a cohesive system, but there are a ton of details that are being glossed over.

For backtesting, I use data that I downloaded using CryptoMarketData.jl. It’s currently a manual process to set that up, and I haven’t documented how to do that.

For live testing and live trading, I intend to setup some websocket clients. I haven’t gotten around to doing that yet, but hopefully very soon.

In my system, it’s all mixed together in one pipeline. I made a diagram of what my simulation pipeline currently looks like.

The key thing to note is that changing from backtesting to live testing to live trading doesn’t change the shape of the pipeline that much. Similar to Lucky.jl, strategy code doesn’t even have to know or care whether it’s in a backtest or live trading situation. It should work the same which is a highly desirable trait to have.

Also, right now, plotting is done manually after a backtest session, but I’ve been doing more experiments with realtime plotting. When I start live testing strategies, realtime plotting will become part of the pipeline, and it’ll be adjacent to the chart_subject. However, it won’t be included in backtesting, because it would slow things down unnecessarily. It’s kinda fun to watch, though.

UPDATE: I just learned I could directly upload and embed videos in Discourse without relying on a 3rd party host. You might want to toggle the full screen view though.

3 Likes

That video is impressive!

Tons of example around for that. I recommend using JSON3 for parsing/encoding JSON.

Wrote this a while back:
DydxV3.jl/src/WebSockets.jl at main · oliviermilla/DydxV3.jl

Works great but I was under the assumption that all subscriptions had to be made through the same socket, which is false. So it’s actually 10x easier, pretty much spin open and close function for each subscription.

This also could be of inspiration: Rocket.jl/src/observable/network.jl at main · ReactiveBayes/Rocket.jl it’s documented.

1 Like

I forgot about dydx, and I didn’t know you wrote a little client library for their exchange. I will check it out.

Well, the V3 exchange is going to close. Hyperliquid is all the range and they focus on v4 and v5 but now that they are on a separate chain (v4), the API isn’t so trivial anymore and v3 cannot be ported. :frowning:

1 Like

Hey @g-gundam, thanks for providing all this info. Very interesting. I am writing a bit more. Maybe you find something to your interest.

Architecture
In the outline I provided above, Redpanda serves as a Central Processing Unit (CPU), unifying the system and making it easily expandable. It supports the Kafka protocol, which is a standard for data streaming. This allows for near real-time data transfer to other software, making connections straightforward. Redpanda is both reliable and scalable.

Data Injection
There are several ways to feed data into the system:
Julia Websockets: Connect to Redpanda using RDKafka.jl.
Direct Connection: Connect directly within Redpanda.
Arroyo: Connect through Arroyo’s GUI, which is likely the most convenient method.

Basic Data Manipulation
You can manipulate your data (e.g., JSON) in Julia or use WASM modules in Redpanda. I typically used Rust for this, as it’s quite easy. While you might be able to use Julia, I’m not entirely sure. The simplest methods are to manipulate data via SQL in Arroyo. As far as I know, Julia is not currently supported by Arroyo.

QuestDB
I liked QuestDB. It is developed by individuals with direct investment experience. I also considered ClickHouse but found it more complex.

Grafana
I like your package based on LightweightCharts.jl. I also like Grafana. The feature I liked most about Grafana is its ability to manage large list of symbols in a super easy way. A short gif is available here: https://questdb.com/images/blog/2023-12-08/chart-12.gif.

Storage
ZFS is very reliable but a bit heavy. It is fast if you need to move backups with large number of files over WAN. BeeGFS is super easy to deploy. You can store a lot with it in a distributed way. Of course both are optional.

Deployment
I used podman compose.

Summary
In summary, my approach would be to maintain separate pipelines for trading, analysis, and visualization, minimizing custom code while ensuring the system remains simple and extensible. For me, it was important that the system operated 24/7 so I rather tried to avoid situations where everything was based on one Julia instance running.

General interest
I like to read Blog | Dean Markwick. I am also interested in fundamental analysis. I am curious about topics related to HFT, however, I know very little about it. As I indicated above, there are various approaches, styles and system architectures. My approach was more focused on analytics then algorithmic trading. Happy to keep in touch.

1 Like

@j_u - Let’s take a step back from the technologies for a bit, and help me understand what kind of analysis you’re doing. Let me use myself as an example first.

  • I’m looking at price and indicators derived from price data all on the same chart.
  • I’m looking for patterns on that chart may imply future behavior on the chart.
  • For example (from earlier in this thread) I hypothesized that if the 330 HMA crosses above the 440 HMA, that has a high probability of signalling the beginning of an uptrend in price.
  • With that in mind, I created a strategy that returns a buy signal when it notices that pattern in the chart.

This is one expression of a very diverse and eclectic discipline called technical analysis, and that’s what I’m trying to automate.

What are you trying to do?

It seems like you’re trying to do something slightly different, but I’m not sure what. When you look at your data, what are you looking for? Help me understand your process and goals.

1 Like

When you look at your data, what are you looking for?

I think that as the Efficient Market Hypothesis (EMH) is facing constant criticism, it seems worthwhile to explore various routes to identify potential market inefficiencies. Personally, I prefer to have an investment thesis supported by a narrative. Whether it’s purely algorithmic, based on fundamentals, or a mix of both depends on the subject being analyzed and the broader context. It’s challenging to answer your question in one paragraph, as my focus can shift quite frequently. Generally, I look for signals that change my perspective.

BTW, are you always focused solely on those crosses?

For instance, what I find interesting about @dm13450’s blog is that the Author is always exploring new ideas — one day it’s Reinforcement Learning, the next it’s Large Language Models, and another day it’s Hawkes processes …

It seems like you’re trying to do something slightly different, but I’m not sure what.

I’m not sure how different it is. I aimed to provide more information about the system I used for a while. My objectives for this system were 24/7 operations, ease of use, and flexibility. I’m not a power user of any of its components. The less I know is probably ZFS.

I’m also uncertain about Redpanda and Arroyo. However, somehow I like both. Redpanda significantly enhances flexibility, allowing you to connect almost anything through various connectors (of course there’s more to it than just that). Arroyo is simply a joy to use. I’ve seen similar systems without these two components. Some interesting ones included Dragonfly (GitHub - dragonflydb/dragonfly: A modern replacement for Redis and Memcached) or Oxla (https://www.oxla.com/) or NATS (https://nats.io/) or MQTT (https://mqtt.org/).

1 Like

No, there are more techniques, but cross-based strategies are kind of like the “Hello, World!” of automated trading, so that’s what I started with. A few more ideas that I use a lot and consider the core building blocks of my analysis are:

  • Support and Resistance - If you put lines on a chart, people will react to them. Some lines get more respect than others.
  • Divergence - This is the idea that trends changes in oscillators often precede trend changes in price.
  • Multi-timeframe Confluence - Imagine if you observed bullish divergence on the 1 day, 4 hour, and 30 minute charts simultaneously. This is the idea that finding the same thing in multiple timeframes makes the probability of the implied outcome of that observation even higher.

…and there’s more, but you can do a search for “technical analysis” if you want to go down that rabbit hole.

1 Like

I just checked this one out. Their SAMPLE BY extension to SQL is really handy.

1 Like

I think I understand what you are trying to communicate. I usually view these concepts in terms of fundamentals or technicals, to keep it simple. In general, I believe I am familiar with technical analysis as you are describing it. I first encountered this topic in primary school through a book on Elliott Waves. The best resource I have seen on this subject, I believe, was UBS Technical Weekly by Mr. Michael Riesner and Mr. Marc Muller. However, I am definitely not as deeply involved in this subject as you seem to be.

I just visited Julia Discourse for other topics I’m interested in, spotted your posts, and decided to speak up. What I wanted to convey is that it might be useful to briefly define the analytical infrastructure, in conjunction with the visualization and execution pipelines.

I’m not sure how much this approach interests you. I wish I could be more helpful, like providing the code; however, I’ve lost almost all my notes on this subject. Nevertheless, I think the version of analytical infrastructure I outlined was quite useful.

1 Like