Incremental Technical Analysis indicators

FemtoTrader · December 20, 2023, 8:46am

Hi,

Unlike existing libraries for technical analysis which typically have to work on the whole input vector in order to calculate new values of indicators I’m looking for a Julia library which implements technical analysis indicators in an incremental approach.

talipp (or tali++) GitHub - nardew/talipp: talipp - incremental technical analysis library for python provides such implementation in Python
Same for this TA-Lib fork GitHub - trufanov-nok/ta-lib-rt: TA-Lib RT is a fork of TA-Lib that provides additional API for incremental calculation of indicators without reprocessing whole data. (C)

Why I think it’s important ?
Because we have in the Julia ecosystem so much timeseries implementation (Temporal.TS, TimeSeries.TimeArray, TSFrames.TSFrame… and many other outdated lib).
So building a TA lib on top of this is probably not a good idea… especially when dealing with Real Time data.

I tried to implement some indicators using such a manner.

SMA (simple moving average) and EMA (exponential moving average) have been implemented and tested against same test cases than talipp.

I tried implementing SMA using CircularBuffer from DataStructures but also using MovingWindow from OnlineStats.

But I’m stuck with SMMA.

I wonder if some of you can watch GitHub - femtotrader/IncTA.jl: Julia Incremental Technical Analysis Indicators (inspired by talipp)
and have a look why SMMA doesn’t behave as expected.

Moreover I also wonder if some other dev are interested by such a project which could be used for backtesting strategies, paper trading and real trading.

Kind regards

PS : SMMA is fixed now… but the road is long to get all indicators running

simsurace · December 20, 2023, 9:08pm

Have you looked at OnlineStats.jl?
EDIT: sorry I just saw that you mentioned it in your post.
In any case, I think it is interesting to add new “statistics/indicators” to the mix. I will have a better look at your library.

g-gundam · December 22, 2023, 1:40am

This is an important library to build, and I’m glad you took the initiative to get it started. I will take a look.

FemtoTrader · December 22, 2023, 3:25pm

Thanks @g-gundam for the kind words. PR are welcome !

@simsurace can you tell me how you could compute for example standard deviation with OnlineStats ? IncTA.jl/src/indicators/StdDev.jl at main · femtotrader/IncTA.jl · GitHub

latest value of statistics that OnlineStats provides is easily available but how to get for example before last, and before before last values. Does OnlineStats have a kind of mechanism for that (I’m personally using CircularBuffer for that purpose)

g-gundam · December 23, 2023, 8:36am

My goal is to give you the Stochastic indicator by this Christmas.

simsurace · December 23, 2023, 9:35am

Perhaps you can wrap Var with StatLag to get something similar?

FemtoTrader · December 24, 2023, 11:47am

Thanks @simsurace for the tip but I still don’t feel confortable with OnlineStats and OnlineStatsBase (especially MovingWindow, Lag, StatLag, CircBuff…

Could you help me to have this unit tests pass

using Test

using OnlineStatsBase
using OnlineStats

const ATOL = 0.00001
const P = 20

const CLOSE_TMPL = [
    10.5,
    9.78,
    10.46,
    10.51,
    10.55,
    10.72,
    10.16,
    10.25,
    9.4,
    9.5,
    9.23,
    8.5,
    8.8,
    8.33,
    7.53,
    7.61,
    6.78,
    8.6,
    9.21,
    8.95,
    9.22,
    9.1,
    8.31,
    8.37,
    8.3,
    7.78,
    8.05,
    8.1,
    8.08,
    7.49,
    7.58,
    8.17,
    8.83,
    8.91,
    9.2,
    9.76,
    9.42,
    9.3,
    9.32,
    9.04,
    9.0,
    9.33,
    9.34,
    8.49,
    9.21,
    10.15,
    10.3,
    10.59,
    10.23,
    10.0,
]

mutable struct SMA_v2{Tval} <: OnlineStat{Tval}
    value::Union{Missing,Tval}
    n::Int
    input::MovingWindow

    function SMA_v2{Tval}(; period = SMA_PERIOD) where {Tval}
        input = MovingWindow(Tval, period)
        new{Tval}(missing, 0, input)
    end
end

function OnlineStatsBase._fit!(ind::SMA_v2, val)
    if ind.n <= length(ind.input.value)
        ind.n += 1
    end
    fit!(ind.input, val)
    #println(ind.input.value)
    ind.value = mean(value(ind.input))
    println(ind)
end
#Base.lastindex(ind::SMA_v2) = length(ind.input.value)

#=
mutable struct Memory{T} <: OnlineStat{T}
    value::Union{Missing,T}
    n::Int
    input::MovingWindow
    function Memory{T}(ind::T; memory = 3) where {T}
        input = MovingWindow(T, memory)
        new{T}(missing, 0, input)
    end
end
=#

@testset "simple indicators" begin
    ind = SMA_v2{Float64}(period = P)
    #ind = Memory(ind)
    fit!(ind, CLOSE_TMPL)
    #@test isapprox(ind[end-2], 9.075500; atol = ATOL)
    #@test isapprox(ind[end-1], 9.183000; atol = ATOL)
    @test isapprox(value(ind), 9.308500; atol = ATOL)  # or ind.value[end]
end

g-gundam · December 24, 2023, 3:10pm

@FemtoTrader I just sent you pull request #1. It took me a little while to get my bearings inside your code, but I think I get how indicators are supposed to be structured in this library. I’m still new to Julia, and I’m not used to slinging types around, but I managed to look at the other indicators and do the right thing (I think). I hope I put things in the right place.

Some Thoughts

I was under the mistaken assumption that Stochastic just took one value, but it needed highs and lows for its math, so I ended up giving it OHLCVs. README.md may need to be updated to reflect that.
This library does its work in Base.push! but the Python’s talipp doesn’t just push but has in in-place update as well. I think it’s important to have that, because suppose you have a WebSocket giving you 1m candles, and in the span of 1 minute, that 1m candle could change many times. In that case, you’d want to update in-place instead of push. You’d only push when the timestamp of the 1m candle changes to the next minute.

g-gundam · December 24, 2023, 4:06pm

I have a stupid question.

Why would I use any of these instead of DataFrame? I presume these exist, because DataFrame wasn’t enough for some reason, but that reason is not clear to me.

FemtoTrader · December 24, 2023, 5:17pm

Because technical indicators that are calculated from a DataFrame column need the whole input vector in order to calculate new values of indicators… in a streaming environnement that’s not really appropriate because you calculate the same values several times.
I suggest reading GitHub - nardew/talipp: talipp - incremental technical analysis library for python which perfectly explains that.

talipp (or tali++) is a Python library implementing financial indicators for technical analysis. The distinctive feature of the library is its incremental computation which fits extremely well real-time applications or applications with iterative input in general.

Unlike existing libraries for technical analysis which typically have to work on the whole input vector in order to calculate new values of indicators, talipp due to its incremental architecture calculates new indicators’ values exclusively based on the delta input data. That implies, among others, it requires O(1) time to produce new values in comparison to O(n) (or worse) required by other libraries.

Supported incremental operations include:

appending new values to the input

updating the last input value

removing arbitrary number of the input values

Currently IncTA only support appending new values to the input.
I will try to tackle updating and removing features also.

But my priority will be to better integrates with OnlineStats (and implements indicators on top of OnlineStatsBase

I must admit that I first need to better understand it (see my question).
I need for example to be able to save previous values of an indicator (Should I use Lag? it seems to be deprecated in favor of CircBuf (from OnlineStatsBase) not CircularBuffer from DataStructures !) but there is also MovingWindow from OnlineStats
Depending only from OnlineStatsBase and not on the whole OnlineStats is probably a better idea.

Here is my current implementation idea

using Test

using OnlineStatsBase

const ATOL = 0.00001
const P = 20

const CLOSE_TMPL = [
    10.5,
    9.78,
    10.46,
    10.51,
    10.55,
    10.72,
    10.16,
    10.25,
    9.4,
    9.5,
    9.23,
    8.5,
    8.8,
    8.33,
    7.53,
    7.61,
    6.78,
    8.6,
    9.21,
    8.95,
    9.22,
    9.1,
    8.31,
    8.37,
    8.3,
    7.78,
    8.05,
    8.1,
    8.08,
    7.49,
    7.58,
    8.17,
    8.83,
    8.91,
    9.2,
    9.76,
    9.42,
    9.3,
    9.32,
    9.04,
    9.0,
    9.33,
    9.34,
    8.49,
    9.21,
    10.15,
    10.3,
    10.59,
    10.23,
    10.0,
]

mutable struct SMA_v2{Tval} <: OnlineStat{Tval}
    value::Union{Missing,Tval}
    n::Int
    input::CircBuff

    function SMA_v2{Tval}(; period = SMA_PERIOD) where {Tval}
        input = CircBuff(Tval, period, rev=false)
        new{Tval}(missing, 0, input)
    end
end

function OnlineStatsBase._fit!(ind::SMA_v2, val)
    if ind.n <= length(ind.input.value)
        ind.n += 1
    end
    fit!(ind.input, val)
    values = value(ind.input)
    ind.value = sum(values) / length(values)  # mean(values)
end

mutable struct Memory{T} <: OnlineStat{T}
    value::Union{Missing,T}
    n::Int
    ind::OnlineStat{T}
    history::CircBuff
    function Memory(ind::OnlineStat{T}; n = 3) where {T}
        history = CircBuff(T, n, rev=false)
        new{T}(missing, 0, ind, history)
    end
end
function OnlineStatsBase._fit!(memory::Memory, val)
    if memory.n <= length(memory.history.value)
        memory.n += 1
    end
    fit!(memory.ind, val)
    val = value(memory.ind)
    fit!(memory.history, val)
    memory.value = val
end
Base.lastindex(ind::Memory) = length(ind.history.value)
Base.getindex(ind::Memory, index) = ind.history[index]

@testset "simple indicators" begin
    ind = SMA_v2{Float64}(period = P)
    ind = Memory(ind, n = 3)
    fit!(ind, CLOSE_TMPL)
    @test isapprox(ind[end-2], 9.075500; atol = ATOL)
    @test isapprox(ind[end-1], 9.183000; atol = ATOL)
    @test isapprox(value(ind), 9.308500; atol = ATOL)  # or ind.value[end]
end

g-gundam · December 24, 2023, 5:53pm

@FemtoTrader I’m going to do Stochastic RSI next.

FemtoTrader · January 1, 2024, 5:09pm

Hi @g-gundam and all

I wish you an happy new year 2024 !
Just a post to let you know the status of the lib

Here is a table showing current status of indicators porting

Name	Description	Dependencies	Implementation status
`AccuDist`	Accumulation and Distribution	-
`ADX`	Average Directional Index	`ATR`	:heavy_exclamation_mark: Doesn’t work as expected - help wanted
`ALMA`	Arnaud Legoux Moving Average	`CircBuff`
`AO`	Awesome Oscillator	`SMA`
`Aroon`	Aroon Up/Down	-	:heavy_exclamation_mark: Doesn’t work as expected - help wanted (need to search in reversed list in order to get the right-most index)
`ATR`	Average True Range	`CircBuff`
`BB`	Bollinger Bands	`SMA`, `StdDev`
`BOP`	Balance Of Power	-
`CCI`	Commodity Channel Index	`MeanDev`
`ChaikinOsc`	Chaikin Oscillator	`AccuDist`, `EMA`
`ChandeKrollStop`	Chande Kroll Stop	`CircBuff`, `ATR`
`CHOP`	Choppiness Index	`CirBuff`, `ATR`	:heavy_exclamation_mark: Doesn’t work as expected - help wanted
`CoppockCurve`	Coppock Curve	`ROC`, `WMA`
`DEMA`	Double Exponential Moving Average	`EMA`
`DonchianChannels`	Donchian Channels	`CircBuff`
`DPO`	Detrended Price Oscillator	`CircBuff`, `SMA`
`EMA`	Exponential Moving Average	`CircBuff`
`EMV`	Ease of Movement	`CircBuff`, `SMA`
`FibRetracement`	Fibonacci Retracement		doesn’t look an indicator just a simple class with 236 382 5 618 786 values
`ForceIndex`	Force Index	prev input val, `EMA`
`HMA`	Hull Moving Average	`WMA`
`Ichimoku`	Ichimoku Clouds	`CircBuff`	5 managed sequences unit tests doesn’t exists in reference implementation
`KAMA`	Kaufman’s Adaptive Moving Average	`CircBuff`
`KeltnerChannels`	Keltner Channels	`ATR`, `EMA` with `input_modifier` to extract close value of a candle
`KST`	Know Sure Thing	`SMA`	:heavy_exclamation_mark: Doesn’t work as expected - help wanted
`KVO`	Klinger Volume Oscillator	`EMA`
`MACD`	Moving Average Convergence Divergence	`EMA`
`MassIndex`	Mass Index	`EMA`, `CircBuff`
`McGinleyDynamic`	McGinley Dynamic	`CircBuff`
`MeanDev`	Mean Deviation	`CircBuff`, `SMA`
`OBV`	On Balance Volume	prev input val
`ParabolicSAR`	Parabolic Stop And Reverse	`CirBuff`
`PivotsHL`	High/Low Pivots	`-`	unit tests in reference implementation are missing.
`ROC`	Rate Of Change	`CircBuff`
`RSI`	Relative Strength Index	`CircBuff`, `SMMA`
`SFX`	SFX	`ATR`, `StdDev`, `SMA` and `input_modifier` (to extract `close`)
`SMA`	Simple Moving Average	`CircBuff`
`SMMA`	Smoothed Moving Average	`CircBuff`
`SOBV`	Smoothed On Balance Volume	`OBV`, `SMA`
`STC`	Schaff Trend Cycle	`MACD`, `Stoch` with `input_modifier` (MACDVal->OHLCV and stoch_d->OHLCV), indicator chaining, `MAFactory` (default `SMA`)
`StdDev`	Standard Deviation	`CircBuff`
`Stoch`	Stochastic	`CircBuff`, `SMA`
`StochRSI`	Stochastic RSI	`RSI`, `SMA`
`SuperTrend`	Super Trend	`CircBuff`, `ATR`	:heavy_exclamation_mark: Doesn’t work as expected - help wanted
`T3`	T3 Moving Average	`EMA` with indicator chaining and input filter
`TEMA`	Triple Exponential Moving Average	`EMA`
`TRIX`	TRIX	`EMA`, indicator chaining
`TSI`	True Strength Index	`EMA`, indicator chaining
`TTM`	TTM Squeeze	`SMA`, `BB`, `DonchianChannels`, `KeltnerChannels` and `input_modifier` to extract `close` value of a candle
`UO`	Ultimate Oscillator	`CircBuff`
`VTX`	Vortex Indicator	`CircBuff`, `ATR`	:heavy_exclamation_mark: Doesn’t work as expected - help wanted
`VWAP`	Volume Weighted Average Price	-
`VWMA`	Volume Weighted Moving Average	`CircBuff`
`WMA`	Weighted Moving Average	`CircBuff`
`ZLEMA`	Zero Lag Exponential Moving Average	`EMA`

We have now 52 technical indicators.

As you can see StochRSI (and many others indicators have been) implemented.
We have now a mechanism for indicator chaining.

What is indicator chaining?

Indicator chaining is like using Lego blocks to create unique combinations. When you chain indicators together, output of one indicator is the input of the following indicator.

T3 for example is composed of 6 EMA (Exponential Moving Average) which are chained

So by chaining them you can replace code like


function _calculate_new_value(ind::T3)
    _ema1 = value(ind.ema1)
    if !ismissing(_ema1)
        fit!(ind.ema2, value(ind.ema1))
        _ema2 = value(ind.ema2)
        if !ismissing(_ema2)
            fit!(ind.ema3, _ema2)
            _ema3 = value(ind.ema3)
            if !ismissing(_ema3)
                fit!(ind.ema4, _ema3)
                _ema4 = value(ind.ema4)
                if !ismissing(_ema4)
                    fit!(ind.ema5, _ema4)
                    _ema5 = value(ind.ema5)
                    if !ismissing(_ema5)
                        fit!(ind.ema6, _ema5)
                    end
                end
            end
        end
    end

    if has_output_value(ind.ema6)
        return ind.c1 * value(ind.ema6) +
               ind.c2 * value(ind.ema5) +
               ind.c3 * value(ind.ema4) +
               ind.c4 * value(ind.ema3)
    else
        return missing
    end
end

by something much simpler such as


function _calculate_new_value(ind::T3)
    if has_output_value(ind.ema6)
        return ind.c1 * value(ind.ema6) +
               ind.c2 * value(ind.ema5) +
               ind.c3 * value(ind.ema4) +
               ind.c4 * value(ind.ema3)
    else
        return missing
    end
end

See T3 code

We also have now a input_filter/input_modifier functions for every indicators so we can for example feed a SISO indicator (single input single output) with candle data (OHLCV) when passing also an input_modifier function which take for example close price of a candle.

A good example of indicator which use such a mechanism is TTM (probably the most complex indicator I have implemented currently)

https://school.stockcharts.com/doku.php?id=technical_indicators:ttm_squeeze

TTM uses BB (Bollinger Bands), DonchianChannels, KeltnerChannels and a moving average (SMA by default but any other moving average can be used thanks to a moving average factory (MAFactory))

See TTM Squeeze code

Here is status of unit tests

Precompiling project...
  1 dependency successfully precompiled in 5 seconds. 45 already precompiled.
     Testing Running tests...
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\KST.jl:72
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\CHOP.jl:32
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\ADX.jl:49
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\SuperTrend.jl:45
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\VTX.jl:39
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\Aroon.jl:31
Test Summary: | Pass  Broken  Total   Time
IncTA.jl      |  788       6    794  29.2s
     Testing IncTA tests passed

So we still have some indicators which doesn’t behave like reference implementation (talipp). I must admit that I haven’t searched quite a long time on each… but enough to lose patience.

If some of you want to have a look, PR are welcome ! I think a good way to fix this could be to install latest development version of talipp (v2.x) and do some “print debugging” on internal state of indicator

Doc also need to be improved (and published !).

This project is not really complex but very time consuming given the number of indicators which are reimplemented in pure Julia (contrary to TALib.j which was a wrapper around C library talib !

My hope will be to have this lib used by some others (especially those who are building backtesters)

FemtoTrader · January 13, 2024, 2:14pm

A first documentation is available at https://femtotrader.github.io/IncTA.jl/
I’m not an english native speaker so if some of you can have a look (for typos / grammar …) that would be very nice.
If you also think that this doc needs improvement please contribute by opening issue / creating PR on GitHub…

g-gundam · January 17, 2024, 11:59pm

I like how you organized the docs. It looks very clean, and you went above and beyond by including videos in the Usage section. The examples in the Learn more about usage are very helpful too. This is probably the best TA library available for Julia now.

feribg · April 4, 2024, 5:03am

I am trying to wrap my head around the library and see some room for improvements performance wise and i have a few questions design wise.

The root fit! method which seems to be very hot uses reflection and branching. Is it not better to make it generic and specialize the indicator type on things such as inputValues, subIndicators etc ?

There’s a lot of branching within the indicators themselves too for the same exact reason.

The second main bottleneck for me in the current implementation is the way the inputValues are handled. The current code assumes every indicator needing market history will get a copy of the InputValues circular buffer and then update it on every fit!, but a very common scenario in calculating those in the real world is a shared price buffer and then each indicator being a reader on that price buffer, which is usually updated only once at the beginning of the indicator loop, the current implementation doesn’t allow for that even if the backing vector and CircBuff are shared and passed to the indicator constructors, every indicator will update it multiple times.

@FemtoTrader im wondering if you would be open to some discussion about implementing some of these changes (even as a different alpha API) or performance is less of a concern than convenience. Have you done any benchmarks on the overhead vs some vectorized Julia TA implementation out of curiosity?

FemtoTrader · April 4, 2024, 4:56pm

I haven’t done any benchmark but I’m open to contributions.

You should have also in mind this Donald Knuth statements

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%

Be aware that in a SMA(5) you only keep 5 (or 6) values… what ever the number of incoming data you have. Not sure that’s a big issue.

feribg · April 4, 2024, 5:52pm

Well yes i guess depends on what you’re doing, if you’re doing 1000s of permutations on intraday data the current approach will become a bottleneck I can run some benchmarks. The branching also eliminates all possibilities for simd and fusing stuff. The issue is not so much the fact that you keep 5 or 10 or whatever number it’s that if you have 10 indicators all updating the buffer needlessly, you’re hot loop get’s destroyed, basically the cache is invalid all the time…

My ideal goal was that streaming and batching indicators can play nicely together and ideally be one and the same, but the more i look into how to make a nice abstraction the tougher it gets and seems like just doing a fuzz automated test between the batch and stream version is a simpler goal than rearchitecting them. I guess im just a lot more willing to make sacrifices in terms of the interface and API in liue of performance.

FemtoTrader · May 2, 2024, 6:23am

Just a small message to let you now that IncTA.jl is now named OnlineTechnicalIndicators.jl because of package naming guidelines. See discussion.