Incremental Technical Analysis indicators

Hi,

Unlike existing libraries for technical analysis which typically have to work on the whole input vector in order to calculate new values of indicators I’m looking for a Julia library which implements technical analysis indicators in an incremental approach.

talipp (or tali++) GitHub - nardew/talipp: talipp - incremental technical analysis library for python provides such implementation in Python
Same for this TA-Lib fork GitHub - trufanov-nok/ta-lib-rt: TA-Lib RT is a fork of TA-Lib that provides additional API for incremental calculation of indicators without reprocessing whole data. (C)

Why I think it’s important ?
Because we have in the Julia ecosystem so much timeseries implementation (Temporal.TS, TimeSeries.TimeArray, TSFrames.TSFrame… and many other outdated lib).
So building a TA lib on top of this is probably not a good idea… especially when dealing with Real Time data.

I tried to implement some indicators using such a manner.

SMA (simple moving average) and EMA (exponential moving average) have been implemented and tested against same test cases than talipp.

I tried implementing SMA using CircularBuffer from DataStructures but also using MovingWindow from OnlineStats.

But I’m stuck with SMMA.

I wonder if some of you can watch GitHub - femtotrader/IncTA.jl: Julia Incremental Technical Analysis Indicators (inspired by talipp)
and have a look why SMMA doesn’t behave as expected.

Moreover I also wonder if some other dev are interested by such a project which could be used for backtesting strategies, paper trading and real trading.

Kind regards

PS : SMMA is fixed now… but the road is long to get all indicators running

4 Likes

Have you looked at OnlineStats.jl?
EDIT: sorry I just saw that you mentioned it in your post.
In any case, I think it is interesting to add new “statistics/indicators” to the mix. I will have a better look at your library.

1 Like

This is an important library to build, and I’m glad you took the initiative to get it started. I will take a look.

1 Like

Thanks @g-gundam for the kind words. PR are welcome !

@simsurace can you tell me how you could compute for example standard deviation with OnlineStats ? IncTA.jl/src/indicators/StdDev.jl at main · femtotrader/IncTA.jl · GitHub

latest value of statistics that OnlineStats provides is easily available but how to get for example before last, and before before last values. Does OnlineStats have a kind of mechanism for that (I’m personally using CircularBuffer for that purpose)

1 Like

My goal is to give you the Stochastic indicator by this Christmas.

1 Like

Perhaps you can wrap Var with StatLag to get something similar?

Thanks @simsurace for the tip but I still don’t feel confortable with OnlineStats and OnlineStatsBase (especially MovingWindow, Lag, StatLag, CircBuff…

Could you help me to have this unit tests pass

using Test

using OnlineStatsBase
using OnlineStats

const ATOL = 0.00001
const P = 20

const CLOSE_TMPL = [
    10.5,
    9.78,
    10.46,
    10.51,
    10.55,
    10.72,
    10.16,
    10.25,
    9.4,
    9.5,
    9.23,
    8.5,
    8.8,
    8.33,
    7.53,
    7.61,
    6.78,
    8.6,
    9.21,
    8.95,
    9.22,
    9.1,
    8.31,
    8.37,
    8.3,
    7.78,
    8.05,
    8.1,
    8.08,
    7.49,
    7.58,
    8.17,
    8.83,
    8.91,
    9.2,
    9.76,
    9.42,
    9.3,
    9.32,
    9.04,
    9.0,
    9.33,
    9.34,
    8.49,
    9.21,
    10.15,
    10.3,
    10.59,
    10.23,
    10.0,
]

mutable struct SMA_v2{Tval} <: OnlineStat{Tval}
    value::Union{Missing,Tval}
    n::Int
    input::MovingWindow

    function SMA_v2{Tval}(; period = SMA_PERIOD) where {Tval}
        input = MovingWindow(Tval, period)
        new{Tval}(missing, 0, input)
    end
end

function OnlineStatsBase._fit!(ind::SMA_v2, val)
    if ind.n <= length(ind.input.value)
        ind.n += 1
    end
    fit!(ind.input, val)
    #println(ind.input.value)
    ind.value = mean(value(ind.input))
    println(ind)
end
#Base.lastindex(ind::SMA_v2) = length(ind.input.value)

#=
mutable struct Memory{T} <: OnlineStat{T}
    value::Union{Missing,T}
    n::Int
    input::MovingWindow
    function Memory{T}(ind::T; memory = 3) where {T}
        input = MovingWindow(T, memory)
        new{T}(missing, 0, input)
    end
end
=#

@testset "simple indicators" begin
    ind = SMA_v2{Float64}(period = P)
    #ind = Memory(ind)
    fit!(ind, CLOSE_TMPL)
    #@test isapprox(ind[end-2], 9.075500; atol = ATOL)
    #@test isapprox(ind[end-1], 9.183000; atol = ATOL)
    @test isapprox(value(ind), 9.308500; atol = ATOL)  # or ind.value[end]
end

@FemtoTrader I just sent you pull request #1. :christmas_tree: It took me a little while to get my bearings inside your code, but I think I get how indicators are supposed to be structured in this library. I’m still new to Julia, and I’m not used to slinging types around, but I managed to look at the other indicators and do the right thing (I think). I hope I put things in the right place.

Some Thoughts

  • I was under the mistaken assumption that Stochastic just took one value, but it needed highs and lows for its math, so I ended up giving it OHLCVs. README.md may need to be updated to reflect that.
  • This library does its work in Base.push! but the Python’s talipp doesn’t just push but has in in-place update as well. I think it’s important to have that, because suppose you have a WebSocket giving you 1m candles, and in the span of 1 minute, that 1m candle could change many times. In that case, you’d want to update in-place instead of push. You’d only push when the timestamp of the 1m candle changes to the next minute.

I have a stupid question.

Why would I use any of these instead of DataFrame? I presume these exist, because DataFrame wasn’t enough for some reason, but that reason is not clear to me.

Because technical indicators that are calculated from a DataFrame column need the whole input vector in order to calculate new values of indicators… in a streaming environnement that’s not really appropriate because you calculate the same values several times.
I suggest reading GitHub - nardew/talipp: talipp - incremental technical analysis library for python which perfectly explains that.

talipp (or tali++) is a Python library implementing financial indicators for technical analysis. The distinctive feature of the library is its incremental computation which fits extremely well real-time applications or applications with iterative input in general.

Unlike existing libraries for technical analysis which typically have to work on the whole input vector in order to calculate new values of indicators, talipp due to its incremental architecture calculates new indicators’ values exclusively based on the delta input data. That implies, among others, it requires O(1) time to produce new values in comparison to O(n) (or worse) required by other libraries.

Supported incremental operations include:

  • appending new values to the input
  • updating the last input value
  • removing arbitrary number of the input values

Currently IncTA only support appending new values to the input.
I will try to tackle updating and removing features also.

But my priority will be to better integrates with OnlineStats (and implements indicators on top of OnlineStatsBase

I must admit that I first need to better understand it (see my question).
I need for example to be able to save previous values of an indicator (Should I use Lag? it seems to be deprecated in favor of CircBuf (from OnlineStatsBase) not CircularBuffer from DataStructures !) but there is also MovingWindow from OnlineStats
Depending only from OnlineStatsBase and not on the whole OnlineStats is probably a better idea.

Here is my current implementation idea

using Test

using OnlineStatsBase

const ATOL = 0.00001
const P = 20

const CLOSE_TMPL = [
    10.5,
    9.78,
    10.46,
    10.51,
    10.55,
    10.72,
    10.16,
    10.25,
    9.4,
    9.5,
    9.23,
    8.5,
    8.8,
    8.33,
    7.53,
    7.61,
    6.78,
    8.6,
    9.21,
    8.95,
    9.22,
    9.1,
    8.31,
    8.37,
    8.3,
    7.78,
    8.05,
    8.1,
    8.08,
    7.49,
    7.58,
    8.17,
    8.83,
    8.91,
    9.2,
    9.76,
    9.42,
    9.3,
    9.32,
    9.04,
    9.0,
    9.33,
    9.34,
    8.49,
    9.21,
    10.15,
    10.3,
    10.59,
    10.23,
    10.0,
]

mutable struct SMA_v2{Tval} <: OnlineStat{Tval}
    value::Union{Missing,Tval}
    n::Int
    input::CircBuff

    function SMA_v2{Tval}(; period = SMA_PERIOD) where {Tval}
        input = CircBuff(Tval, period, rev=false)
        new{Tval}(missing, 0, input)
    end
end

function OnlineStatsBase._fit!(ind::SMA_v2, val)
    if ind.n <= length(ind.input.value)
        ind.n += 1
    end
    fit!(ind.input, val)
    values = value(ind.input)
    ind.value = sum(values) / length(values)  # mean(values)
end

mutable struct Memory{T} <: OnlineStat{T}
    value::Union{Missing,T}
    n::Int
    ind::OnlineStat{T}
    history::CircBuff
    function Memory(ind::OnlineStat{T}; n = 3) where {T}
        history = CircBuff(T, n, rev=false)
        new{T}(missing, 0, ind, history)
    end
end
function OnlineStatsBase._fit!(memory::Memory, val)
    if memory.n <= length(memory.history.value)
        memory.n += 1
    end
    fit!(memory.ind, val)
    val = value(memory.ind)
    fit!(memory.history, val)
    memory.value = val
end
Base.lastindex(ind::Memory) = length(ind.history.value)
Base.getindex(ind::Memory, index) = ind.history[index]

@testset "simple indicators" begin
    ind = SMA_v2{Float64}(period = P)
    ind = Memory(ind, n = 3)
    fit!(ind, CLOSE_TMPL)
    @test isapprox(ind[end-2], 9.075500; atol = ATOL)
    @test isapprox(ind[end-1], 9.183000; atol = ATOL)
    @test isapprox(value(ind), 9.308500; atol = ATOL)  # or ind.value[end]
end
1 Like

@FemtoTrader I’m going to do Stochastic RSI next.

1 Like

Hi @g-gundam and all

I wish you an happy new year 2024 !
Just a post to let you know the status of the lib

Here is a table showing current status of indicators porting

Name Description Input Output Dependencies Implementation status
AccuDist Accumulation and Distribution :candle: :1234: - :heavy_check_mark:
ADX Average Directional Index :candle: :m: ATR :heavy_exclamation_mark: Doesn’t work as expected - help wanted
ALMA Arnaud Legoux Moving Average :1234: :1234: CircBuff :heavy_check_mark:
AO Awesome Oscillator :candle: :1234: SMA :heavy_check_mark:
Aroon Aroon Up/Down :candle: :m: - :heavy_exclamation_mark: Doesn’t work as expected - help wanted (need to search in reversed list in order to get the right-most index)
ATR Average True Range :candle: :1234: CircBuff :heavy_check_mark:
BB Bollinger Bands :1234: :m: SMA, StdDev :heavy_check_mark:
BOP Balance Of Power :candle: :1234: - :heavy_check_mark:
CCI Commodity Channel Index :candle: :1234: MeanDev :heavy_check_mark:
ChaikinOsc Chaikin Oscillator :candle: :1234: AccuDist, EMA :heavy_check_mark:
ChandeKrollStop Chande Kroll Stop :candle: :m: CircBuff, ATR :heavy_check_mark:
CHOP Choppiness Index :candle: :1234: CirBuff, ATR :heavy_exclamation_mark: Doesn’t work as expected - help wanted
CoppockCurve Coppock Curve :1234: :1234: ROC, WMA :heavy_check_mark:
DEMA Double Exponential Moving Average :1234: :1234: EMA :heavy_check_mark:
DonchianChannels Donchian Channels :candle: :m: CircBuff :heavy_check_mark:
DPO Detrended Price Oscillator :1234: :1234: CircBuff, SMA :heavy_check_mark:
EMA Exponential Moving Average :1234: :1234: CircBuff :heavy_check_mark:
EMV Ease of Movement :candle: :1234: CircBuff, SMA :heavy_check_mark:
FibRetracement Fibonacci Retracement :question: :question: doesn’t look an indicator just a simple class with 236 382 5 618 786 values
ForceIndex Force Index :candle: :1234: prev input val, EMA :heavy_check_mark:
HMA Hull Moving Average :1234: :1234: WMA :heavy_check_mark:
Ichimoku Ichimoku Clouds :1234: :m: CircBuff 5 managed sequences :question: unit tests doesn’t exists in reference implementation
KAMA Kaufman’s Adaptive Moving Average :1234: :1234: CircBuff :heavy_check_mark:
KeltnerChannels Keltner Channels :candle: :m: ATR, EMA with input_modifier to extract close value of a candle :heavy_check_mark:
KST Know Sure Thing :1234: :m: SMA :heavy_exclamation_mark: Doesn’t work as expected - help wanted
KVO Klinger Volume Oscillator :candle: :1234: EMA :heavy_check_mark:
MACD Moving Average Convergence Divergence :1234: :m: EMA :heavy_check_mark:
MassIndex Mass Index :candle: :1234: EMA, CircBuff :heavy_check_mark:
McGinleyDynamic McGinley Dynamic :1234: :1234: CircBuff :heavy_check_mark:
MeanDev Mean Deviation :1234: :1234: CircBuff, SMA :heavy_check_mark:
OBV On Balance Volume :candle: :1234: prev input val :heavy_check_mark:
ParabolicSAR Parabolic Stop And Reverse :candle: :m: CirBuff :heavy_check_mark:
PivotsHL High/Low Pivots :candle: :m: - :construction: unit tests in reference implementation are missing.
ROC Rate Of Change :1234: :1234: CircBuff :heavy_check_mark:
RSI Relative Strength Index :1234: :1234: CircBuff, SMMA :heavy_check_mark:
SFX SFX :candle: :m: ATR, StdDev, SMA and input_modifier (to extract close) :heavy_check_mark:
SMA Simple Moving Average :1234: :1234: CircBuff :heavy_check_mark:
SMMA Smoothed Moving Average :1234: :1234: CircBuff :heavy_check_mark:
SOBV Smoothed On Balance Volume :candle: :1234: OBV, SMA :heavy_check_mark:
STC Schaff Trend Cycle :1234: :1234: MACD, Stoch with input_modifier (MACDVal->OHLCV and stoch_d->OHLCV), indicator chaining, MAFactory (default SMA) :heavy_check_mark:
StdDev Standard Deviation :1234: :1234: CircBuff :heavy_check_mark:
Stoch Stochastic :candle: :m: CircBuff, SMA :heavy_check_mark: :christmas_tree:
StochRSI Stochastic RSI :1234: :m: RSI, SMA :heavy_check_mark:
SuperTrend Super Trend :candle: :m: CircBuff, ATR :heavy_exclamation_mark: Doesn’t work as expected - help wanted
T3 T3 Moving Average :1234: :1234: EMA with indicator chaining and input filter :heavy_check_mark:
TEMA Triple Exponential Moving Average :1234: :1234: EMA :heavy_check_mark:
TRIX TRIX :candle: :m: EMA, indicator chaining :heavy_check_mark:
TSI True Strength Index :1234: :1234: EMA, indicator chaining :heavy_check_mark:
TTM TTM Squeeze :candle: :m: SMA, BB, DonchianChannels, KeltnerChannels and input_modifier to extract close value of a candle :heavy_check_mark:
UO Ultimate Oscillator :candle: :1234: CircBuff :heavy_check_mark:
VTX Vortex Indicator :candle: :m: CircBuff, ATR :heavy_exclamation_mark: Doesn’t work as expected - help wanted
VWAP Volume Weighted Average Price :candle: :1234: - :heavy_check_mark:
VWMA Volume Weighted Moving Average :candle: :1234: CircBuff :heavy_check_mark:
WMA Weighted Moving Average :1234: :1234: CircBuff :heavy_check_mark:
ZLEMA Zero Lag Exponential Moving Average :1234: :1234: EMA :heavy_check_mark:

We have now 52 technical indicators.

As you can see StochRSI (and many others indicators have been) implemented.
We have now a mechanism for indicator chaining.

What is indicator chaining?

Indicator chaining is like using Lego blocks to create unique combinations. When you chain indicators together, output of one indicator is the input of the following indicator.

T3 for example is composed of 6 EMA (Exponential Moving Average) which are chained

So by chaining them you can replace code like


function _calculate_new_value(ind::T3)
    _ema1 = value(ind.ema1)
    if !ismissing(_ema1)
        fit!(ind.ema2, value(ind.ema1))
        _ema2 = value(ind.ema2)
        if !ismissing(_ema2)
            fit!(ind.ema3, _ema2)
            _ema3 = value(ind.ema3)
            if !ismissing(_ema3)
                fit!(ind.ema4, _ema3)
                _ema4 = value(ind.ema4)
                if !ismissing(_ema4)
                    fit!(ind.ema5, _ema4)
                    _ema5 = value(ind.ema5)
                    if !ismissing(_ema5)
                        fit!(ind.ema6, _ema5)
                    end
                end
            end
        end
    end

    if has_output_value(ind.ema6)
        return ind.c1 * value(ind.ema6) +
               ind.c2 * value(ind.ema5) +
               ind.c3 * value(ind.ema4) +
               ind.c4 * value(ind.ema3)
    else
        return missing
    end
end

by something much simpler such as


function _calculate_new_value(ind::T3)
    if has_output_value(ind.ema6)
        return ind.c1 * value(ind.ema6) +
               ind.c2 * value(ind.ema5) +
               ind.c3 * value(ind.ema4) +
               ind.c4 * value(ind.ema3)
    else
        return missing
    end
end

See T3 code

We also have now a input_filter/input_modifier functions for every indicators so we can for example feed a SISO indicator (single input single output) with candle data (OHLCV) when passing also an input_modifier function which take for example close price of a candle.

A good example of indicator which use such a mechanism is TTM (probably the most complex indicator I have implemented currently)

https://school.stockcharts.com/doku.php?id=technical_indicators:ttm_squeeze

TTM uses BB (Bollinger Bands), DonchianChannels, KeltnerChannels and a moving average (SMA by default but any other moving average can be used thanks to a moving average factory (MAFactory))

See TTM Squeeze code

Here is status of unit tests

Precompiling project...
  1 dependency successfully precompiled in 5 seconds. 45 already precompiled.
     Testing Running tests...
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\KST.jl:72
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\CHOP.jl:32
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\ADX.jl:49
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\SuperTrend.jl:45
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\VTX.jl:39
┌ Warning: WIP - buggy
└ @ IncTA C:\Users\femto\.julia\dev\IncTA\src\indicators\Aroon.jl:31
Test Summary: | Pass  Broken  Total   Time
IncTA.jl      |  788       6    794  29.2s
     Testing IncTA tests passed

So we still have some indicators which doesn’t behave like reference implementation (talipp). I must admit that I haven’t searched quite a long time on each… but enough to lose patience.

If some of you want to have a look, PR are welcome ! I think a good way to fix this could be to install latest development version of talipp (v2.x) and do some “print debugging” on internal state of indicator

Doc also need to be improved (and published !).

This project is not really complex but very time consuming given the number of indicators which are reimplemented in pure Julia (contrary to TALib.j which was a wrapper around C library talib !

My hope will be to have this lib used by some others (especially those who are building backtesters)

2 Likes

A first documentation is available at https://femtotrader.github.io/IncTA.jl/
I’m not an english native speaker so if some of you can have a look (for typos / grammar …) that would be very nice.
If you also think that this doc needs improvement please contribute by opening issue / creating PR on GitHub…

2 Likes

I like how you organized the docs. It looks very clean, and you went above and beyond by including videos in the Usage section. The examples in the Learn more about usage are very helpful too. This is probably the best TA library available for Julia now.

3 Likes