[ANN] Durbyn.jl — Time Series Forecasting in Julia

@langestefan I should clarify that I wasn’t referring specifically to the packages mentioned, but speaking more generally: I can’t just add dependencies simply because they exist. Every dependency introduces long-term maintenance risks, so I’m careful about what I include.

I’m also not undermining the importance of Optim.jl — it’s a great package. But in this particular use case, it didn’t perform as well as needed. More broadly, time series forecasting in Julia is still not a mature area, and sometimes it requires specialized tools. That’s exactly what Durbyn.jl is trying to provide. For example, in the R ecosystem, tsibble was developed specifically for time series even though tibble and data frames already existed - because the needs were different.

Without tsibble, the tidy forecasting framework (Fable) in R simply wouldn’t have been possible.

Similarly, why should I bring in GLM.jl just to compute OLS residuals, when I can simply do:

β = X \ y
fitted = X * β
residuals = y - fitted


function ols(y, X)
β = X \ y
fitted = X * β
residuals = y - fitted
n, p = size(X)
df_residual = n - p
σ2 = sum(residuals .^ 2) / df_residual
XtX = X' * X
cov_β = σ2 * inv(XtX)
se = sqrt.(diag(cov_β))
return OlsFit(β, fitted, residuals, σ2, cov_β, se, df_residual)
end

and wrap that in a lightweight function called ols? That avoids adding a heavy dependency while still providing the functionality needed.

I’m a bit puzzled by the criticism here. For my use case I just need a very small, labeled matrix container—row/column names with shape checks—nothing like the full feature set of DataFrames.jl. Pulling in DataFrames for this would add a heavy dependency which Duryn doesn’t need.


struct NamedMatrix{T}
    data::Matrix{T}
    rownames::Union{Vector{String},Nothing}
    colnames::Vector{String}

    function NamedMatrix{T}(data::Matrix{T},
                            rownames::Union{Vector{String},Nothing},
                            colnames::Vector{String}) where {T}
        if rownames !== nothing && size(data,1) != length(rownames)
            error("Row names do not match number of rows")
        end
        if size(data,2) != length(colnames)
            error("Col names do not match number of columns")
        end
        new{T}(data, rownames, colnames)
    end
end

1 Like

Sure, that makes sense. But packages like StatsModels.jl, Optim.jl, Tables.jl are used by lots of downstream packages with many thousands of users that depend on it. So for me personally that is enough ensurance that those packages will continue to be maintained. But this is always a risk with open source software and I don’t think you can ever avoid it.

I haven’t looked at your implementation, but can’t you just plug in any BBO from BlackBoxOptim.jl? Why specifically nelder-mead?

GLM.jl is a very old package from 2012. But actually, there are lots of reasons why you would not want to write your own solver and pick something proven/battle hardened instead (absolutely not GLM.jl because it is a very high level package). I don’t want to discourage your development by the way, if your solver works better then it’s better. But as a user, I would prefer to just pick something off the shelf which I am already using in other packages.

I think you misunderstand that suggestion. You do not need a dependency on DataFrames.jl to support DataFrames.jl as an input. You only need to support the Tables.jl interface, and then you can take any input which is compatible with it. That includes dataframes, but your package isn’t even aware it is taking a dataframe as input :slight_smile:

1 Like

Can I ask how you’ve determined that your Nelder-Mead implementation is faster (for equivalent tolerances) than the Optim.jl one?

If I just run the example from the Optim Nelder Mead docs I get:

julia> using Chairmarks, Optim

julia> f(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
f (generic function with 1 method)

julia> optimize(f, [.0, .0])
 * Status: success

 * Candidate solution
    Final objective value:     3.525527e-09

 * Found with
    Algorithm:     Nelder-Mead

 * Convergence measures
    √(Σ(yᵢ-ȳ)²)/n ≤ 1.0e-08

 * Work counters
    Seconds run:   0  (vs limit Inf)
    Iterations:    60
    f(x) calls:    117


julia> nmmin(f, [.0, .0], NelderMeadOptions())
(x_opt = [1.0006866976618765, 1.0021003276109683], f_opt = 5.324607341588884e-5, n_iter = 100, fail = 0, evals = 74)

julia> @b optimize(f, [.0, .0])
10.200 μs (320 allocs: 7.375 KiB)

julia> @b nmmin(f, [.0, .0], NelderMeadOptions())
8.600 μs (825 allocs: 36.078 KiB)

so the Durbyn implementation (which I just copy-pasted from the nmmin.jl file in the repo) is about 15% faster (while allocating 2.5x the memory), but finding an objective value that’s 4 orders of magnitude worse, so this difference seems more of a tolerance thing. When tuning the Durbyn implementation tolerance to get a similar objective value:

julia> nmmin(f, [.0, .0], NelderMeadOptions(; abstol = 1e-8))
(x_opt = [0.999950642570824, 0.9999048113257563], f_opt = 3.6778357780875667e-9, n_iter = 100, fail = 0, evals = 106)

julia> @b nmmin(f, [.0, .0], NelderMeadOptions(; abstol = 1e-8))
12.100 μs (1169 allocs: 51.016 KiB)

I see 20% lower performance (and 4x memory allocation) relative to Optim.jl

Of course the function I’m optimizing here might not be a good proxy for the typical workload in Durbyn, but my point more broadly is that it’s not easy to reliably benchmark these things, especially when building a user-facing package where you might not be able to perfectly anticipate the workloads.

More broadly, optimization is a fundamental building block of the Julia ecosystem, and there are many domain experts in the community willing to help out with optimization problems. By rolling your own you’re missing out on one of the greatest strenghts of the Julia community.

And just to add - the flipside of that is of course that if you actually have a Nelder Mead implementation that reliably outperforms the Optim, the best thing to do for the community would be to upstream this into Optim.jl so that everyone (not just time series forecasters using Durbyn) can benefit from it! This will also buy you community support on maintaining or improving your implementation.

4 Likes

I did experiment with different solvers, it was not performing well for ets() that is why I start working on NM implementation.

Data frame interface is some long way to go but an interface similar to R Fable would be nice maybe as another package.

@nilshg Optimazation in ETS function done vie opim · taf-society/Durbyn.jl@1db4d0f · GitHub

I worked a lot on Optim.jl to be integrated in Durbyn, experimented also a lot.
This commit is only one example. There were many other experiments that I did not commit to git.
As I repeatedly said Optim.jl is a great package, but for this case I will go in house.
Please feel free to fork the package change the optimization function and create pull request if it perform better then Durbyn implementation of the nm.

2 Likes

Thank you very much for your package, Resul Akay. It looks extremely well thought out with a super clean interface. I also hope that the TAFS Forecasting Ecosystem, supported by the Time Series Analysis and Forecasting Society, will become a permanent part of the Julia Ecosystem. Taking this opportunity, may I ask if you are planning to further develop the package and to support more contemporary time series analysis and forecasting methods in the future?

4 Likes

Congrats! A very nice package.

It’s also good to see that we have a Julia counterpart of the R’s well-known forecast package. I’ve just tried it out and looks like the auto_arima function works like a charm.

Thank you!

2 Likes

@j_u Yes, I’m leading the development of Durbyn.jl and planning to extend it further — including ML-based forecasting. I also develop in R and Python, but Julia has been an amazing language to work with for high-performance forecasting.

Thank you very much for your kind words and encouragement! I really appreciate the friendliness of the community. I realize I may have set off on the wrong foot in some earlier discussions, and at times it felt a bit like criticism was directed at me personally. But I take this as part of the learning process, and I’m committed to making Durbyn and the TAFS Forecasting Ecosystem a lasting contribution to Julia’s ecosystem.

10 Likes

I made a PR to show how you could implement the Tables.jl interface: Tables.jl interface example and other tweaks by langestefan · Pull Request #3 · taf-society/Durbyn.jl · GitHub

As you can see it’s only a single function, very simple.

But your package currently doesn’t have a unified interface, so you would have to implement the dispatch for every single model. That’s not very nice. To make it more flexible, you can have a single forecast method which then dispatches to specific algorithms.

I also noticed you didn’t have any unit tests or a runtests.jl, so I added that. Now you can do ] test to run unit tests. It also shows how to use your forecast models with a table as input.

3 Likes

@langestefan Thank you for the effort. I will check.
Yes, tests are still missing, I had my hands full :slight_smile:

1 Like

When I was talking about data frame interface I was talking about a following process which allow to do the forecast at scale:

  1. Shape the data into a panel which imported from a csv or excel which contains let’s say 100K products
pt   = PanelTable(data; groups=[:product_id], date=:date, freq=:month, ml_data=false)

  1. Create a ModelSpec (example)
models = ModelSpec(
  ETS(y = . ),     # ETS example (univariate) auto ets
  ARIMA(y = p() + d() + q() + P(0) + D(0) + Q(0)), #non seasonal auto_arima with drift term
 )

  1. Fit 100K time series (products) in parallel over series
fit = fit(model_spec = models, data = pt, parallel = true, ncore = -1)

4a) Forecast a fixed horizon (univariate models like ETS)

fc = forecast(fit, 12)              # 12 steps ahead for all products

4b) Or, if the model uses exogenous features:

fc = forecast(fitobj, newdata = X_future)  # X_future aligned with horizon and groups

the base models will always be with array, if I introduce tables at this stage I know I will be regretting in the future. Thanks again for your effort.

But this you can also do perfectly with StatsModels.jl right?

My PR did not change that, it just added a dispatch on tables. Nothing was changed about your model internals. That’s the power of julia’s dispatch!

I think the way you have implemented the interface right now you just made it more inconvenient to maintain for yourself.

yes, I use it a lot.

I will check. thanks for suggestion.

I also organizing a workshop with many leading forecasting experts to talk about “The grammar of forecasting” from leading python and R developers and academic community.

1 Like