[ANN] PortfolioOptimisers.jl: Ape together strong

Package Announcements
, , , ,
17

PortfolioOptimisers.jl v0.31.0

Alright, it’s been some time since v0.26.0. These updates have brought gigantic changes. What had started as a trepidatious wade into AI assisted development, has turned into a fully-fledged clanker assisted/enabled workflow.

Now before you think it’s all vibecoded, it’s not. I’ve made a post about my careful approach to using clankers here. I must say, it’s been a huge net positive for the library and my free time. They have enabled the addition of features that aren’t complete game-changers but nice to haves that are annoying to try to implement. They’ve also straight up supercharged the development of other features I’ve been wanting to add for a while. Here’s a little list of non-feature things where the clankers have been straight up buffs to the library’s general maintenance.

  1. Standardise and greatly improve documentation—including missing references, algorithm explanations, missing mathematical descriptions. There are some rough edges, for example, the clankers added some very specific numeric examples that they worked through while checking the code’s correctness and behaviour.
  2. I’ve set up a process which sweeps through the code to ensure compliance with standards, code correctness, improve type stability, reduce complexity, and keep file size in check whilst keeping a logical framework.
  3. It’s allowed me to flesh out examples. At the time of writing (while the latest PR is under review) there are 60 examples and 9 user guide entries.
  4. It’s found and fixed a variety of bugs. To my great delight, not that many correctness ones. Though there was a pretty egregious one in denoising, which didn’t affect most cases and was thus missed by the tests. Most of the fixes have had to do with how the library composes itself.
  5. Many new features.
  6. Hugely increased coverage, not of code (though that’s improved too), but of scenarios (see point 3).

New feature summary

  • Quantile portfolios. Uses especially calibrated L1-norms to apply an uncertainty set on the expected returns (or any characteristic vector) to produce equal weighted (assuming a plain return maximisation with no other constraints) portfolios for a specific number or percentage of securities based on how performant each one has been. It’s got two versions:
    • An unsigned version, i.e. the best performing x % (relative), or the best performing n (absolute).
    • A signed version, i.e. the best and worst x % (relative), or the best and worst performing n (absolute).
  • New budget constraints, and the ability to turn the old ones from slack constraints into exact ones.
  • Ability to provide more than one expected return. This enables using linear combinations of characteristics vectors which work with all the existing expected returns machinery. A fun use case is to use ESG scores and use the quantile portfolio formulation to pick the best ones based on that. Another interesting usecase could be a centrality score which could be used to generate portfolios whose assets sit on the periphery or the nucleus of the universe. It could also be any other vector or combination of them.
  • Phylogeny constraints gained a continuous analogue based on the distances in a weighted adjacency graph rather than by degrees of separation.
  • Lots of fields gained the ability to be calibrated on the fly. Inspired by the quintile portfolio, i thought it would be a good idea to provide a way to automatically calibrate various parameters based on the the returns distribution. The point wasn’t to find the best values to get the best fits, but rather finding the parameters which return the most robust portfolios without losing all signal. This has been extended to all L-type penalties and constraints, but also significance, deformation, tail weight, and ambiguity sets. It’s pretty cool.
  • Online walk-forward (and by extension multiple randomised) cross validation. Most priors and moment estimators (including higher order ones) gained the ability to be updated online, thus saving on recompute time at every step.
  • Cross sectional factor prior. This lets you find the relationship between external, non-returns data to the behaviour of securities. For example, if you want to correlate the orange man’s degen hour twitter feed to the price of geriatric nappies and thus the value of walmart, you can do so now.
  • Weight drift, previous weight source, and performance evaluation. In sequential cross validation, it’s now possible to choose whether to compute the drifted weights. As the portfolio evolves in time, its asset returns cause weights returned by the optimisation to change as the relative value of each security changes as the securities gain or lose value as dictated by the returns. Or keep them at the target weights, which are the weights returned by the optimiser for the previous period. And whether the constraints, risk measures and fees which use the previous weights (like turnover and tracking) take the drifted or target weights.
  • Everything is now NaN aware. Which means the library now deals with late listings, delistings, holidays, cloudflare outages, etc. This is in the context of batch and online cross-validation. Both batch and online will handle late listing, delisting, and halts.
  • Forced proportional and fixed liquidation fees, for when a security stops being investible.
  • Vanilla moment estimators (including high order ones) now handle late listing, delisting, and halts based on a coverage policy, whose supertype can be subtyped by users to create their own.
  • Regime adjusted moments. These natively handle late listing, delisting, and halts.

Bug fixes

  • Bug fix to matrix denoising (dang).
  • Bug fix with how fixed fees were applied (oof).
  • Bug fix in one of the GerberIQ templates (yikes).
  • Bug fix to one of the Black-Litterman priors with double counting the risk free rate (F in chat)
  • A failed optimisation in walk forward cross validation would forward its vector of NaN weights to the next unlucky schmuck and frick shit up for the rest of em. It now forwards the last successful weights.
  • Lots of bugs with esoteric compositions not returning auxiliary data. Really only used for bookkeeping and reporting (these could be manually generated by running through the steps explicitly so not that big a deal, just a nuisance).

Misc

  • Tons of ergonomics changes, including the cleanup of overloadable function APIs. For example defining custom constraints and objective function penalties.
  • More security changes, mainly to do with preventing accidental OOM. Can be set via Preferences.jl or by calling a function to change their values.
  • Pretty printing no longer defaults to printing nothing. It’s now opt-in, so pretty printing will now be concise and not take 3 pages of the terminal for the more esoteric compositions.

Breaking changes and migration guide

Since there’s been quite a few breaking changes instead of pasting them here I’ll refer you to the release notes and new migration guide in the docs. The migration guides were clanker written because frankly i trust their pedantry more than i trust my own.

Plans

  • Thanks to the kind folk behind the development of Copulas.jl and VineCopulas.jl, i have all i need to implement the synthetic returns prior. So that’s the next feature on the list.
  • Online portfolio optimisation/selection. What currently exists in the library concerns the prior and moments, as the optimisation models are rebuilt every time. However, there’s a few truly online methods and those aren’t in the library yet.
  • Keep sweeping the library to improve the docs, find and fix bugs, improve type stability, coverage, performance, and ergonomics.
  • I’m thinking about the 1.0 roadmap, but that may be some time away. There’s probably going to be a systematic revision of the API before it because I enjoy being able to break things without remorse.