# Instability in StatsModels.jl term creation

**URL:** <https://discourse.julialang.org/t/instability-in-statsmodels-jl-term-creation/116571>\
**Category:** New to Julia\
**Created:** [July 3, 2024, 8:12am UTC](https://discourse.julialang.org/t/instability-in-statsmodels-jl-term-creation/116571 "2024-07-03T08:12:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [July 3, 2024, 8:12am UTC](https://discourse.julialang.org/t/instability-in-statsmodels-jl-term-creation/116571/1 "2024-07-03T08:12:54Z")

</div>

I am trying to follow closely the documentation of `StatsModels.jl` to get a function term working as syntaxic sugar to define survival response in regression models. The issue is that i get a lot of type instabilities… I managed to get down to the following MWE :

```julia
using StatsModels, StatsAPI, DataFrames

struct SurvTerm{X, Y} <: AbstractTerm
    T::X
    Δ::Y
end
Surv(T::Float64, Δ::Bool) = (T, Δ)
function StatsModels.apply_schema(t::FunctionTerm{typeof(Surv)},
    sch::StatsModels.Schema,
    Mod::Type{<:Any})
    return apply_schema(SurvTerm(t.args...), sch, Mod) 
end
function StatsModels.apply_schema(t::SurvTerm{X,Y},
    sch::StatsModels.Schema,
    Mod::Type{<:Any}) where {X,Y}
    T = apply_schema(t.T, sch, Mod)
    Δ = apply_schema(t.Δ, sch, Mod)
    isa(T, ContinuousTerm) || throw(ArgumentError("Surv only works with continuous terms (got $T)"))
    isa(Δ, ContinuousTerm) || throw(ArgumentError("Surv only works with discrete terms (got $Δ)"))
    return SurvTerm(T, Δ)
end
function StatsModels.modelcols(t::SurvTerm, d::NamedTuple)
    T = modelcols(t.T, d)
    Δ = modelcols(t.Δ, d)
    return hcat(T,Δ)
end

# then test it : 
df = DataFrame(
    time = -log.(rand(1000)), 
    status = rand(1000) .<= 0.8, 
    sex = ifelse.(rand(1000) .<= 0.5, :male, :female)
)

@code_warntype @formula(Surv(time,status)~sex) # already unstable
@code_warntype schema(df) # stable. 
@code_warntype apply_schema(@formula(Surv(time,status)~sex), schema(df)) # very unstable.

```

Can you help me understand what I missed ?

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [July 3, 2024, 4:12pm UTC](https://discourse.julialang.org/t/instability-in-statsmodels-jl-term-creation/116571/2 "2024-07-03T16:12:04Z")

</div>

We discussed this a bit on slack, but wanted to follow up here. Thanks for the MWE!

Type instability isn’t necessarily a problem _per se_, and it’s basically unavoidable in the design that StatsModels currently uses (with the _types_ of the output of `apply_schema` dependent on the _values_ in the schema). I’ll also note that the example is still type-unstable _without_ using `Surv` at all (e.g., `@formula(time~sex)`), so it’s not that you’re doing anything wrong here.

Where type instability can cause an issue is if it prevents the compiler from being able to generate efficient, specialized code for numerical operations. Even when there’s type instability, as long as the performance-critical stuff is behind a function boundary, the types of the arguments can be specialized on.

So, this MWE demonstrates that there’s type instability, which isn’t IMO unexpected and basically impossible to remove from StatsModels _without_ putting a _lot more_ type information into the schema type (e.g,. making it a NamedTuple or something like that and using generated funtions to lookup values) _and_ requiring that all tables are _also_ fully typed (again, using something like a NamedTuple-of-Vectors and using generated functions to look columns up). I haven’t seen this instability causing performance problems in practice, but if it _is_ causing performance issues I’d love to see an MWE of that and we’ll see if there’s anything we can do!