GLM/FixedEffectModels Error: expansion cannot be computed

Shresth-Garg · February 24, 2022, 4:07pm

I recently updated my Julia setup to 1.7.2, along with updating all the packages. I am getting a strange error when running a linear model under both GLM and FixedEffectModels. I get this error only on some specifications. I also tried the same code and data on a different computer (which still has the old version (1.7.1) and packages) and things work fine.

The specifications are below. Test data to replicate the error can be found at [link].

model = reg(df_test, @formula(var1 ~ var2))
model = lm(@formula(var1 ~ var2), df_test)

The error I get is:

ERROR: expansion can't be computed
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] beta_inc_asymptotic_asymmetric(a::Float64, b::Float64, x::Float64, y::Float64, w::Float64, epps::Float64)
    @ SpecialFunctions C:\Users\shres\.julia\packages\SpecialFunctions\rNt4H\src\beta_inc.jl:384
  [3] _beta_inc(a::Float64, b::Float64, x::Float64, y::Float64)
    @ SpecialFunctions C:\Users\shres\.julia\packages\SpecialFunctions\rNt4H\src\beta_inc.jl:865
  [4] _beta_inc
    @ C:\Users\shres\.julia\packages\SpecialFunctions\rNt4H\src\beta_inc.jl:736 [inlined]
  [5] beta_inc
    @ C:\Users\shres\.julia\packages\SpecialFunctions\rNt4H\src\beta_inc.jl:729 [inlined]
  [6] betaccdf
    @ C:\Users\shres\.julia\packages\StatsFuns\dTYga\src\distrs\beta.jl:44 [inlined]
  [7] fdistccdf(ν1::Float64, ν2::Float64, x::Float64)
    @ StatsFuns C:\Users\shres\.julia\packages\StatsFuns\dTYga\src\distrs\fdist.jl:18
  [8] ccdf
    @ C:\Users\shres\.julia\packages\Distributions\T2SAc\src\univariates.jl:624 [inlined]
  [9] _broadcast_getindex_evalf
    @ .\broadcast.jl:670 [inlined]
 [10] _broadcast_getindex
    @ .\broadcast.jl:643 [inlined]
 [11] getindex
    @ .\broadcast.jl:597 [inlined]
 [12] macro expansion
    @ .\broadcast.jl:961 [inlined]
 [13] macro expansion
    @ .\simdloop.jl:77 [inlined]
 [14] copyto!
    @ .\broadcast.jl:960 [inlined]
 [15] copyto!
    @ .\broadcast.jl:913 [inlined]
 [16] copy
    @ .\broadcast.jl:885 [inlined]
 [17] materialize
    @ .\broadcast.jl:860 [inlined]
 [18] coeftable(mm::LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}; level::Float64)
    @ GLM C:\Users\shres\.julia\packages\GLM\gt3bb\src\lm.jl:225
 [19] coeftable
    @ C:\Users\shres\.julia\packages\GLM\gt3bb\src\lm.jl:222 [inlined]
 [20] coeftable(model::StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), 
Tuple{}}})
    @ StatsModels C:\Users\shres\.julia\packages\StatsModels\m1jYD\src\statsmodel.jl:188
 [21] coeftable
    @ C:\Users\shres\.julia\packages\StatsModels\m1jYD\src\statsmodel.jl:188 [inlined]
 [22] show(io::IOContext{Base.TTY}, model::StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}})
    @ StatsModels C:\Users\shres\.julia\packages\StatsModels\m1jYD\src\statsmodel.jl:199
 [23] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, x::StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}})
    @ Base.Multimedia .\multimedia.jl:47
 [24] (::REPL.var"#43#44"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:266
 [25] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:510
 [26] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:259
 [27] display(d::REPL.REPLDisplay, x::Any)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:271
 [28] display(x::Any)
    @ Base.Multimedia .\multimedia.jl:328
 [29] #invokelatest#2
    @ .\essentials.jl:716 [inlined]
 [30] invokelatest
    @ .\essentials.jl:714 [inlined]
 [31] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:293
 [32] (::REPL.var"#45#46"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:277
 [33] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:510
 [34] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:275
 [35] (::REPL.var"#do_respond#66"{Bool, Bool, REPL.var"#77#87"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:846
 [36] #invokelatest#2
    @ .\essentials.jl:716 [inlined]
 [37] invokelatest
    @ .\essentials.jl:714 [inlined]
 [38] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2493
 [39] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL C:\Users\shres\AppData\Local\Programs\Julia-1.7.2\share\julia\stdlib\v1.7\REPL\src\REPL.jl:1232
 [40] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL .\task.jl:423

Any suggestions would be appreciated!

pdeffebach · February 24, 2022, 4:33pm

I can’t replicate your error. Both regressions work fine for me with 1.7.2 and a fresh environment.

Can you share with us the output of ] st?

Shresth-Garg · February 24, 2022, 4:36pm

Thanks for looking into it!

Here’s what I get (note that it’s sparse because I tried deleting everything and starting from scratch):

(@v1.7) pkg> st
      Status `C:\Users\shres\.julia\environments\v1.7\Project.toml`
  [336ed68f] CSV v0.10.2
  [a93c6f00] DataFrames v1.3.2
  [9d5cd8c9] FixedEffectModels v1.6.4
  [38e38edf] GLM v1.6.1

pdeffebach · February 24, 2022, 4:40pm

My hunch is that this has something to do with CSV.jl. Can you try doing something like

test_df = identity.(test_df)

and see if that improves things?

I still can’t replicate with CSV.jl but maybe your csv file is different.

julia> using DataFrames, CSV, FixedEffectModels, GLM;

julia> df = DataFrame(var1 = rand(100), var2 = rand(100));

julia> CSV.write("test_df.csv", df);

julia> test_df = CSV.read("test_df.csv", DataFrame);

julia> model = lm(@formula(var1 ~ var2), df_test)
StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}}

var1 ~ 1 + var2

Coefficients:
────────────────────────────────────────────────────────────────────────
                 Coef.  Std. Error     t  Pr(>|t|)  Lower 95%  Upper 95%
────────────────────────────────────────────────────────────────────────
(Intercept)  0.395065    0.0658136  6.00    <1e-07   0.26446    0.52567
var2         0.0682516   0.112477   0.61    0.5454  -0.154956   0.291459
────────────────────────────────────────────────────────────────────────

julia> model = reg(df_test, @formula(var1 ~ var2))
                             Linear Model                             
=======================================================================
Number of obs:                 100   Degrees of freedom:              2
R2:                          0.004   R2 Adjusted:                -0.006
F-Stat:                   0.368209   p-value:                     0.545
=======================================================================
var1        |  Estimate Std.Error  t value Pr(>|t|) Lower 95% Upper 95%
-----------------------------------------------------------------------
var2        | 0.0682516  0.112477 0.606803    0.545 -0.154956  0.291459
(Intercept) |  0.395065 0.0658136  6.00279    0.000   0.26446   0.52567
=======================================================================

Shresth-Garg · February 24, 2022, 4:52pm

Doing the identity.(test_df) does not help - I still get the same error.

I can successfully run the code snippet in your reply. Even in my main data set, a lot of the specifications work, however some fail. I haven’t been able to find a pattern under which things fail, so can’t come up with an example which does not involve reading in a file from CSV.

pdeffebach · February 24, 2022, 4:54pm

Try doing

typeof.(eachcol(test_df))

and see what it looks like. Maybe some of your columns have weird types without realizing.

Shresth-Garg · February 24, 2022, 5:10pm

Both of the cols are vectors of Float64:

typeof.(eachcol(df))
2-element Vector{DataType}:
 Vector{Float64} (alias for Array{Float64, 1})
 Vector{Float64} (alias for Array{Float64, 1})

I also tried reinstalling 1.6.5 (LTS) from scratch but somehow get the same error.

Sorry I failed to mention this, but I was using the master branch of FixedEffectModels (by running ] add FixedEffectModels#master) when I first got the error. There was a PR fix that I wanted to use, and I decided to update my setup before pulling in that version. I went back to the published version in the fresh reinstall but the error persists.

pdeffebach · February 24, 2022, 5:36pm

Very weird.

Please file an issue with StatsModels.jl and with CSV.jl to see if either of those are the problem.

Shresth-Garg · February 24, 2022, 5:36pm

Thanks - will do!

Shresth-Garg · February 24, 2022, 7:47pm

If others are facing similar issue, I was able to fix it by downgrading the StatsFuns package from 0.9.16 (latest) to 0.9.15. I will file an issue there.

Edit: Nevermind, there’s already a thread there.

matthieu · February 25, 2022, 4:40pm

Should be fixed now that SpecialFunctions v2.1.4 is tagged

a.frist · March 2, 2022, 1:46am

How did you actually downgrade the package? I’m having the same issue but I cannot pin StatsFun because it’s a second order dependency.

a.frist · March 2, 2022, 1:53pm

I was getting the same issue. Here is was my Julia environment:

(@v1.5) pkg> st
Status `~/.julia/environments/v1.5/Project.toml`
  [fbe9abb3] AWS v1.74.0
  [1c724243] AWSS3 v0.9.5
  [c7e460c6] ArgParse v1.1.4
  [6e4b80f9] BenchmarkTools v1.3.1
  [336ed68f] CSV v0.10.2
  [a93c6f00] DataFrames v1.3.2
  [1313f7d8] DataFramesMeta v0.10.0
  [38e38edf] GLM v1.6.1 ⚲
  [9b87118b] PackageCompiler v1.4.1
  [626c502c] Parquet v0.8.3
  [49802e3a] ProgressBars v1.4.0
  [438e738f] PyCall v1.93.0
  [1277b4bf] ShiftedArrays v1.0.0

Simply running “update” in the Julia package manner seems to have fixed the issue. I noticed that SpecialFunctions v2.1.4 was picked up, which lends credence to what @matthieu pointed out about v2.1.4 being the solution.