# Using all independent variables with @formula in a multiple linear model

**URL:** <https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691>\
**Category:** New to Julia\
**Tags:** glm\
**Created:** [July 26, 2020, 1:22am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691 "2020-07-26T01:22:04Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![pauloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauloat/32/16517_2.png) [@pauloat](https://discourse.julialang.org/u/pauloat)\
**Post date:** [July 26, 2020, 1:22am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/1 "2020-07-26T01:22:04Z")

</div>

Hi, I’m trying to learn how to do linear models and wanted to know if there something like in R, where you can use a dot (“.”) instead of writing all the indepent variables in a linear regression model, for example in R:

> LinearRegressor = lm(Y ~ ., data = dataset)

in Julia, I’m using GLM like this

> fm = @formula(Y ~ X1 + X2 + … + Xn)  
> LinearRegressor = lm(fm, df)

and I had no problems until the moment, but now I need to do a LM with 28 independent variables and wanted to know if theres is a simple way to do it.

Thanks.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 26, 2020, 1:32pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/2 "2020-07-26T13:32:28Z")

</div>

The dot notation does not exist at the moment in `StatsModels`, so the best bet is to programmatically build up the formula using the `Term` constructor directly:

```julia
lm(Term(:Y) ~ sum(Term.(Symbol.(names(df[:, Not(:Y)])))), df)

```

---

<div class="post-metadata">

**Author:** ![pauloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pauloat/32/16517_2.png) [@pauloat](https://discourse.julialang.org/u/pauloat)\
**Post date:** [July 26, 2020, 9:19pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/3 "2020-07-26T21:19:17Z")

</div>

Edit2: Ok, is working now but I need to use your solution in the in the lm() directly, instead of being able to create a @formula variable. would be nice to be able to do a @formula variable, specially for cross validation and other that need to work with that. But for the time being, thank you again.

Thanks, it was give me an error:

> LoadError: ArgumentError: non-call expression encountered: Term.(Symbol.(names(df[:, Not(:Y])))

~~but I solve it creating a variable with the sum of the independent variables like this~~

> ~~IndependVars = sum(Term.(Symbol.(names(df[:, Not(:Y)]))))~~

~~and using it as~~

> ~~fm = @formula(Y ~ IndependVars)~~

edit: my solution is not working, the lm thinks that IndependVars is a column in my df…

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [July 26, 2020, 9:43pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/4 "2020-07-26T21:43:33Z")

</div>

You are on the right track, but you don’t need the `@formula` call when constructing names programmatically.

```julia
julia> df = DataFrame(y = rand(100), x1 = rand(100), x2 = rand(100));

julia> lm(term(:y) ~ sum(term.(Symbol.(names(df, Not(:y))))), df)
StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

y ~ 1 + x1 + x2

Coefficients:
───────────────────────────────────────────────────────────────────────────────
               Estimate Std. Error t value Pr(>|t|) Lower 95% Upper 95%
───────────────────────────────────────────────────────────────────────────────
(Intercept) 0.64731 0.0719799 8.99292 <1e-13 0.504449 0.79017
x1 0.0392958 0.0932668 0.421327 0.6744 -0.145813 0.224405
x2 -0.259494 0.100676 -2.57752 0.0115 -0.459308 -0.0596804
───────────────────────────────────────────────────────────────────────────────

```

This is definitely not easy syntax. But overall the system is a bit more flexible than R, where using the _rest_ of the names is easy but using a specific list of names for independent variables is very hard.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 27, 2020, 8:01am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/5 "2020-07-27T08:01:35Z")

</div>

To slightly expand on Peter’s answer, what you tried originally was essentially constructing the `Term`s twice, as the `@formula` macro converts its arguments to Terms already. It being a macro, it ultimately has to generate some code that you could have written yourself by hand. To see this:

```julia
julia> using MacroTools

julia> @macroexpand @formula(Y ~ X1 + X2)
:(StatsModels.Term(:Y) ~ StatsModels.Term(:X1) + StatsModels.Term(:X2))

julia> @formula(Y ~ X1 + X2) == (Term(:Y) ~ Term(:X1) + Term(:X2))
true

```

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [December 12, 2022, 3:56pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/6 "2022-12-12T15:56:44Z")

</div>

As I still get likes on my old answer above but can’t edit it, I’ll just add here that I’ve since added a `term` method for strings, so one can do `term.(names(df))` instead of `Term.(Symbol.(names(df)))`.

---

<div class="post-metadata">

**Author:** ![moshi](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@moshi](https://discourse.julialang.org/u/moshi)\
**Post date:** [January 19, 2023, 2:13pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/7 "2023-01-19T14:13:18Z")

</div>

Hi  
Using `term()` is indeed quite useful to programmatically run regressions.

```julia
covariates =["x1", "x2"]
lm(term(:y) ~ sum(term.(covariates)), df)

```

This works perfectly in any linearly separable specification!

However, I maybe missing something in using `term()` when I want to incorporate interactions and fixed effects. In particular, it would be great to specify a vector of covariates which has interactions and fixed effects like:

```julia
#DataFrame df contains columns names y, x1, x2, z1, and z2
covariates = ["x1*x2", "x3", "fe(z1)", "fe(z2)", "fe(z1)&fe(z2)"]  

# run the fixed effects estimation as:
using FixedEffectModels

reg(df, (term(:y) ~ sum(term.(covariates)) ))

```

Even without the fixed effects, including “x1\*x2” or “x1&x2” in `covariates` produces error when used inside of `term()` because it thinks that `:x1*x2` exists as a column in the dataframe.

Would appreciate any help on this! Thanks!

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [January 19, 2023, 2:28pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/8 "2023-01-19T14:28:35Z")

</div>

That’s because `term` does not implement the full formula parser implemented in the `@formula` macro, you’d have to implement that yourself. Unfortunately the output you’re targeting especially with package specific extensions of the base StatsModels syntax is a bit more complicated 😅

```julia
julia> @macroexpand(@formula(y ~ x1*x2 + fe(x3)*fe(x4)))
:(StatsModels.Term(:y) ~ StatsModels.Term(:x1) + StatsModels.Term(:x2) + 
StatsModels.capture_call(fe, ((x3,)->fe(x3)), (:x3,), $(Expr(:copyast, :($(QuoteNode(:(fe(x3))))))), 
[StatsModels.Term(:x3)]) + StatsModels.capture_call(fe, ((x4,)->fe(x4)), (:x4,), $(Expr(:copyast, :($(QuoteNode(:(fe(x4))))))), 
[StatsModels.Term(:x4)]) + StatsModels.Term(:x1) & StatsModels.Term(:x2) + 
StatsModels.capture_call(fe, ((x3,)->fe(x3)), (:x3,), $(Expr(:copyast, :($(QuoteNode(:(fe(x3))))))), [StatsModels.Term(:x3)]) & StatsModels.capture_call(fe, ((x4,)->fe(x4)), (:x4,), $(Expr(:copyast, :($(QuoteNode(:(fe(x4))))))), [StatsModels.Term(:x4)]))

```

If you look at the definition of `term` in the source, it doesn’t quite match that complexity…

```julia
term(s::Symbol) = Term(s)
term(s::AbstractString) = term(Symbol(s))

```

Maybe @dave.f.kleinschmidt has an idea, or views on whether `term` should be able to parse more complicated expressions?

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [January 21, 2023, 1:45am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/9 "2023-01-21T01:45:21Z")

</div>

> views on whether `term` should be able to parse more complicated expressions?

Like @nilshg pointed out, `term` is (by design) extremely simple, and basically just to make it easy to do things like `term.([1, :a, :b, :c])` (there’s another method that creats a `ConstantTerm` from non-string/non-symbol arguments). If you _really, truly, actually_ need parsing, then you’re back in macro land and you should do something like (I’m very sorry) `@eval @formula $(parse(my_formula_string))` but **you probably do not actually want or need to do that** (that should always be the first thing you think when you find yourself reaching for `@eval` 😬 )

I’m not sure about whether `fe` works on `Term`s or not but that’s generally the recommended path (we talked a bit about this in a [JuliaCon talk](https://www.youtube.com/watch?v=h1AZnMUBEec) from last year on [RegressionFormulae.jl](https://github.com/kleinschmidt/RegressionFormulae.jl). That is, we now recommend that if you provide “special syntax” like `fe(x)` for use inside a `@formula`, you should also provide methods for `fe(::Term)` that allows you to do things like `fe(term(:z1))`.

So, _if_ FixedEffects.jl works like this, then you would do something like

```julia
term(:y) ~ term(1) + term(:x1) + term(:x2) + term(:x1) & term(:x2) + term(:x3) + fe(term(:z1)) ... # etc.

```

I’ll add that while we’ve meant for quite a while now to add methods like `Base.:(*)(a::Term, b::Term) = a + b + a&b`, that work has gotten a bit bogged down. If/when [this PR](https://github.com/JuliaStats/StatsModels.jl/pull/183) lands, IIRC it includes a method like that, so you would then be able to write

```julia
term(:y) ~ term(1) + term(:x1) * term(:x2) # ...

```

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [January 21, 2023, 9:19am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/10 "2023-01-21T09:19:20Z")

</div>

Hello! Could you kindly give me advice for one problem:

I want to make models with `@formula` from vector of variables `vars` (`vars` is `Vector{Symbol}`, length of vars can be any):

```julia
models = [@eval @formula(log($i) ~ $(formulation) + $(period) + $(sequence)) for i in vars]

```

And I got (when try to work with GLM or MixedModels):

```julia
ERROR: MethodError: no method matching (::Main.MetidaBioeq.var"#42#43")(::Float64)
The applicable method may be too new: running in world age 55033, while current world is 55034.
Closest candidates are:
  (::Main.MetidaBioeq.var"#42#43")(::Any) at none:0 (method too new to be called from this world context.)
Stacktrace:

```

How to make automatic function term formula correctly?

Without `log` works fine:

```julia
models = [@eval @formula($i ~ $(formulation) + $(period) + $(sequence)) for i in vars]

```

but I don’t want to modyfy DataFrame and make additional variables like `log_var` ets.

Is it possible to solve it?

---

<div class="post-metadata">

**Author:** ![moshi](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@moshi](https://discourse.julialang.org/u/moshi)\
**Post date:** [January 22, 2023, 4:06pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/11 "2023-01-22T16:06:03Z")

</div>

Yes, I think as step 1, allowing for `*` and `&` itself will be very helpful !

As step 2, even if estimations like Fixed effects which require special syntax for them, instead of allowing them within `Term` is something like the following possible where you allow term to take into account the literal special syntax that will go inside @formula

```julia
covariates = ["x1*x2", "x3"] # such that modified term can work through the interactions
#and then something like 
special_syntax = "fe(z1) + fe(z1)&fe(z2)"
reg(df, 
          (term(:y) ~ sum(term.(covariates)) , special_syntax )
)

```

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [January 22, 2023, 6:35pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/12 "2023-01-22T18:35:45Z")

</div>

> [@moshi](#):
>
> Yes, I think as step 1, allowing for `*` and `&` itself will be very helpful !

Just to be clear: `&` works on `Term`s as of right now. It’s just `*` that doesn’t work directly on `Term`s yet! So while it’s slightly inconvenient you can still do something like

```julia
x1, x2, x3 = term.((:x1, :x2, :x3))
covariates = [x1, x2, x1 & x2, x3]
term(:y) ~ sum(covariates)

```

which is equivalent to

```julia
@formula(y ~ x1 * x2 + x3)

```

What is not supported and will almost certainly never be supported is parsing any kind of function calls from strings when creating terms at ‘run time’ (e.g., outside of a macro). So that rules out `term("x1 * x2")`. If you really, truly need to parse strings into expressions and splice those expressions into a formula, then the way to do it is

```julia
covariate_strings = ["x1 * x2", "x3", "fe(z1) + fe(z1) & fe(z2)"]
covariate_expr = Meta.parse(join(covariate_strings, " + "))
formula = @eval @formula y ~ $(covariate_expr)

```

But as I said above, this is discouraged and shouldn’t (in theory) be necessary.

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [January 22, 2023, 6:55pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/13 "2023-01-22T18:55:45Z")

</div>

> [@PharmCat](#):
>
> but I don’t want to modyfy DataFrame and make additional variables like `log_var` ets.
> 
> Is it possible to solve it?

currently no, not without `@eval`. The main goal of [FunctionTerm is dead, long live FunctionTerm by kleinschmidt · Pull Request #183 · JuliaStats/StatsModels.jl · GitHub](https://github.com/JuliaStats/StatsModels.jl/pull/183) is to make eval unnecessary in these cases; with that, you could do

```julia
julia> ff = FunctionTerm(log, [term(:a)], :(log(a)))
(a)->log(a)

julia> d = (; a=1:3)
(a = 1:3,)

julia> modelcols(ff, d)
3-element Vector{Float64}:
 0.0
 0.6931471805599453
 1.0986122886681098

```

This is equivalent to a `FunctionTerm` created inside a formula:

```julia
julia> f = @formula(0 ~ log(a)).rhs
(a)->log(a)

julia> f == ff
true

julia> modelcols(f, d)
3-element Vector{Float64}:
 0.0
 0.6931471805599453
 1.0986122886681098

```

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [January 23, 2023, 11:58am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/14 "2023-01-23T11:58:56Z")

</div>

Thank you for reply!  
Can you tell when this PR will be merged?

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [January 23, 2023, 3:01pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/15 "2023-01-23T15:01:48Z")

</div>

Well, on the one hand, it’s been open for…2 and a half years. On the other hand, I pushed more commits yesterday that addressed nearly all of @nalimilan’s feedback so…I’d estimate anywhere from today to another year 😁

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [January 23, 2023, 3:19pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/16 "2023-01-23T15:19:41Z")

</div>

I’m looking forward to it. 😄 Me (and maybe some other developers) strongly waiting for 0.7 with this PR because it’s blocking factor for some features in dev plans… 🧐

---

<div class="post-metadata">

**Author:** ![moshi](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@moshi](https://discourse.julialang.org/u/moshi)\
**Post date:** [January 28, 2023, 9:49pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/17 "2023-01-28T21:49:16Z")

</div>

Hi

I wanted to ask a very basic question on trying to create term objects but in a for loop.

```julia
julia> vars =["A", "B"]
2-element Vector{String}:
 "A"
 "B"

julia> [i = term.(Symbol("$i")) for i in vars]
2-element Vector{Term}:
 A(unknown)
 B(unknown)

julia> B
ERROR: UndefVarError: B not defined

```

So I am confused that the loop seems to work fine, but when I call B I get the error of it being undefined. However if I do the same but not in a loop it works perfect:

```julia
julia> B = term.(Symbol("B"))
B(unknown)

julia> B
B(unknown)

```

Maybe I am missing something very basic.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 29, 2023, 10:58am UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/18 "2023-01-29T10:58:23Z")

</div>

Yes, this is a misunderstanding about the way Julia works.

Things inside vectors don’t have names and they don’t create variables in the local environment. In the second line, `A(unknown)` and `B(unknown)` are not names variables. They are elements in the vector.

---

<div class="post-metadata">

**Author:** ![moshi](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@moshi](https://discourse.julialang.org/u/moshi)\
**Post date:** [January 29, 2023, 4:56pm UTC](https://discourse.julialang.org/t/using-all-independent-variables-with-formula-in-a-multiple-linear-model/43691/19 "2023-01-29T16:56:04Z")

</div>

I see. Thanks! This solves it:

```julia
julia> covs = [i = term.(Symbol("$i")) for i in vars]
2-element Vector{Term}:
 A(unknown)
 B(unknown)

julia> covs
2-element Vector{Term}:
 A(unknown)
 B(unknown)

```