# Best way for linear regression problem on product features

**URL:** <https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786>\
**Category:** General Usage\
**Tags:** performance, regression, qr\
**Created:** [April 7, 2025, 7:52am UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786 "2025-04-07T07:52:46Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [April 7, 2025, 5:11pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/22 "2025-04-07T17:11:48Z")

</div>

Oops, I see. I’m sorry for the confusion. Thanks for the effort anyway! Do you think a modification of your method could still work, or does it crucially exploit all of the Kronecker-product structure?

---

<div class="post-metadata">

**Author:** ![mikmoore](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikmoore/32/31109_2.png) [@mikmoore](https://discourse.julialang.org/u/mikmoore)\
**Post date:** [April 7, 2025, 5:50pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/23 "2025-04-07T17:50:07Z")

</div>

To be honest, I still can’t see why  
 \min\_b \| \text{vec}(\text{diag}(Y)) - (X \otimes Z) \text{vec}(b) \|\_2   
(as suggested in the second post) _isn’t_ the problem you’re trying to solve. So I don’t understand the problem and definitely can’t recommend how to solve it. Sorry.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [April 7, 2025, 7:24pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/24 "2025-04-07T19:24:20Z")

</div>

> [@stevengj](#):
>
> Right, your equation is equivalent to

In fact, this was not correct. Minimizing \Vert Y - Z B X^T \Vert (or the equivalent expression with Kronecker products) looks at _all_ the elements of Z B X^T, when in fact you only want to minimize the differences in the _diagonal_ elements.

So, @mikmoore’s solution is actually not applicable here because it is solving the wrong problem (my fault), in addition to being too slow (because \operatorname{diagm}(y) is too big here).

The iterative algorithm ([above](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/5)) as well as the direct algorithm ([above](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/8)) solve the correct problem, though.

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [April 9, 2025, 11:32am UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/25 "2025-04-09T11:32:25Z")

</div>

In my use-case unfortunately, the method does not converge in reasonable time, so I’ll need to use an approximate method that I developed earlier in this use case. I’ll try to think about whether it is possible to develop a more direct method at a later stage.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [April 9, 2025, 12:18pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/26 "2025-04-09T12:18:29Z")

</div>

> [@JADekker](#):
>
> In my use-case unfortunately, the method does not converge in reasonable time

That may mean that your problem is badly conditioned — have you considered using a Tikhonov (ridge regression) regularization? That corresponds to setting a damping parameter \lambda \> 0 in `IterativeSolvers.lsmr`. This should improve the conditioning and convergence rate, and robustify the solutions to noise, at the expense of residual — you’ll have to experiment to find an appropriate regularization strength for your application.

The corresponding “direct”/non-iterative algorithm [(ala the code above)](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/8) is a variant of the normal equations:

```julia
B = reshape(cholesky(Hermitian(Wᵀ * Wᵀ' + λ * I)) \ (Wᵀ * Y), m, n)

```

(which is dominated by the O(N(mn)^2) cost of computing W^T W).

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [April 9, 2025, 12:22pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/27 "2025-04-09T12:22:35Z")

</div>

I originally thought my problem would not be badly conditioned, but that may be a very good direction to look into. Thanks!

---

<div class="post-metadata">

**Author:** ![Mattriks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mattriks/32/351_2.png) [@Mattriks](https://discourse.julialang.org/u/Mattriks)\
**Post date:** [April 10, 2025, 4:46am UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/28 "2025-04-10T04:46:39Z")

</div>

I thought I’d try another method. Inverse regression methods are useful if one is interested in a low-rank predictor subspace:

```julia
using DataFrames, LinearAlgebra, Statistics

function indmat(Y::AbstractVector; nq=4)
    # Slice Y using quantiles
    q = quantile(Y, range(0, 1.0, length=nq+1))
    @show q
    z = [zeros(nq-1); 0.1]
    ind = (Y .>= q[1:end-1]') .& (Y .< (q[2:end] .+ z)')
    return ind*1.0
end

function lda(X, Y)
    # Inverse regression
    # similar to linear discriminant analysis
    μX = mean(X, dims=1)
    Xw = svd(X.-μX).U # whiten X
    Xw ./= std(Xw, dims=1) # make unit variance
    Ya = indmat(Y) # slice Y
    Xhat = Ya*(cov(Ya)\cov(Ya,Xw))
    # Try a truncated or approx SVD here instead?
    F = svd(Xhat)
    DC = Xw*F.V[:, axes(Ya, 2)] # Features or Direction Components
    df = DataFrame(DC.-mean(DC, dims=1), [:DC1, :DC2, :DC3, :DC4])
    df.Y = Y  
    return df
end

m, n, N = 100, 500, 100
X, Z, Y = rand(N, n), rand(N, m), rand(N)
Wᵀ =reshape([conj(X[i,k] * Z[i,ℓ]) for ℓ in 1:m, k in 1:n, i in 1:N], m*n,N)
@show size(Wᵀ);

df = lda(Wᵀ', Y)
dfs = stack(df, Not(:Y)) 
plot(dfs, x=:value, y=:Y, xgroup=:variable, Geom.subplot_grid(
Geom.point, free_x_axis=true), Guide.xlabel(nothing))

```

 ![subspace](https://global.discourse-cdn.com/julialang/original/3X/c/7/c7cb020ae76294d1a70c22b9fe0344d174d484a9.png)

These random numbers have interesting features! If you want a predictive model, then simply build it from the DCs.

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [April 10, 2025, 1:54pm UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/29 "2025-04-10T13:54:36Z")

</div>

Thanks! In my problem unfortunately, I can’t do data reduction (I know that all basis functions may matter, and the problem is in fact one of deterministic least-squares)

---

<div class="post-metadata">

**Author:** ![Mattriks](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mattriks/32/351_2.png) [@Mattriks](https://discourse.julialang.org/u/Mattriks)\
**Post date:** [April 11, 2025, 12:05am UTC](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786/30 "2025-04-11T00:05:49Z")

</div>

It’s simple to transfer from the subspace back to the original space (and would be interesting to see how close the prediction is). It should also be unbiased, whereas the ridge method is biased, but can be iteratively corrected ([https://doi.org/10.1080/13504850500425840](https://doi.org/10.1080/13504850500425840)).

[Previous page](https://discourse.julialang.org/t/best-way-for-linear-regression-problem-on-product-features/127786.md?page=1)