Multiple linear regression model using Flux.jl

Specific Domains Machine Learning
1

Hello everyone,

I’m new to machine learning and I was trying to extend the basic example from Flux documentation to a multiple linear regression model where the input S and the output V are both vectors with 3 elements each, x, y and z.

As long as the relation between S and V is identity, I get great predictions using my model. Simply changing the function between S and V as in the following example so that V_y = 2S_y, leads to wrong results.

What could be the cause of this problem?

## Random Data set
F = 3
D = 1800

S_train, S_test = rand(1:100, F, D), rand(1:100, F, D)

## V = ƒ(S), both V & S are vectors with 3 elements
function ƒ(S::Matrix)
    V = zero(S)
    D = size(S, 1)
    
    D > 1 && for (x, y, z) in Iterators.partition(eachindex(S), D)
        V[x] = S[x]
        V[y] = 2S[y]
        V[z] = S[z]
    end

    isone(D) && (V .= map(actual, S))
    return V
end
V_train, V_test = ƒ(S_train), ƒ(S_test)

# default activation function: identity
using Flux: train!

opt = Descent(1e-7) ## optimizer: classic gradient descent strategy
data = [(V_train, S_train)]
model = Dense(F => F)
parameters = Flux.params(model)
loss(x, y) = Flux.Losses.mse(model(x), y)

for _ in Base.OneTo(100_000)
    train!(loss, parameters, data, opt)
end

model(S_test)
3×1800 Matrix{Float32}:
 79.9905  51.0251  83.9732   1.19865  …   7.04939  99.9106  69.8579  21.1866
 12.4992  36.5154  16.4895  20.1102      41.5276   20.9554  21.4264  43.6019
 80.0067  62.9638  77.0303  81.7296      25.9406   68.1154  14.1989  97.7402

V_test
3×1800 Matrix{Int64}:
 80   51  84   1  40  86   97   77  61  …  82  49  28  75    7  100  70   21
 50  146  66  80  44  42  154  194   4     30   8  62  54  166   84  86  174
 80   63  77  82   3  68   94   35   9     78  77  81  32   26   68  14   98
2

Train calls the loss function as loss(d...) if each datum d is a tuple, i.e., with your definition of data it is called as loss( (V_train, S_train)... ) = loss(V_train, S_train). On the other hand, your loss function is defined as taking input and target as arguments (in that order). Thus, your model learns to map V to S instead of the intended mapping. Flipping either the order of targets and inputs in your data set or in the arguments of the loss function will fix it.

2 Likes