Why is the loss function increasing when fitting a line?

_jovian
Thu, 10 Jul 2025 09:43:07 +0000
New to Julia

Hello everyone,

I’m following the “Fitting a Line” from the Flux.jl guide. It works as expected with the given data, as well as when modifying the predicted function (e.g. actual(x) = 3x - 1).

I was trying to change the length of the training dataset (from 6 to 11 for the *_train variables, and from 5 to 10 for the *_test variables). However, when doing so, the loss function (MSE) is increasing instead of decreasing.

Can you please help me to understand what I’m doing wrong? Here is the script I’m using:

using Flux
using Flux: train!
using Statistics

actual(x) = 4x + 2

x_train, x_test = hcat(0:10...), hcat(11:20...)
y_train, y_test = actual.(x_train), actual.(x_test)

loss(model, x, y) = mean(abs2.(model(x) .- y))
predict = Dense(1 => 1)
data = [(x_train, y_train)]
opt = Descent()

# Checking the value of the loss function
predict.weight, predict.bias
## (Float32[-0.6373584;;], Float32[0.0])
loss(predict, x_train, y_train)
## 849.4254f0

# Training the neuron
train!(loss, predict, data, opt)
# Checking the value of the loss function
predict.weight, predict.bias
## (Float32[33.82415;;], Float32[5.0373588])
loss(predict, x_train, y_train)
## 32046.893f0

# Another round?
train!(loss, predict, data, opt)
predict.weight, predict.bias
## (Float32[-177.98224;;], Float32[-25.394264])
loss(predict, x_train, y_train)
## 1.2097169f6

# One last time
train!(loss, predict, data, opt)
predict.weight, predict.bias
## (Float32[1123.2878;;], Float32[162.06683])
loss(predict, x_train, y_train)
## 4.5665416f7

3 posts - 3 participants

Read full topic