Trying to get started with Flux.jl. I’m trying estimate a linear regression. I have the following:
using Flux
using Random
Random.seed!(1234)
β = [10., 20., 30.] # true parameters
predict(x) = (x*β)
function loss(x, y)
ŷ = predict(x)
sum((ŷ .- y).^2)
end
x = hcat(rand(500,2), ones(500,1)) # create data
y = x*β .+ 10*rand() # population equation
θ = Flux.Params([β])
gs = Flux.gradient( () -> loss(x, y), θ )
opt = Flux.Optimise.Descent()
Flux.train!(loss, θ, [(x, y)], opt)
β |> print
However I do not get back the OLS estimates (x'*x)^(-1)*(x'*y). Could there be an issue with the optimizer? I didn’t see any options for setting tolerance in the documentation.
Note that Flux.train! only trains for one “epoch”. In this case, that means one iteration of gradient descent with the default step size. The training section of the documentation elaborates on how to train your model for many “epochs”.
In general you will have to implement your own convergence criteria, though in this simple case training for a few thousand epochs with the default settings will likely be sufficient.
Yep, that was it. Coming from another discipline, I thought “training” meant estimation. It’s very strange to me to have the training stop after one iteration…
Yeah, it is somewhat confusing nomenclature. In general, it’s something of a general design philosophy of Flux to provide basic tools and assume very little about what you want to do with them. If you look at the source code of Flux.train!, for example, besides the error checking it’s really just a few lines of code computing the loss function gradient and updating the parameters.