Problems using Flux

New to Julia
1

Hi, I’m Juan from Argentina.
I can’t get the following program to work and I don’t understand what’s wrong, could someone help me or give me a hint.
This is the code:

using Flux,Plots, Images,MLDatasets
using Flux: crossentropy, onecold, onehotbatch, train!,params
using LinearAlgebra, Random,Statistics

Random.seed!(1)

X_train_raw, y_train_raw = MLDatasets.MNIST(split=:train)[:]
x_test_raw, y_test_raw = MLDatasets.MNIST(split=:test)[:]

X_train = Flux.flatten(X_train_raw)
x_test = Flux.flatten(x_test_raw)

y_train = onehotbatch(y_train_raw,0:9)
y_test = onehotbatch(y_test_raw,0:9)

data = [X_train,y_train]

###########################################################
model = Chain(Dense(28*28=>32,relu),Dense(32=>10),softmax)
###########################################################

loss(x,y) = crossentropy(model(x),y)
ps = params(model)
learning_rate = 0.01
#opt = Adam(learning_rate)
opt = Flux.setup(Adam(learning_rate), model)
loss_history =
epochs = 500

#for epochs in 1:epochs

Flux.train!(loss,model,data,opt)
train_loss = loss(X_train,y_train)
push!(loss_history,train_loss)
println("Epoch = $epoch : Trainin_loss = $train_loss")

#end

Thanks

2

Could you explain what you think is wrong?

3

Hi, this is the error:

ERROR: MethodError: no method matching (::Dense{typeof(relu), Matrix{Float32}, Vector{Float32}})(::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})

Closest candidates are:
(::Dense)(::AbstractVecOrMat)
@ Flux ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:170
(::Dense)(::AbstractArray)
@ Flux ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:177

Stacktrace:
[1] macro expansion
@ ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:101 [inlined]
[2] _pullback(ctx::Zygote.Context{false}, f::Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, args::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:101
[3] macro expansion
@ ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:53 [inlined]
[4] _pullback
@ ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:53 [inlined]
[5] _pullback(::Zygote.Context{false}, ::typeof(Flux._applychain), ::Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}, ::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:0
[6] _pullback
@ ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:51 [inlined]
[7] _pullback(ctx::Zygote.Context{false}, f::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}}, args::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:0
[8] _pullback
@ ~/Documents/Julia/Julia/Pruebas.jl:25 [inlined]
[9] _pullback(::Zygote.Context{false}, ::typeof(loss), ::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}}, ::Matrix{Float32})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:0
[10] _apply
@ ./boot.jl:838 [inlined]
[11] adjoint
@ ~/.julia/packages/Zygote/HTsWj/src/lib/lib.jl:203 [inlined]
[12] _pullback
@ ~/.julia/packages/ZygoteRules/OgCVT/src/adjoint.jl:66 [inlined]
[13] _pullback
@ ~/.julia/packages/Flux/EHgZm/src/train.jl:107 [inlined]
[14] _pullback(ctx::Zygote.Context{false}, f::Flux.Train.var"#4#5"{typeof(loss), Tuple{Matrix{Float32}}}, args::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface2.jl:0
[15] pullback(f::Function, cx::Zygote.Context{false}, args::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface.jl:44
[16] pullback
@ ~/.julia/packages/Zygote/HTsWj/src/compiler/interface.jl:42 [inlined]
[17] withgradient(f::Function, args::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})
@ Zygote ~/.julia/packages/Zygote/HTsWj/src/compiler/interface.jl:132
[18] macro expansion
@ ~/.julia/packages/Flux/EHgZm/src/train.jl:107 [inlined]
[19] macro expansion
@ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
[20] train!(loss::Function, model::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}}, data::Vector{AbstractMatrix}, opt::NamedTuple{(:layers,), Tuple{Tuple{NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}, NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}, Tuple{}}}}; cb::Nothing)
@ Flux.Train ~/.julia/packages/Flux/EHgZm/src/train.jl:105
[21] train!(loss::Function, model::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}}, data::Vector{AbstractMatrix}, opt::NamedTuple{(:layers,), Tuple{Tuple{NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}, NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}, Tuple{}}}})
@ Flux.Train ~/.julia/packages/Flux/EHgZm/src/train.jl:102
[22] top-level scope
@ ~/Documents/Julia/Julia/Pruebas.jl

4

So the problem is twofold.

First, the loss function should be defined as loss(model, x,y) = crossentropy(model(x),y) and not with an implicit model as loss(x,y) = crossentropy(model(x),y) (the latter assumes differentiation with implicit parameters, which is now abandoned.

The second problem is that data samples should be Vector (or anything iterable with enumerate) of tuples and not of vectors. So whereas this is correct data = [(X_train,y_train)], this is wrong data = [X_train,y_train].

A minor comment is that I am not sure if you want to do a full batch, which is what you do now.

Below find a fixed code, where I have frankly not bothered with minibatching.

using Flux, MLDatasets
using Flux: crossentropy, onecold, onehotbatch, train!,params
using LinearAlgebra, Random, Statistics

Random.seed!(1)

x_train_raw, y_train_raw = MLDatasets.MNIST(split=:train)[:]
x_test_raw, y_test_raw = MLDatasets.MNIST(split=:test)[:]

x_train = Flux.flatten(x_train_raw)
x_test = Flux.flatten(x_test_raw)

y_train = onehotbatch(y_train_raw,0:9)
y_test = onehotbatch(y_test_raw,0:9)


###########################################################
model = Chain(Dense(28*28=>32,relu),Dense(32=>10),softmax)
###########################################################

loss(model, x,y) = crossentropy(model(x),y)
learning_rate = 0.01
opt = Flux.setup(Adam(learning_rate), model)

data = [(x_train[:,1:100],y_train[:,1:100])]
train!(loss, model, data, opt)
1 Like
5

Thanks for your concern, but it keeps giving me the same error:

ERROR: MethodError: no method matching (::Dense{typeof(relu), Matrix{Float32}, Vector{Float32}})(::Chain{Tuple{Dense{typeof(relu), Matrix{Float32}, Vector{Float32}}, Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}, typeof(softmax)}})

Closest candidates are:
(::Dense)(::AbstractVecOrMat)
@ Flux ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:170
(::Dense)(::AbstractArray)
@ Flux ~/.julia/packages/Flux/EHgZm/src/layers/basic.jl:177

6

I think this won’t work, due to a quirk of train! that sometimes it splats elements of data, but not always – the last sentence here:

julia> data = [(x_train[:,1:100], y_train[:,1:100])];

julia> train!(loss, model, data, opt)

help?> train!
search: trainmode! y_train x_train y_train_raw x_train_raw trailing_ones trailing_zeros withgradient

  train!(loss, model, data, opt_state)

  Uses a loss function and training data to improve the model's parameters according to a
  particular optimisation rule encoded in opt_state. Iterates through data once, evaluating for
  each d in data either loss(model, d...) if d isa Tuple, or else loss(model, d) for other d.
1 Like
7

@mcabbott is right, I had a typo there, where I have not fixed the problem of how data should be represented. I have fixed the example and it should be right now. @Jcverni Can you check it again?
For completness, I tested the script with julia 1.9.0 and the package versions are

Status `/private/tmp/Project.toml`
  [587475ba] Flux v0.13.16
  [eb30cadb] MLDatasets v0.7.9
2 Likes
8

Thank you so much. It worked!!

1 Like