Flux & Plots

Overview

  1. I have good experience with Flux in the past, but now have problems with fitting a 2-input, 1-output data set: the result is rather different from what I get with linear regression.
  2. Also, I struggle with doing a surface plot of the resulting Flux FNN model.

I’m using the following packages:

using Plots; pyplot();
clibrary(:colorcet)
using LaTeXStrings;
using LinearAlgebra;
using Flux;
using Statistics;

and have introduced some global variables for consistency of plotting:

M1 = :v
MS1 = 7
MC1 = :orange
LW1 = 2.5
LS1 = :solid
LA1 = 0.8
# Plotting colors
seed = colorant"blue"
COL = distinguishable_colors(6,seed)

Linear regression result

Data taken from https://people.sc.fsu.edu/~jburkardt/datasets/regression/regression.html, see https://people.sc.fsu.edu/~jburkardt/datasets/regression/x09.txt

  • Input x: (person weight [ kg ], person age [years] )
  • Output y: person blood fat content `[?]´
X = [[84.,46],[73,20],[65,52],[70,30],[76,57],[69,25],[63,28],[72,36],
[79,57],[75,44],[27,24],[89,31],[65,52],[57,23],[59,60],[69,48],
[60,34],[79,51],[75,50],[82,34],[59,46],[67,23],[85,37],[55,40],
[63,30]] |> x->reduce(hcat,x)
#
Y = [354.,190,405,263,451,302,288,385,402,365,209,290,346,254,
395,434,220,374,308,220,311,181,274,303,244]'
#
x1 = range(minimum(X[1,:]),maximum(X[1,:]),length=50)
x2 = range(minimum(X[2,:]),maximum(X[2,:]),length=50);
#
#
plot(X[1,:],X[2,:],Y',seriestype=:scatter3d,markercolor=MC1,marker=M1,markersize=MS1,label="")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,10))

Result:

Linear regression of plane:

ϕ(x) = [1,x[1],x[2]]     # basis functions
Φ = [ϕ(X[:,i]) for i in 1:size(X,2)]  |> x -> reduce(hcat,x)    # regressor
β_lin = Y/Φ;   # parameters
L_lin = norm(Y-β_lin*Φ);     # loss
f_lin = (x1,x2) -> β_lin*ϕ([x1,x2])    # mapping for surface plot
#
plot(x1,x2,f_lin,seriestype=:surface,color=:blues)
plot!(X[1,:],X[2,:],Y',seriestype=:scatter,markersize=MS1,marker=M1,markercolor=MC1,label="data")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,10))

leads to:

Flux and neural network fitting

data = [(X,Y)]
opt = ADAM(0.05, (0.99, 0.999));
#
#
nz = 5    # number of nodes in hidden layer
mod = Chain(Dense(2,nz,tanh),Dense(nz,1))    # 1 hidden layer, 2 inputs, 1 linear output
# Loss function
loss(x, y) = mean((mod(x).-y).^2)
# Extracting parameters from model
p = Flux.params(mod);
#
#
nE = 20_000 # number of epochs
for i in 1:nE
    Flux.train!(loss,p,data,opt)
end

The results (using prompt julia> to differentiate inputs and outputs):

julia> Tracker.data(mod(X))
1×25 Array{Float32,2}:
 310.72  310.72  310.72  310.72  310.72  …  310.72  310.72  310.72  310.72

Question: Why on earth does Flux give a totally flat/constant plane? Compare with the sloping plane from linear regression…

Flux and Plots

I really like the Plots way of doing surface plots where I can specify two input vectors and one scalar output mapping (as compared to, e.g., MATLAB), see the above code for plotting the plane from linear regression.

I’d like to plot the Flux mapping as a surface plot, doing something like:

plot(x1, x2, f_flux, seriestype=:surface)

but struggle with defining f_flux… The problem is related to defining f_flux with two scalar inputs. If I use a vector input, there is no problem to compute the output:

julia> f_flux_vector = x -> Tracker.data(mod(x))[:];
julia> f_flux_vector(X)
25-element Array{Float32,1}:
 310.72003
 310.72003
   ...
 310.72003

But Plots needs a function with two scalar inputs. However, if I define f_flux with two scalar inputs, plot doesn’t work for me…

Question: how can I define the Flux FNN mapping with two inputs so that I can plot the surface?

Suggestions are appreciated!

1 Like

OK… I figured out how to make surface plots of FNNs form Flux. See the model above. I now add:

f_nn = (x1,x2) -> Tracker.data(mod([x1;x2]))[1]
#
plot(x1,x2,f_nn,seriestype=:surface,color=:blues)
plot!(X[1,:],X[2,:],Y',seriestype=:scatter,markersize=MS1,marker=M1,markercolor=MC1,label="data")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,20))

The result is as follows (for one instance of training…):

In most cases of training, the surface is completely flat. In this instance, there is some minor change at the edge of the surface. Still, why is the fit so poor?

Could it be that I need (a) more data, or (b) even more epochs?

Just as a random guess. Could you try normalize your input data?

2 Likes

May be a good idea… I’ll try it tomorrow.

1 Like

Ah – I was curious, so I tested scaling right away…

Now, it seems to work… I just need to tune some hyper parameters… :-).

Thanks a lot!

2 Likes

I was having almost exactly the same problem and had searched for so long before finding this. I was about to make a post of my own, but now I’ll test this and if I’ll report back the results. Thanks for the detailed example!

1 Like