# Flux & Plots

**URL:** <https://discourse.julialang.org/t/flux-plots/30107>\
**Category:** General Usage\
**Created:** [October 20, 2019, 12:19pm UTC](https://discourse.julialang.org/t/flux-plots/30107 "2019-10-20T12:19:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [October 20, 2019, 12:19pm UTC](https://discourse.julialang.org/t/flux-plots/30107/1 "2019-10-20T12:19:00Z")

</div>

### Overview

1. I have good experience with Flux in the past, but now have problems with fitting a 2-input, 1-output data set: the result is rather different from what I get with linear regression.
2. Also, I struggle with doing a surface plot of the resulting Flux FNN model.

I’m using the following packages:

```julia
using Plots; pyplot();
clibrary(:colorcet)
using LaTeXStrings;
using LinearAlgebra;
using Flux;
using Statistics;

```

and have introduced some global variables for consistency of plotting:

```julia
M1 = :v
MS1 = 7
MC1 = :orange
LW1 = 2.5
LS1 = :solid
LA1 = 0.8
# Plotting colors
seed = colorant"blue"
COL = distinguishable_colors(6,seed)

```

### Linear regression result

Data taken from [REGRESSION - Linear Regression Datasets](https://people.sc.fsu.edu/~jburkardt/datasets/regression/regression.html), see [https://people.sc.fsu.edu/~jburkardt/datasets/regression/x09.txt](https://people.sc.fsu.edu/~jburkardt/datasets/regression/x09.txt)

- Input x: (person weight [`kg`], person age `[years]` )
- Output y: person blood fat content `[?]´

```julia
X = [[84.,46],[73,20],[65,52],[70,30],[76,57],[69,25],[63,28],[72,36],
[79,57],[75,44],[27,24],[89,31],[65,52],[57,23],[59,60],[69,48],
[60,34],[79,51],[75,50],[82,34],[59,46],[67,23],[85,37],[55,40],
[63,30]] |> x->reduce(hcat,x)
#
Y = [354.,190,405,263,451,302,288,385,402,365,209,290,346,254,
395,434,220,374,308,220,311,181,274,303,244]'
#
x1 = range(minimum(X[1,:]),maximum(X[1,:]),length=50)
x2 = range(minimum(X[2,:]),maximum(X[2,:]),length=50);
#
#
plot(X[1,:],X[2,:],Y',seriestype=:scatter3d,markercolor=MC1,marker=M1,markersize=MS1,label="")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,10))

```

Result:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/b/dbd9f1b5a0dabefd12189149efe58b7250152cb4.jpeg)

Linear regression of plane:

```julia
ϕ(x) = [1,x[1],x[2]] # basis functions
Φ = [ϕ(X[:,i]) for i in 1:size(X,2)] |> x -> reduce(hcat,x) # regressor
β_lin = Y/Φ; # parameters
L_lin = norm(Y-β_lin*Φ); # loss
f_lin = (x1,x2) -> β_lin*ϕ([x1,x2]) # mapping for surface plot
#
plot(x1,x2,f_lin,seriestype=:surface,color=:blues)
plot!(X[1,:],X[2,:],Y',seriestype=:scatter,markersize=MS1,marker=M1,markercolor=MC1,label="data")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,10))

```

leads to:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/e/8e0084ac2c29689e67ca9ca315eddf19347e1e7c.jpeg)

### Flux and neural network fitting

```julia
data = [(X,Y)]
opt = ADAM(0.05, (0.99, 0.999));
#
#
nz = 5 # number of nodes in hidden layer
mod = Chain(Dense(2,nz,tanh),Dense(nz,1)) # 1 hidden layer, 2 inputs, 1 linear output
# Loss function
loss(x, y) = mean((mod(x).-y).^2)
# Extracting parameters from model
p = Flux.params(mod);
#
#
nE = 20_000 # number of epochs
for i in 1:nE
    Flux.train!(loss,p,data,opt)
end

```

The results (using prompt `julia> ` to differentiate inputs and outputs):

```julia
julia> Tracker.data(mod(X))
1×25 Array{Float32,2}:
 310.72 310.72 310.72 310.72 310.72 … 310.72 310.72 310.72 310.72

```

**Question** : Why on earth does Flux give a totally flat/constant plane? Compare with the sloping plane from linear regression…

### Flux and Plots

I really like the Plots way of doing surface plots where I can specify two input vectors and one scalar output mapping (as compared to, e.g., MATLAB), see the above code for plotting the plane from linear regression.

I’d like to plot the Flux mapping as a surface plot, doing something like:

```julia
plot(x1, x2, f_flux, seriestype=:surface)

```

but struggle with defining `f_flux`… The problem is related to defining `f_flux` with two scalar inputs. If I use a vector input, there is no problem to compute the output:

```julia
julia> f_flux_vector = x -> Tracker.data(mod(x))[:];
julia> f_flux_vector(X)
25-element Array{Float32,1}:
 310.72003
 310.72003
   ...
 310.72003

```

But Plots needs a function with two scalar inputs. However, if I define `f_flux` with two scalar inputs, `plot` doesn’t work for me…

**Question** : how can I define the Flux FNN mapping with two inputs so that I can plot the surface?

**Suggestions are appreciated!**

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [October 22, 2019, 6:53pm UTC](https://discourse.julialang.org/t/flux-plots/30107/2 "2019-10-22T18:53:04Z")

</div>

OK… I figured out how to make surface plots of FNNs form Flux. See the model above. I now add:

```julia
f_nn = (x1,x2) -> Tracker.data(mod([x1;x2]))[1]
#
plot(x1,x2,f_nn,seriestype=:surface,color=:blues)
plot!(X[1,:],X[2,:],Y',seriestype=:scatter,markersize=MS1,marker=M1,markercolor=MC1,label="data")
plot!(xlabel="Weight [kg]", ylabel="Age [years]", zlabel="Blood fat content")
plot!(title="Humans: blood fat vs. weight, age",camera=(-60,20))

```

The result is as follows (for one instance of training…):

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/7/d7a6eba3b7e7dee7e008343fd0969575ef5e243c.jpeg)

In most cases of training, the surface is completely flat. In this instance, there is some minor change at the edge of the surface. Still, why is the fit so poor?

**Could it be that I need (a) more data, or (b) even more epochs?**

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [October 22, 2019, 7:23pm UTC](https://discourse.julialang.org/t/flux-plots/30107/3 "2019-10-22T19:23:38Z")

</div>

Just as a random guess. Could you try normalize your input data?

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [October 22, 2019, 7:33pm UTC](https://discourse.julialang.org/t/flux-plots/30107/4 "2019-10-22T19:33:18Z")

</div>

May be a good idea… I’ll try it tomorrow.

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [October 22, 2019, 8:13pm UTC](https://discourse.julialang.org/t/flux-plots/30107/5 "2019-10-22T20:13:07Z")

</div>

Ah – I was curious, so I tested scaling right away…

 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/c/1c2bf50ec909dcb835fdd83f7c3f97f278561953.jpeg)

Now, it seems to work… I just need to tune some hyper parameters… :-).

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![dmoored4](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmoored4/32/18311_2.png) [@dmoored4](https://discourse.julialang.org/u/dmoored4)\
**Post date:** [September 25, 2020, 3:59pm UTC](https://discourse.julialang.org/t/flux-plots/30107/6 "2020-09-25T15:59:55Z")

</div>

I was having almost exactly the same problem and had searched for so long before finding this. I was about to make a post of my own, but now I’ll test this and if I’ll report back the results. Thanks for the detailed example!