Dropout in mxnet

Specific Domains Machine Learning
1

I’m trying to understand how to use dropout in mxnet and start with a simple regression example:

#=
Regression with dropout
=#
using MXNet

# create training and evaluation data sets
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data"
rawdata = readdlm(download(url))
x = rawdata[:,1:13]'
y = rawdata[:,14:14]'
TrainInput = x[:,1:400]
TrainOutput = y[1:400]
ValidationInput = x[:,401:506]
ValidationOutput = y[401:506]

# how to set up data providers using data in memory
batchsize = 50
trainprovider = mx.ArrayDataProvider(:data => TrainInput, batch_size=batchsize, shuffle=true, :label => TrainOutput)
evalprovider = mx.ArrayDataProvider(:data => ValidationInput, batch_size=batchsize, shuffle=true, :label => ValidationOutput)

# model definition
data = mx.Variable(:data)
label = mx.Variable(:label)
net = @mx.chain     mx.Variable(:data) =>
                    mx.Dropout(p=0.1) =>
                    mx.FullyConnected(num_hidden=1) =>
                    mx.LinearRegressionOutput(label)
model = mx.FeedForward(net, context=mx.cpu())

# set up the optimizer
optimizer = mx.Nadam()

# train model
mx.fit(model, optimizer, initializer=mx.NormalInitializer(0.0,0.001), eval_metric=mx.MSE(), trainprovider, eval_data=evalprovider, n_epoch = 100)

# obtain predictions
test = mx.ArrayDataProvider(:data => ValidationInput, :label => ValidationOutput)
predy = mx.predict(model, test)

My questions are now:

  1. How can I make predictions in each epoch and store in some array?
  2. How can I extract the weights for each epoch?
  3. How can I make mx.FullyConnected(num_hidden=2) to work?