StackOverflowError in Bayesian Neural Networks Tutorial

Farlein · March 23, 2020, 2:05am

Hi there,

I am learning Bayesian Neural Networks (BNN) using Turing. I have copied the codes from the tutorial, https://turing.ml/dev/tutorials/3-bayesnn/.

The original code trains a BNN model with a synthetic dataset with 80 rows. The step “ch = sample(bayes_nn(hcat(xs…), ts), HMC(0.05, 4), N);” costs 0:02:03 on my machine. If I change the “N = 80” to N=800, it costs 0:03:25. Pretty fast! However, if I change it N=8000, it gives me the error “StackOverflowError”. I have copied some rows of the detailed error information at the bottom of this post.

I want to build a BNN model to predict Admission Yield, and the dataset has about 40,000 rows and 90 variables, so I need to learn how to train a BNN model with relatively large dataset. Would you please help me to solve the error? Please let me know if I need to provide any other information.

Thanks,
Chuan

mohamed82008 · March 23, 2020, 2:33am

I have seen this error before and it seems to be a Tracker issue with large loops. Zygote doesn’t have this problem. If you go on Turing#master you can use Zygote for AD with:

using Zygote, Turing; Turing.setadbackend(:zygote)

However, Zygote will take a lot of memory when compiling the gradient the first time.

Farlein · March 23, 2020, 1:28pm

Thanks, Mohamed. I have added Turing#master and am testing it with Turing.setadbackend(:zygote). It runs!

However, for 8000 rows, it is estimated to cost 10:39:00, which is much longer than 0:03:25 for N=800.

Thanks,
Chuan

Farlein · March 23, 2020, 1:35pm

Another question I want to ask is about ForwardDiff. If I use Turing.setadbackend(:forward_diff), the program runs fast with 8000 rows for 0:05:40. However, the acceptance rate is constantly 0 for the 5000 samples by HMC, and thus the std is just 0 for nn_params. It is not the case with 80 rows. With 80 rows, I see the std is larger than 0 for each nn_params in the chain ch from “ch = sample(bayes_nn(hcat(xs…), ts), HMC(0.05, 4), N);”.

mohamed82008 · March 24, 2020, 7:49am

With HMC, as you increase the number of data points, you need to lower the step size. Otherwise, just use NUTS.

Farlein · March 24, 2020, 12:35pm

Thanks for your reply, Mohamed. I will lower the step size.