Issue with BNN example from Turing Tutorial

Hi, student and first time poster here.
I’ve been trying to work through the tutorials from the Turing documentation, and can’t seem to get the Bayesian Neural Network to run. More specifically, it seems to stem from the model being passed into the sample function.
I get the same error from both the Generic Bayesian Neural Network example and the more in depth example earlier in the tutorial. Here’s a link for the curious:
https://turing.ml/dev/tutorials/3-bayesnn/

# Import libraries.
using Turing, Flux, Plots, Random
# Specify the network architecture.
network_shape = [
    (3,2, :tanh),
    (2,3, :tanh), 
    (1,2, :σ)]

# Regularization, parameter variance, and total number of
# parameters.
alpha = 0.09
sig = sqrt(1.0 / alpha)
num_params = sum([i * o + i for (i, o, _) in network_shape])

# This modification of the unpack function generates a series of vectors
# given a network shape.
function unpack(θ::AbstractVector, network_shape::AbstractVector)
    index = 1
    weights = []
    biases = []
    for layer in network_shape
        rows, cols, _ = layer
        size = rows * cols
        last_index_w = size + index - 1
        last_index_b = last_index_w + rows
        push!(weights, reshape(θ[index:last_index_w], rows, cols))
        push!(biases, reshape(θ[last_index_w+1:last_index_b], rows))
        index = last_index_b + 1
    end
    return weights, biases
end

# Generate an abstract neural network given a shape, 
# and return a prediction.
function nn_forward(x, θ::AbstractVector, network_shape::AbstractVector)
    weights, biases = unpack(θ, network_shape)
    layers = []
    for i in eachindex(network_shape)
        push!(layers, Dense(weights[i],
            biases[i],
            eval(network_shape[i][3])))
    end
    nn = Chain(layers...)
    return nn(x)
end

# General Turing specification for a BNN model.
@model bayes_nn(xs, ts, network_shape, num_params) = begin
    θ ~ MvNormal(zeros(num_params), sig .* ones(num_params))
    preds = nn_forward(xs, θ, network_shape)
    for i = 1:length(ts)
        ts[i] ~ Bernoulli(preds[i])
    end
end

# Set the backend.
Turing.setadbackend(:reverse_diff)

# Perform inference.
num_samples = 500
ch2 = sample(bayes_nn(hcat(xs...), ts, network_shape, num_params), NUTS(0.65), num_samples);

Here’s the stacktrace, it’s a long one:

ERROR: LoadError: UndefVarError: forward not defined
Stacktrace:
 [1] _forward(::typeof(DistributionsAD.zygote_ldiv), ::LinearAlgebra.Adjoint{Float64,LinearAlgebra.UpperTriangular{Float64,LinearAlgebra.Diagonal{Float64,Array{Float64,1}}}}, ::TrackedArray{…,Array{Float64,1}}) at C:\Users\nickp\.julia\packages\DistributionsAD\7Met8\src\common.jl:107
 [2] #track#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Tracker.track), ::typeof(DistributionsAD.zygote_ldiv), ::LinearAlgebra.Adjoint{Float64,LinearAlgebra.UpperTriangular{Float64,LinearAlgebra.Diagonal{Float64,Array{Float64,1}}}}, ::Vararg{Any,N} where N) at C:\Users\nickp\.julia\packages\Tracker\JhqMQ\src\Tracker.jl:52
 [3] track at C:\Users\nickp\.julia\packages\Tracker\JhqMQ\src\Tracker.jl:52 [inlined]
 [4] zygote_ldiv at C:\Users\nickp\.julia\packages\DistributionsAD\7Met8\src\common.jl:105 [inlined]
 [5] _logpdf(::DistributionsAD.TuringMvNormal{Array{Float64,1},LinearAlgebra.Cholesky{Float64,LinearAlgebra.Diagonal{Float64,Array{Float64,1}}}}, ::TrackedArray{…,SubArray{Float64,1,Array{Float64,1},Tuple{UnitRange{Int64}},true}}) at C:\Users\nickp\.julia\packages\DistributionsAD\7Met8\src\multivariate.jl:49
 [6] _logpdf(::MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}}, ::TrackedArray{…,SubArray{Float64,1,Array{Float64,1},Tuple{UnitRange{Int64}},true}}) at C:\Users\nickp\.julia\packages\DistributionsAD\7Met8\src\multivariate.jl:55
 [7] logpdf at C:\Users\nickp\.julia\packages\DistributionsAD\7Met8\src\multivariate.jl:39 [inlined]
 [8] logpdf_with_trans at C:\Users\nickp\.julia\packages\Bijectors\hFDmJ\src\Bijectors.jl:417 [inlined]
 [9] assume(::Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}, ::MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}}, ::Turing.Core.RandomVariables.VarName{:θ}, ::Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},TrackedArray{…,Array{Float64,1}},Array{Set{Turing.Selector},1}}}},Tracker.TrackedReal{Float64}}) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:451
 [10] macro expansion at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\compiler.jl:101 [inlined]
 [11] macro expansion at .\untitled-f87f5efba838ef24ae6689ad3bf587e2:91 [inlined]
 [12] (::var"##inner_function#483#83")(::Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},TrackedArray{…,Array{Float64,1}},Array{Set{Turing.Selector},1}}}},Tracker.TrackedReal{Float64}}, ::Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}, ::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\compiler.jl:438
 [13] #_#3 at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\Turing.jl:72 [inlined]
 [14] Model at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\Turing.jl:72 [inlined]
 [15] runmodel! at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\RandomVariables.jl:716 [inlined]
 [16] (::Turing.Core.var"#f#32"{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64},Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}},Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}})(::TrackedArray{…,Array{Float64,1}}) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\ad.jl:141
 [17] #20 at C:\Users\nickp\.julia\packages\Tracker\JhqMQ\src\back.jl:148 [inlined]
 [18] forward(::Tracker.var"#20#22"{Turing.Core.var"#f#32"{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64},Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}},Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}}}, ::Tracker.Params) at C:\Users\nickp\.julia\packages\Tracker\JhqMQ\src\back.jl:135
 [19] forward(::Function, ::Array{Float64,1}) at C:\Users\nickp\.julia\packages\Tracker\JhqMQ\src\back.jl:148
 [20] gradient_logp_reverse(::Array{Float64,1}, ::Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}, ::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\ad.jl:145
 [21] gradient_logp at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\core\ad.jl:78 [inlined]
 [22] ∂logπ∂θ at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:403 [inlined]
 [23] ∂H∂θ at C:\Users\nickp\.julia\packages\AdvancedHMC\1ngch\src\hamiltonian.jl:29 [inlined]
 [24] phasepoint at C:\Users\nickp\.julia\packages\AdvancedHMC\1ngch\src\hamiltonian.jl:60 [inlined]
 [25] #find_good_eps#4(::Int64, ::typeof(AdvancedHMC.find_good_eps), ::Random._GLOBAL_RNG, ::AdvancedHMC.Hamiltonian{AdvancedHMC.Adaptation.DiagEuclideanMetric{Float64,Array{Float64,1}},Turing.Inference.var"#logπ#34"{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64},Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}},Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}},Turing.Inference.var"#∂logπ∂θ#33"{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64},Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}},Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}}}, ::Array{Float64,1}) at C:\Users\nickp\.julia\packages\AdvancedHMC\1ngch\src\trajectory.jl:534
 [26] #find_good_eps at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:0 [inlined]
 [27] #find_good_eps#5 at C:\Users\nickp\.julia\packages\AdvancedHMC\1ngch\src\trajectory.jl:593 [inlined]
 [28] find_good_eps at C:\Users\nickp\.julia\packages\AdvancedHMC\1ngch\src\trajectory.jl:593 [inlined]
 [29] #HMCState#39(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type{Turing.Inference.HMCState}, ::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}, ::Random._GLOBAL_RNG) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:548
 [30] Turing.Inference.HMCState(::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::Turing.Sampler{NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric},Turing.Inference.SamplerState{Turing.Core.RandomVariables.VarInfo{NamedTuple{(:θ,),Tuple{Turing.Core.RandomVariables.Metadata{Dict{Turing.Core.RandomVariables.VarName{:θ},Int64},Array{MvNormal{Float64,PDMats.PDiagMat{Float64,Array{Float64,1}},Array{Float64,1}},1},Array{Turing.Core.RandomVariables.VarName{:θ},1},Array{Float64,1},Array{Set{Turing.Selector},1}}}},Float64}}}, ::Random._GLOBAL_RNG) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:529
 [31] Turing.Sampler(::NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric}, ::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::Turing.Selector) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:313
 [32] Sampler at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\hmc.jl:305 [inlined]
 [33] #sample#2(::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(sample), ::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, 
:ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric}, ::Int64) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\Inference.jl:148
 [34] sample(::Turing.Model{Tuple{:θ},Tuple{:xs,:ts,:network_shape,:num_params},var"##inner_function#483#83",NamedTuple{(:xs, :ts, :network_shape, :num_params),Tuple{Array{Float64,2},Array{Float64,1},Array{Tuple{Int64,Int64,Symbol},1},Int64}},NamedTuple{(:xs, :ts, :network_shape, :num_params),NTuple{4,Symbol}}}, ::NUTS{Turing.Core.TrackerAD,(),AdvancedHMC.Adaptation.DiagEuclideanMetric}, ::Int64) at C:\Users\nickp\.julia\packages\Turing\fbY6B\src\inference\Inference.jl:147
 [35] top-level scope at untitled-f87f5efba838ef24ae6689ad3bf587e2:103
in expression starting at untitled-f87f5efba838ef24ae6689ad3bf587e2:103

I’m running Julia v1.3.0 and the latest version of turing.
Thanks!

Try ]add DistributionsAD and if you have it, then ]up DistributionsAD.

2 Likes

Worked! Thanks so much, been sitting with that one for a while now.

No worries, next time try asking earlier :slight_smile: