How can I solve stack over flow error while using Turing?

Manu_Francis · March 5, 2020, 3:13am

Hi,

I am trying to use Turing package with :reverse_diff as AD background. In my program, the variable size is very high and the program looks similar as follows:

@model test_prob(y, M, N, K, ::Type{T}=Float64) where {T} = begin
     pre_y = zeros(T,K,N)
    out = zeros(T,M)
    #prior belief
    for m in 1:M
        out[m] ~ Uniform(1.0,15.0)
    end
    #function from my custom package updates prey_y variable
    MyPkg.my_func(pre_y,out)    #observations
    for n in 1:N
        for k in 1:K
           y[k,n] ~ truncated(Normal(prey_y[k,n],1.0),0.0,Inf)
        end
   end

end 
N = 256
K = 10
M = 4096
y = rand(Float64,K,N)
Turing.setadbackend(:reverse_diff)
chain = sample(test_prob(y, M, N, K), HMC(0.001, 10), 100)

Here MyPkg.my_func(pre_y,out) is a function in my custom package, on that I am currently working. If I increased N=361 and M= 6859, I am getting stack over flow error as following:

StackOverflowError:
get at abstractdict.jl:596 [inlined]
in at abstractdict.jl:667 [inlined]
haskey at abstractdict.jl:17 [inlined]
#haskey#8 at forward.jl:18 [inlined]
haskey at forward.jl:18 [inlined]
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:121
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#261#264"{Tracker.TrackedReal{Float64},Tracker.TrackedReal{Float64}},Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float64}}}, ::Tracker.TrackedReal{Float64}) at back.jl:113
back(::Tracker.Grads, ::Tracker.Tracked{Float64}, ::Tracker.TrackedReal{Float64}) at back.jl:125
foreach at back.jl:113 [inlined]
back_(::Tracker.Grads, ::Tracker.Call{Tracker.var"#...

How can I solve this error? If I need to increase the stack size, how can I increase?

Thanks in Advance !
Manu

pixel27 · March 5, 2020, 4:09am

I dont’ think Julia sets the stack size, the OS does. If you are on linux you can view the stack size with:

ulimit -s

And change the stack size by specifying a number:

ulimit -s 16384

The default (at least on my machine) is 8192, so 16384 would double my stack size. I don’t know what the equivalent is under windows.

Manu_Francis · March 5, 2020, 4:24am

@pixel27: Thanks for your reply. But I am using windows. So, I am confused