StackOverflowError after only 14 calls to the same function?

General Usage
1

How many times does a function have to be called before a StackOverflowError is triggered? When I try with a simple recursive function call, it seems to be about 80,000. However, in my program, I have a function that is not recursive, but happens to call itself quite a few times - it is doing some post-processing analysis where one variable depends on others that depend on others, etc. but there is no cyclic dependency. The stacktrace below shows the error, but there are no ‘repeated’ entries, and the stack is only 30 calls deep in total. As far as I can see, the deepest function is called from a line that is not already in the stack, so I’m pretty sure this isn’t actually a stack overflow. So what is wrong, and what should I do about it?

Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: LoadError: StackOverflowError:
Stacktrace:
  [1] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4434
  [2] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
  [3] get_variable
    @ ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293 [inlined]
  [4] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4546
  [5] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
  [6] get_variable
    @ ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293 [inlined]
  [7] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4942
  [8] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{…})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
  [9] get_variable
    @ ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293 [inlined]
 [10] _get_all_moment_variables(run_info::@NamedTuple{…}; it::Nothing, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4183
 [11] _get_all_moment_variables
    @ ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4174 [inlined]
 [12] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4635
 [13] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
 [14] get_variable(run_info::@NamedTuple{…}, variable_name::String)
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293
 [15] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4681
 [16] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
 [17] get_variable(run_info::@NamedTuple{…}, variable_name::String)
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293
 [18] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:5141
 [19] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
 [20] get_variable(run_info::@NamedTuple{…}, variable_name::String)
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293
 [21] _get_variable_internal(run_info::@NamedTuple{…}, variable_name::Symbol; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:5738
 [22] get_variable(run_info::@NamedTuple{…}, variable_name::String; normalize_advection_speed_shape::Bool, kwargs::@Kwargs{})
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4295
 [23] get_variable(run_info::@NamedTuple{…}, variable_name::String)
    @ moment_kinetics.load_data ~/physics/moment_kinetics-master-clean2/moment_kinetics/src/load_data.jl:4293
 [24] timestep_diagnostics(run_info::Vector{Any}, run_info_dfns::Vector{Any}; plot_prefix::String, it::Nothing, electron::Bool)
    @ makie_post_processing ~/physics/moment_kinetics-master-clean2/makie_post_processing/makie_post_processing/src/timestep_diagnostics.jl:304
 [25] makie_post_process(run_dir::Vector{String}, new_input_dict::Dict{String, Any}; restart_index::Nothing, plot_prefix::Nothing)
    @ makie_post_processing ~/physics/moment_kinetics-master-clean2/makie_post_processing/makie_post_processing/src/high_level_interface.jl:185
 [26] makie_post_process
    @ ~/physics/moment_kinetics-master-clean2/makie_post_processing/makie_post_processing/src/high_level_interface.jl:76 [inlined]
 [27] makie_post_process(run_dir::String; input_file::String, restart_index::Nothing, plot_prefix::Nothing)
    @ makie_post_processing ~/physics/moment_kinetics-master-clean2/makie_post_processing/makie_post_processing/src/high_level_interface.jl:45
 [28] top-level scope
    @ ~/physics/moment_kinetics-master-clean2/test_scripts/check-makie_post_processing.jl:99
 [29] include(mapexpr::Function, mod::Module, _path::String)
    @ Base ./Base.jl:307
 [30] top-level scope
    @ REPL[1]:1
in expression starting at /home/john/physics/moment_kinetics-master-clean2/test_scripts/check-makie_post_processing.jl:99
Some type information was truncated. Use `show(err)` to see complete types.
2

That’s recursion by definition, maybe you mean indirect recursion?

A call stack isn’t typically limited by a number of calls, but rather a memory segment typically determined by an operating system. Stack frame size varies across methods, so how many method calls exceed the call stack varies as well. Can’t really tell why your stack frames might be bigger without the code.

3 Likes
3

Some general guidelines can be given. “Large” functions, i.e. with many temporary variables, will typically require more stack. The compiler tries hard to use the stack instead of heap allocations. So when there are more temporary values to keep track of than what fits in the cpu-registers, the stack is typically used. In particular, immutable values like StaticArray and other tuples will typically end up on the stack.

1 Like
4

Btw, it’s possible to increase the stack size for a function by running it in a separate Task with larger stack size. The method is described here: Experiments with Julia Stack Sizes and Enzyme

It’s just to use the second argument of Task, which even is documented now:

function with_stacksize(f::F, n) where {F<:Function}
    fetch(schedule(Task(f, n)))
end

with_stacksize(N) do  # N bytes stack
    ...
end