# Recursive closures/inner functions -- how to avoid boxing?
**URL:** https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758
**Category:** General Usage
**Tags:** question, closure
**Created:** [January 14, 2025, 10:34am UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758 "2025-01-14T10:34:21Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)
#### Post date: [January 14, 2025, 10:34am UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/1 "2025-01-14T10:34:21Z")
Consider code like
```julia
julia> function outer()
function inner()
false && inner()
end
inner
end
julia> typeof(outer()).types
svec(Core.Box)
```
So this kind of code creates a box for the local variable `inner` which is captured by `inner` itself, because it is needed in the method body of `inner()`.
Alas, I see no obvious way of using `let` blocks to avoid the boxing.
So my questions are:
1. is this kind of boxing semantically necessary?
2. is this a recent lowering bug?
3. is there a nice trick to avoid the boxing?
4. is this a missing point in [Performance Tips · The Julia Language](https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-captured) ?
If this was my own personal code, I wouldn’t particularly care and just use an explicit callable struct. But that is inappropriate for shared projects (drive-by-fixes on github are nice, drive-by-refactorings are obnoxious).
An example in the wild is [julia/base/file.jl at 9b1ea1a880e1c47ebdc549a12fca288b5cc60013 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/9b1ea1a880e1c47ebdc549a12fca288b5cc60013/base/file.jl#L1153) where the local variable `_walkdir` is boxed due to the reentrency of the `_walkdir` method body.
Apologies if I failed to find prior discussions on that – I’d also appreciate a pointer.
---
### Author:  [@abraemer](https://discourse.julialang.org/u/abraemer)
#### Post date: [January 14, 2025, 11:33am UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/2 "2025-01-14T11:33:06Z")
> [@foobar\_lv2](#):
>
> 1. is there a nice trick to avoid the boxing?
You could do:
```julia-repl
julia> function outer2()
function inner(_inner)
false && _inner()
end
() -> inner(inner)
end
```
This doesn’t seem to have boxes anywhere but tbh I am not quite sure how to check thoroughly:
```julia-repl
julia> typeof(outer2()).types
svec(var"#inner#3")
julia> typeof(outer2()).types[1]
var"#inner#3"
julia> typeof(outer2()).types[1].types
svec()
```
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [January 14, 2025, 11:45am UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/3 "2025-01-14T11:45:28Z")
@uniment was on quite the crusade against this issue a while ago, and there’s some issues and discussions all over the place here on Discourse and Github about it from that era.
Here’s some breadcrumbs if you want to explore (a subset of) the discussion that happened back then:
- [Performant Recursive Anonymous Functions](https://discourse.julialang.org/t/performant-recursive-anonymous-functions/90984)
- [Locally-Scoped Named Functions have Surprising Behavior that Causes Poor Performance · Issue #47760 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/47760)
- [implement local const · Issue #5148 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/5148)
- [Do not capture recursive local function's self-identifier (to avoid box) · Issue #53295 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/53295)
One hacky trick you can use here is `var"#self#"` to refer to the function itself, i.e.
```julia-repl
julia> function outer()
function inner()
false && var"#self#"()
end
inner
end
outer (generic function with 1 method)
julia> typeof(outer()).types
svec()
julia> @btime outer()()
1.082 ns (0 allocations: 0 bytes)
false
```
This is not a particularly good idea though, and I would not recommend doing it in general. The PR I mention in ["A Tragedy of Julia’s Type System" - #36 by Mason](https://discourse.julialang.org/t/a-tragedy-of-julia-s-type-system/124619/36) should be able to fix this issue as well.
---
### Author:  [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)
#### Post date: [January 14, 2025, 12:33pm UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/4 "2025-01-14T12:33:09Z")
> [@Mason](#):
>
> This is not a particularly good idea though, and I would not recommend doing it in general.
Why?
That seems like a perfectly workable suggestion that should go into the official performance tips and be used everywhere, like e.g. in Base / walkdir?
There’s no reason to be ashamed of compiler limitations, we just need to be honest about them (instead of cosplaying “temporarily embarrassed millionaire”).
If lowering is currently, and for the last 8 years, too dumb to unbox recursive closures that are referred by name, then `var"#self#"` has been the idiomatic way of writing recursive inner functions for the last 8 years, full stop.
Also, big thanks, TIL!
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [January 14, 2025, 12:41pm UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/5 "2025-01-14T12:41:50Z")
> [@foobar\_lv2](#):
>
> Why?
>
> That seems like a perfectly workable suggestion that should go into the official performance tips and be used everywhere, like e.g. in Base / walkdir?
It’s a bad idea because
1. it’s an interal implementation detail that may not be stable
2. it’s a very easy to hit sharp edges and ‘surprising’ behaviour with it.
For instance, consider
```julia
function outer()
function inner()
[var"self"() for i in 1:10]
end
end
```
or
```julia
function outer()
function inner()
t = @spawn var"#self#"()
end
end
```
One could very easily be forgiven for thinking that in both of those cases that the `var"#self#"`'s refer to `inner` but in fact, they refer to inner closures sneakily created by the list comprehension and the `@spawn` macro respectively, in ways that are semi-opaque to the user.
This makes it a massive potential footgun to go around referring to `var"#self#"` unless you deeply understand the exact details of what lexical scope you are calling it from.
---
### Author:  [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)
#### Post date: [January 14, 2025, 1:09pm UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/6 "2025-01-14T13:09:00Z")
Ok, var"#self#" has some sharp edges. And if it’s not officially supported (i.e. not guaranteed to work and have same semantics until 2.0) then it’s not really an option anyways.
Compiler limitations and capability shape the language.
For example, inductive range check elimination completely reshaped when `@inbounds` is appropriate.
Given today’s compiler, what is today’s idiomatic way of expressing `Base.walkdir` (linked above)?
Is it “don’t do recursive inner functions, use an explicit callable struct”?
That is something we should
1. explain in the performance tips
2. apply to `walkdir` in `Base/file.jl` etc (what’s good for the goose is good for the gander)
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [January 14, 2025, 1:40pm UTC](https://discourse.julialang.org/t/recursive-closures-inner-functions-how-to-avoid-boxing/124758/7 "2025-01-14T13:40:28Z")
> [@foobar\_lv2](#):
>
> Given today’s compiler, what is today’s idiomatic way of expressing `Base.walkdir` (linked above)?
>
> Is it “don’t do recursive inner functions, use an explicit callable struct”?
>
> That is something we should
>
> 1. explain in the performance tips
> 2. apply to `walkdir` in `Base/file.jl` etc (what’s good for the goose is good for the gander)
The easiest way is to just pass the function to itself. Instead of the `walkdir` example, I’ll show `fib`:
```julia-repl
julia> let
# Create a local function _fib whose first argument is a function fib
function _fib(fib, n)
if n <= 1
return n
else
# inside _fib, we recurse using fib, *not* _fib
fib(fib, n-1) + fib(fib, n-2)
end
end
# now create a new local function fib that calls _fib(_fib, n)
fib(n) = _fib(_fib, n)
end
(::var"#fib#18"{var"#_fib#17"}) (generic function with 1 method)
julia> @btime $ans(10)
212.570 ns (0 allocations: 0 bytes)
55
```
I showed a macro in [Performant Recursive Anonymous Functions - #17 by Mason](https://discourse.julialang.org/t/performant-recursive-anonymous-functions/90984/17) that automates this process:
```julia
using ExprTools: splitdef, combinedef
using MacroTools: postwalk, @capture
"""
This is only worth using for locally scoped recursive functions.
"""
macro fast_recursion(fdef)
d = splitdef(fdef)
name = get!(d, :name, gensym(:f))
fargs = copy(get!(d, :args, []))
fkwargs = get!(d, :kwargs, [])
_name = gensym(name)
__name = gensym(Symbol(:_, name))
d[:body] = postwalk(d[:body]) do ex
if @capture(ex, f_(args__))
if f == name
return :($__name($__name, $(args...),))
end
elseif @capture(ex, f(args __; kwargs__ ))
if f == name
return :($__name($__name, $(args...); $(kwargs...),))
end
end
ex
end
d[:name] = _name
d[:args] = pushfirst!(d[:args], __name)
quote
$(combinedef(d))
$name($(fargs...); $(fkwargs...),) = $_name($_name, $(fargs...); $(fkwargs...),)
end |> esc
end
```
and then we see
```julia-repl
julia> @btime let
fib(n) = n ≤ 1 ? n : fib(n-1) + fib(n-2)
fib(10)
end
2.620 μs (2 allocations: 32 bytes)
55
julia> @btime let
@fast_recursion fib(n) = n ≤ 1 ? n : fib(n-1) + fib(n-2)
fib(10)
end
212.815 ns (0 allocations: 0 bytes)
55
```
Docs to improve walkdir and the performance tips would be a great idea!
And if someone wants to put `@fast_recursion` into a little package and register it, you’d certainly have my blessing (as with any code code I post here or other julia help channels)