# Programmatically generate a function akin to Python closure

**URL:** <https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [August 16, 2021, 9:38pm UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514 "2021-08-16T21:38:11Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mys\_721tx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mys_721tx/32/28238_2.png) [@mys\_721tx](https://discourse.julialang.org/u/mys_721tx)\
**Post date:** [August 16, 2021, 9:38pm UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/1 "2021-08-16T21:38:11Z")

</div>

Hello all,

I am implementing a step function. Current I use the following function to generate an expression and returns side expression in a generated function.

```nohighlight
struct TimePoint
    t
    v
end

function step(x, pts::Vector{TimePoint})
    pt_first = popfirst!(pts)
    pt_last = pop!(pts)

    orig_expr = Expr(:if, :($x < $(pt_first.t)), :(return $(pt_first.v)))
    expr = orig_expr
    for pt in pts
        push!(expr.args, Expr(:elseif, :($x < $(pt.t)), :(return $(pt.v))))
        expr = expr.args[end]
    end
    push!(expr.args, :(return $(pt_last.v)))
    return orig_expr
end

```

I would like to have a function/macro that behave akin to Python closure in this way I can I parameterize the generated function for different step sizes and values. How should I proceed?

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [August 16, 2021, 9:56pm UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/2 "2021-08-16T21:56:23Z")

</div>

Is this what you want?

```julia
function makestep(ts, vs)
    # Should probably raise an error here if ts is not sorted.
    t -> let idx = findfirst(t .< ts)
        if idx == nothing
            idx = lastindex(vs)
        end
        vs[idx]
    end
end

```

```julia
using GLMakie
ts = cumsum(rand(10))
vs = rand(10)
step = makestep(ts, vs)
lines(step.(1:0.01:5))

```

 ![step](https://global.discourse-cdn.com/julialang/original/3X/5/e/5ef33cabaf8fb75f1502733dda32031c4394eb31.png)

---

<div class="post-metadata">

**Author:** ![mys\_721tx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mys_721tx/32/28238_2.png) [@mys\_721tx](https://discourse.julialang.org/u/mys_721tx)\
**Post date:** [August 17, 2021, 12:30am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/3 "2021-08-17T00:30:32Z")

</div>

> [@contradict](#):
>
> ```julia
> function makestep(ts, vs)
> # Should probably raise an error here if ts is not sorted.
> t -> let idx = findfirst(t .< ts)
> if idx == nothing
> idx = lastindex(vs)
> end
> vs[idx]
> end
> end
> 
> ```

That looks great! I run a benchmark with `@generated conc` and `conc2 = makestep(...)`. The results is the following:

```julia
julia> @btime conc(10)
  0.023 ns (0 allocations: 0 bytes)
2

julia> @btime conc2(10)
  73.117 ns (2 allocations: 128 bytes)
2

```

It looks like the expression expansion outperforms the function version. Would there be anyway to archive it with `Expr`?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 17, 2021, 6:25am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/4 "2021-08-17T06:25:11Z")

</div>

Side comment: Note that the fields of your `TimePoint` type are of type `Any` (default). This is [a common performance gotcha](https://docs.julialang.org/en/v1/manual/performance-tips/#Avoid-fields-with-abstract-type) which you should avoid. Either specify concrete types or use type parameters:

```julia
struct TimePoint{T, V}
    t::T
    v::V
end

```

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [August 17, 2021, 7:47am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/5 "2021-08-17T07:47:06Z")

</div>

> [@mys\_721tx](#):
>
> ```julia
> julia> @btime conc(10)
> 0.023 ns (0 allocations: 0 bytes)
> 
> ```

If your benchmark tells you that your function runs in less than a nanosecond, then something has gone wrong. Unless you have a 40GHz processor that somehow is able to run your function in a single cycle, or the compiler is really able to fully calculate your function at compile time, this is clearly not right.

You can look into using the `Ref` trick for [BenchmarkTools](https://github.com/JuliaCI/BenchmarkTools.jl):

> Sometimes, interpolating variables into very simple expressions can give the compiler more information than you intended, causing it to “cheat” the benchmark by hoisting the calculation out of the benchmark code  
> `julia> a = 1; b = 2`  
> 2
> 
> `julia> @btime $a + $b`  
> 0.024 ns (0 allocations: 0 bytes)  
> 3  
> As a rule of thumb, if a benchmark reports that it took less than a nanosecond to perform, this hoisting probably occured. You can avoid this by referencing and dereferencing the interpolated variables
> 
> `julia> @btime $(Ref(a))[] + $(Ref(b))[]`  
> 1.277 ns (0 allocations: 0 bytes)  
> 3

---

<div class="post-metadata">

**Author:** ![mys\_721tx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mys_721tx/32/28238_2.png) [@mys\_721tx](https://discourse.julialang.org/u/mys_721tx)\
**Post date:** [August 17, 2021, 7:54am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/6 "2021-08-17T07:54:52Z")

</div>

With increased sample sizes, the time is more reasonable now.

```julia
julia> @benchmark conc(data) setup=(data=rand(UInt8))
BenchmarkTools.Trial: 10000 samples with 1000 evaluations.
 Range (min … max): 1.572 ns … 19.083 ns ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 1.622 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 1.719 ns ± 0.593 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

  █                                                           
  █▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  1.68 ns Histogram: frequency by time 1.6 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

julia> @benchmark conc2(data) setup=(data=rand(UInt8))
BenchmarkTools.Trial: 10000 samples with 965 evaluations.
 Range (min … max): 81.361 ns … 10.622 μs ┊ GC (min … max): 0.00% … 98.22%
 Time (median): 94.457 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 118.672 ns ± 341.463 ns ┊ GC (mean ± σ): 10.27% ± 3.55%

  █                                                              
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁▁ ▁▁▁ ▁
  117 ns Histogram: frequency by time 83.8 ns <

 Memory estimate: 128 bytes, allocs estimate: 2.

```

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [August 17, 2021, 4:01pm UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/7 "2021-08-17T16:01:31Z")

</div>

I’m not much good at performance tuning, but I suspect the problem is those 2 allocations. That might be allocating the result of `t .< ts` (there are some [tools](https://docs.julialang.org/en/v1/manual/profile/#Memory-allocation-analysis) for digging in to this), so maybe switching to a simple loop would be faster, something like

```julia
function makestep(ts, vs)
    function step(t)
        for idx in eachindex(ts)
            if t < ts[idx]
                return vs[idx]
            end
        end
        return vs[end]
    end
end

```

---

<div class="post-metadata">

**Author:** ![mys\_721tx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mys_721tx/32/28238_2.png) [@mys\_721tx](https://discourse.julialang.org/u/mys_721tx)\
**Post date:** [August 18, 2021, 5:16am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/8 "2021-08-18T05:16:11Z")

</div>

On mine (version 1.6.2 on macOS 11) a return statement is needed:

```nohighlight
function makestep(ts::Vector{T}, vs::Vector{V}) where {T, V}
    return function step(t)
        for idx in eachindex(ts)
            if t < ts[idx]
                return vs[idx]
            end
        end
        return vs[end]
    end
end

```

The performance has indeed improved:

```julia
julia> @benchmark conc3(data) setup=(data=rand(UInt8))
BenchmarkTools.Trial: 10000 samples with 997 evaluations.
 Range (min … max): 18.114 ns … 57.385 ns ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 19.534 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 19.806 ns ± 1.956 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

  █                                                            
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  20.6 ns Histogram: frequency by time 18.8 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

```

---

<div class="post-metadata">

**Author:** ![mys\_721tx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mys_721tx/32/28238_2.png) [@mys\_721tx](https://discourse.julialang.org/u/mys_721tx)\
**Post date:** [August 27, 2021, 2:31am UTC](https://discourse.julialang.org/t/programmatically-generate-a-function-akin-to-python-closure/66514/9 "2021-08-27T02:31:50Z")

</div>

I ended up using `ConstantInterpolation` from `DataInterpolations.jl`.

The performance is comparable.

```julia
julia> conc4 = ConstantInterpolation([1, 2, 4, 8], [10, 20, 30, 40], dir=:right)
8-element ConstantInterpolation{Vector{Int64}, Vector{Int64}, Symbol, true, Int64}:
  1
  2
  4
  8
 10
 20
 30
 40

julia> @benchmark conc4(data) setup=(data=rand(UInt8))
BenchmarkTools.Trial: 10000 samples with 997 evaluations.
 Range (min … max): 20.411 ns … 111.894 ns ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 21.641 ns ┊ GC (median): 0.00%
 Time (mean ± σ): 23.268 ns ± 5.258 ns ┊ GC (mean ± σ): 0.00% ± 0.00%

  ▅▄█▇▅▂▃▁ ▁ ▁▂ ▂▁ ▂
  ████████▇▇▇▆▆▆█▇▆▇█▆▆███████▇▆▅▇▇▇▆▇▇▇▆▇▇▇▇▆▇▆▆▇▇▆▄▁▅▅▄▅▅▅▄▅ █
  20.4 ns Histogram: log(frequency) by time 47.2 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

```