# Function closure with large arrays/matrices scaling horribly
**URL:** https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027
**Category:** Performance
**Tags:** performance, closure
**Created:** [November 5, 2021, 8:38pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027 "2021-11-05T20:38:41Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@vivian-rogers](https://discourse.julialang.org/u/vivian-rogers)
#### Post date: [November 5, 2021, 8:38pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/1 "2021-11-05T20:38:41Z")
Hi!
So I am trying to set up a useful function f to pass around the rest of my code as follows:
```julia
function f_prepostproc(params)
# do stuff
f = f_generator(params)
# evaluate f at certain x values, save to file
return f
end
function f_generator(params)
A = A_gen(params)
B = B_gen(params)
function f(x)
# do an operation on A, B(x), get C (large matrix)
return C #
end
return f
end
function main(params)
f = f_prepostproc(params)
dostuff(f)
end
```
Unfortunately, this takes upwards of an hour to return from this call stack, especially as the size of the matrices A and array B get large (~14,000 x ~14,000 and ~80,000 respectively). After this, the code takes a negligible amount of time to actually evaluate the f(x) as it is passed around in the main function. I am new to function closure in julia, which I understand is invoked here. Is there anything immediately obvious with this setup, or should I rework it to define f(x) under the main scope?
Many thanks
---
### Author:  [@zgornel](https://discourse.julialang.org/u/zgornel)
#### Post date: [November 5, 2021, 8:48pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/2 "2021-11-05T20:48:17Z")
Are you sure it is not a swapping issue?
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [November 5, 2021, 8:52pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/3 "2021-11-05T20:52:59Z")
So OP should probably scale down the problem a bit and post a complete MWE?
---
### Author:  [@vivian-rogers](https://discourse.julialang.org/u/vivian-rogers)
#### Post date: [November 5, 2021, 9:06pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/4 "2021-11-05T21:06:58Z")
I doubt it, as I should have more than enough memory to deal with these sparse matrices.
But that is a good guess, and I think I was running into that before I got an extra 32Gb stick in my machine. I am not sure how to check the swapfile usage in julia, will look into that.
---
### Author:  [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [November 5, 2021, 9:12pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/5 "2021-11-05T21:12:35Z")
> [@vivian-rogers](#):
>
> I am not sure how to check the swapfile usage in julia, will look into that.
Swap file is part of your OS. So maybe you should look into that manual? Without a complete MWE we can only guess about possible memory usage.
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [November 5, 2021, 11:16pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/6 "2021-11-05T23:16:16Z")
> [@vivian-rogers](#):
>
> After this, the code takes a negligible amount of time to actually evaluate the f(x) as it is passed around in the main function. I am new to function closure in julia, which I understand is invoked here. Is there anything immediately obvious with this setup, or should I rework it to define f(x) under the main scope?
Since the inner function captures the outer variables, my guess would be that this capturing requires redefinition (and thus recompilation) of the inner function. This would be consistent with the observed behavior of `f` being fast after the initial call, as the function is already compiled.
Do you have timings/proper benchmarks to compare to? An avenue for investigating which part is actually slow is PPerf.jl, the Profile stdlib or tools like ProfView.jl.
---
### Author:  [@zgornel](https://discourse.julialang.org/u/zgornel)
#### Post date: [November 6, 2021, 3:19pm UTC](https://discourse.julialang.org/t/function-closure-with-large-arrays-matrices-scaling-horribly/71027/7 "2021-11-06T15:19:33Z")
Makes sense. Precompilation with large arrays can be slow. To monitor the swap usage, one can just use `htop` interactively (on \*nix systems).
@vivian-rogers perhaps calling f with smaller matrix sizes (just for precompilation) might speed up things.