User-defined functions with large vector inputs

Specific Domains Optimization (Mathematical)
1

JuMP documentation suggest to use splatting for user-defined functions with vector inputs. For medium size data this works pretty well but for large scale vectors I encounter a memory issue:

julia> f(x...) = sum(x[i] for i in 1:length(x))
julia> y = randn(1_000_000);

julia> f(y...)
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] f(::Float64, ::Vararg{Float64,N} where N) at ./REPL[18]:1
 [2] top-level scope at REPL[20]:1
2

Hi @edljk, unfortunately I don’t see an easy workaround for this. JuMP’s user-defined functions were designed for functions with a moderate number of arguments. We’d need substantial changes to JuMP’s AD code to support vector arguments that are too large for splatting.

1 Like
3

Thanks for your fast answer!

  • no workaround even if the JuMP user-defined function does not use autodiff (gradient function provided)?

  • do you know where does the splatting limitation come from? Size of available memory?

4

Correct. The issue is with JuMP’s representation of the expression graph. All elements in JuMP’s expression graphs are scalars.

Not sure exactly. That’s more a question about the Julia compiler.

5

ok thanks!