# ANN: Nonconvex.jl a toolbox for AD-based constrained non-convex optimization

**URL:** <https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362>\
**Category:** Package Announcements\
**Tags:** package, optimization\
**Created:** [April 15, 2021, 4:59pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362 "2021-04-15T16:59:13Z")\
**Posts on this page:** 15\
**Page:** 2

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 18, 2021, 2:50pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/21 "2021-04-18T14:50:13Z")

</div>

I don’t think projection will work for MMA without generalising the algorithm in some way. May be possible but it’s not trivial.

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [April 18, 2021, 7:24pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/22 "2021-04-18T19:24:23Z")

</div>

Can we use this package to optimize over parameters which are organized in structs (instead of having all parameters arranged linearly in a vector or an array)?

Given that this uses Zygote, I am thinking it might not be so difficult to make it work with more general parameter structures, just like `Flux.jl` does (via the handy `params` construct).

What are your thoughts on this? From the README it seems at the moment Nonconvex.jl only handles having all parameters in a single array, am I correct?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 19, 2021, 2:36am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/23 "2021-04-19T02:36:08Z")

</div>

For now it’s only vectors. I do plan to have conversion functions between vectors and other data structures. Supporting non-vector data structures natively may be possible for pure Julia solvers but not for the other ones. The conversion function approach is more general and can be used to support sparse matrices, complex numbers, etc.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 21, 2021, 9:41am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/24 "2021-04-21T09:41:15Z")

</div>

Can you please provide an example of how one would one use `ForwardDiff` instead of `Zygote` for a specific objective function with this package?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 21, 2021, 11:12am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/25 "2021-04-21T11:12:24Z")

</div>

I added an example to the README [https://github.com/mohamed82008/Nonconvex.jl#hack-to-use-other-automatic-differentiation-backends](https://github.com/mohamed82008/Nonconvex.jl#hack-to-use-other-automatic-differentiation-backends). @oxinabox may not approve though.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [April 21, 2021, 11:33am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/26 "2021-04-21T11:33:21Z")

</div>

I approve of this, it looks like the right way to do it to me.

If you do it to a function you don’t own it is type-piracy.  
But I mean its probably not the wrong thing to do.  
Since if you want it to be evaluated in forward mode for optimization purposes you porbably in general want it to be evaluated in forward mode even if used as part of a larger system that a reverse-mode AD is doing.  
And now your AD is mixed-mode.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [April 21, 2021, 11:37am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/27 "2021-04-21T11:37:14Z")

</div>

> [@mohamed82008](#):
>
> For now it’s only vectors. I do plan to have conversion functions between vectors and other data structures. Supporting non-vector data structures natively may be possible for pure Julia solvers but not for the other ones. The conversion function approach is more general and can be used to support sparse matrices, complex numbers, etc.

You might be interested in ParameterHandling.jl which has doing this as it’s main purpose via `flatten`

> **[GitHub - JuliaGaussianProcesses/ParameterHandling.jl: Foundational tooling...](https://github.com/JuliaGaussianProcesses/ParameterHandling.jl)**
>
> Foundational tooling for handling collections of parameters in models - GitHub - JuliaGaussianProcesses/ParameterHandling.jl: Foundational tooling for handling collections of parameters in models

It’s @willtebbutt’s work.

Similar code also exists in FiniteDifferences.jl as `to_vec`.  
though we would like to remove it and replace it with ChainRule’s differnetial types which naturally can perform addition with arbitary data-types without having to push it down to a vectors and back-up.  
But you probably can’t do that isn’t IPOPT probably demands a Vector, right?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 21, 2021, 11:38am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/28 "2021-04-21T11:38:15Z")

</div>

> You might be interested in ParameterHandling.jl which has doing this as it’s main purpose via `flatten`

Awesome! I will look into it, thanks.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 21, 2021, 11:43am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/29 "2021-04-21T11:43:58Z")

</div>

> [@oxinabox](#):
>
> But you probably can’t do that isn’t IPOPT probably demands a Vector, right?

Correct.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 21, 2021, 3:17pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/30 "2021-04-21T15:17:04Z")

</div>

Thanks. This is fine as a workaround, but since many packages implement some kind of AD shim that takes a real-valued function and provides a gradient (eg [GalacticOptim](https://galacticoptim.sciml.ai/stable/API/optimization_function/#Defining-Optimization-Functions-Via-AD), [LogDensityProblems](https://tamaspapp.eu/LogDensityProblems.jl/stable/#Automatic-differentiation), many more), it would be great to arrive at a single, lightweight package that does this for all AD frameworks.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 21, 2021, 3:19pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/31 "2021-04-21T15:19:45Z")

</div>

That was the point of [https://github.com/JuliaDiff/AbstractDifferentiation.jl](https://github.com/JuliaDiff/AbstractDifferentiation.jl). I need to finish that PR though.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [April 26, 2021, 11:57am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/32 "2021-04-26T11:57:49Z")

</div>

## Update - Release v0.5.1

* * *

Mixed integer optimization is now available in [Nonconvex.jl](https://github.com/mohamed82008/Nonconvex.jl#mixed-integer-optimization-with-juniper-and-ipopt) using Juniper and Ipopt. To my knowledge, this is the first mixed integer nonlinear programming package in Julia that uses Zygote for automatic differentiation and doesn’t require an explicit JuMP model definition. Happy optimization!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 2, 2021, 1:11pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/33 "2021-05-02T13:11:23Z")

</div>

How would I modify that for the Jacobian (of the constraint)?

```julia
function ChainRulesCore.rrule(::typeof(constraint), x::AbstractVector)
    val = constraint(x)
    jac = ForwardDiff.jacobian(constraint, x)
    val, Δ -> (NO_FIELDS, Δ * jac)
end

```

does not work (`DimensionMismatch`).

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 2, 2021, 1:12pm UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/34 "2021-05-02T13:12:27Z")

</div>

`Δ * jac` → `jac' * Δ`

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [September 5, 2021, 11:30am UTC](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362/35 "2021-09-05T11:30:22Z")

</div>

## Update - Release v1.0

I released [Nonconvex](https://github.com/JuliaNonconvex/Nonconvex.jl) 1.0 and started the JuliaNonconvex organisation. In the 1.0 release, Nonconvex.jl is just a shell package that can be used to load individual solver/wrapper packages using the `Nonconvex.@load` macro. Individual solvers and wrappers live in their own packages in the JuliaNonconvex org. The documentation also lives in the Nonconvex.jl repo. There are plenty of features that I haven’t announced yet on this forum which are available on 1.0:

- Arbitrary decision variable types are now supported. This includes tuples, named tuples, arrays of arrays, dictionaries, structs, etc. These can also be nested together or put in a heterogeneous vector.
- A multi-start version of all the algorithms is now available making using of Hyperopt.jl to re-start the optimisation at different initial solutions using various strategies.
- A JuMP model can now be converted to a Nonconvex model. So you can start from a JuMP model, define all your variables and linear constraints there and then change that model to a Nonconvex model to define the nonlinear functions, make use of Zygote and/or make use of the optimisation algorithms available in Nonconvex.jl.
- An experimental constrained surrogate-assisted optimisation meta-algorithm is now available. It can replace expensive functions by Gaussian process surrogates and solve the surrogate model using any of the other optimisation algorithms in Nonconvex.jl (hence the meta part). It’s experimental because it needs more fine tuning and experimenting. Inequality and equality constraints are supported.
- The augmented Lagrangian algorithm in Nonconvex (wrapped from Percival.jl) now handles arbitrary precision number types correctly.
- Thanks to @noil-reed 's GSoC project, Nonconvex.jl now has nonconvex semidefinite constraint support via an interior point meta-algorithm that can turn any nonlinear programming solver to a nonlinear semidefinite programming solver by iteratively solving the barrier problem. This means that you can constrain any (non-convex) matrix-valued function of the decision variables to be positive semidefinite. This is a recent work and the documentation is underway but you can find an example [here](https://github.com/JuliaNonconvex/NonconvexSemidefinite.jl/blob/master/test/runtests.jl).
- @noil-reed also implemented some zero-order local search algorithms for bound constrained problems that don’t require gradients. See the multi trajectory search algorithm in the documentation for more details.

Happy optimisation and as always feel free to test the package, open issues, start discussions, and/or of course reach out to contribute!

[Previous page](https://discourse.julialang.org/t/ann-nonconvex-jl-a-toolbox-for-ad-based-constrained-non-convex-optimization/59362.md?page=1)