I put in a PR with Optim about the ldiv! thing. It’s because Optim.InverseDiagonal only has ldiv! defined for plain Arrays in the other argument slots rather than AbstractArrays.
Also, I don’t think the upper and lower bound have to be ComponentArrays. I was able to run this example without them being.