# Cannot get HSL to work with AmplNLWriter

**URL:** <https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903>\
**Category:** Optimization (Mathematical)\
**Created:** [November 21, 2024, 4:24pm UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903 "2024-11-21T16:24:57Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![pulsipher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pulsipher/32/10461_2.png) [@pulsipher](https://discourse.julialang.org/u/pulsipher)\
**Post date:** [November 21, 2024, 4:24pm UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/1 "2024-11-21T16:24:57Z")

</div>

I have installed HSL\_jll.jl as provided by HSL and also installed HSL.jl. With this I can get Ipopt to work with ma27 when I use Ipopt directly:

```julia
using JuMP, Ipopt, HSL, HSL_jll
model = Model(Ipopt.Optimizer)
@variable(model, x[1:2]>=0) 
@objective(model, Min, sum(x.^2))
@constraint(model, sin(x[1]) <= -0.5)
set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)
set_optimizer_attribute(model, "linear_solver", "ma27")
optimize!(model)

```

However, when I try to use AmplNLWriter, it fails:

```julia
using JuMP, Ipopt_jll, HSL, HSL_jll, AmplNLWriter
model = Model(() -> AmplNLWriter.Optimizer(Ipopt_jll.amplexe))
@variable(model, x[1:2]>=0) 
@objective(model, Min, sum(x.^2))
@constraint(model, sin(x[1]) <= -0.5)
set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)
set_optimizer_attribute(model, "linear_solver", "ma27")
optimize!(model)

```

```julia
Ipopt 3.14.16: hsllib=/home/anaconda/Notebooks/Joshua/HSL_jll.jl-2023.11.7/override/lib/x86_64-linux-gnu-libgfortran5/libhsl.so
linear_solver=ma27

Exception of type: DYNAMIC_LIBRARY_FAILURE in file "Common/IpLibraryLoader.cpp" at line 86:
 Exception message: libmpifort.so.12: cannot open shared object file: No such file or directory

EXIT: Library loading failure.

```

I’ve tried a variety of things, but cannot get this to work. I need this for benchmarking study I am doing where I want to include AMPL-based AD.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [November 21, 2024, 7:08pm UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/2 "2024-11-21T19:08:22Z")

</div>

Hmm this might be a question for @amontoison.

What version of everything?

---

<div class="post-metadata">

**Author:** ![amontoison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amontoison/32/218741_2.png) [@amontoison](https://discourse.julialang.org/u/amontoison)\
**Post date:** [November 21, 2024, 7:50pm UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/3 "2024-11-21T19:50:46Z")

</div>

I am able to reproduce the issue and it’s working if you remove this line:

```julia
set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)

```

Ipopt is looking for a shared library `libhsl` by default if you set the HSL linear solvers, so if the library is dynamically loaded (which is done by `import HSL_jll`), Ipopt could find it automatically.

When I wrote the documentation for `HSL_jll.jl`, I wasn’t sure if it would work on all platforms without it, so as a precaution, I ensured that the path is provided.

I don’t know why it’s not working if we provide the path. I suspect that Ipopt uses other environment variables to load shared libraries. We can see with this exemple that my Ubuntu can’t detect all dependencies of `HSL` outside Julia because they are not in `LD_LIBRARY_PATH`.

```julia
julia> using MPICH_jll

shell> ldd /home/alexis/Applications/libHSL/lbt_HSL_jll.jl-2023.11.7/HSL_jll.jl-2023.11.7/override/lib/x86_64-linux-gnu-libgfortran5/libhsl.so
	linux-vdso.so.1 (0x00007fff99097000)
	libblastrampoline.so.5 => not found
	libmetis.so => /lib/x86_64-linux-gnu/libmetis.so (0x00007ef56f58a000)
	libmpifort.so.12 => not found
	libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007ef56e000000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ef56f4a1000)
	libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ef56f457000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ef56f437000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ef56dc00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ef56f60b000)
	libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ef56e3b8000)

```

---

<div class="post-metadata">

**Author:** ![pulsipher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pulsipher/32/10461_2.png) [@pulsipher](https://discourse.julialang.org/u/pulsipher)\
**Post date:** [November 22, 2024, 1:37am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/4 "2024-11-22T01:37:21Z")

</div>

When I remove

```julia
set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)

```

I get:

```julia
Ipopt 3.14.16: linear_solver=ma27

Exception of type: DYNAMIC_LIBRARY_FAILURE in file "Common/IpLibraryLoader.cpp" at line 86:
 Exception message: libhsl.so: cannot open shared object file: No such file or directory

EXIT: Library loading failure.

```

Any thoughts on how to get this to recognize the right path?

My package environment is as follows on an Ubuntu machine:

```julia
(@v1.11) pkg> st
Status `~/.julia/environments/v1.11/Project.toml`
  [7c4d4715] AmplNLWriter v1.2.2
  [052768ef] CUDA v5.5.2
  [45b445bb] CUDSS v0.3.2
  [1037b233] ExaModels v0.8.0
  [34c5aeac] HSL v0.4.3
  [ffdf776e] InfiniteExaModels v0.1.0 `https://github.com/infiniteopt/InfiniteExaModels.jl#main`
  [20393b10] InfiniteOpt v0.5.9 `https://github.com/infiniteopt/InfiniteOpt.jl#master`
  [b6b21f68] Ipopt v1.6.7
  [4076af6c] JuMP v1.23.4
  [2621e9c9] MadNLP v0.8.4
  [d72a61cc] MadNLPGPU v0.7.3
  [309f4015] MathOptSymbolicAD v0.2.1
  [f4238b75] NLPModelsIpopt v0.10.2
  [c36e90e8] PowerModels v0.21.3
  [017b0a0e] HSL_jll v2023.11.7+0 `../../../Notebooks/Joshua/HSL_jll.jl-2023.11.7`
  [9cc047cb] Ipopt_jll v300.1400.1600+0

```

---

<div class="post-metadata">

**Author:** ![amontoison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amontoison/32/218741_2.png) [@amontoison](https://discourse.julialang.org/u/amontoison)\
**Post date:** [November 22, 2024, 1:58am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/5 "2024-11-22T01:58:18Z")

</div>

I have an idea on how to fix it.  
The paths of all libraries visible to Julia can be recovered with:

```julia
using Libdl
Libdl.dllist()

```

We probably need to add an environment variable, just like we did with `LBT` in the past, so that `ampl_exe` can detect all dependencies.  
We need to update `AmplNLWriter.jl` with @odow for this.

```julia
using Libdl

libs = Libdl.dllist()
dirs = unique(dirname(path) for path in libs)
separator = Sys.iswindows() ? ";" : ":"
library_paths = join(dirs, separator)

if Sys.iswindows()
  println("PATH=" * library_paths)
elseif Sys.isapple()
  println("DYLD_LIBRARY_PATH=" * library_paths)
else
  println("LD_LIBRARY_PATH=" * library_paths)
end

```

---

<div class="post-metadata">

**Author:** ![amontoison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amontoison/32/218741_2.png) [@amontoison](https://discourse.julialang.org/u/amontoison)\
**Post date:** [November 22, 2024, 2:16am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/6 "2024-11-22T02:16:53Z")

</div>

I tested by setting the environment variable before starting Julia.  
It should be the same result if we create it automatically for the user in `AmplNLWriter.jl` before that we execute `ampl_exe`:

```shell
alexis@HP-Spectre:~/Bureau/git$ LD_LIBRARY_PATH=:/lib/x86_64-linux-gnu:/home/alexis/julia/julia-1.11.1/bin/../lib:/lib64:/home/alexis/julia/julia-1.11.1/bin/../lib/julia:/home/alexis/julia/julia-1.11.1/lib/julia:/home/alexis/Applications/OpenBLAS/lib:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Base64:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Markdown:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/InteractiveUtils:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Serialization:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Logging:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Test:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/SuiteSparse_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/SparseArrays:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/StyledStrings:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Unicode:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/REPL:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Printf:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Dates:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/TOML:/home/alexis/.julia/compiled/v1.11/Preferences:/home/alexis/.julia/compiled/v1.11/JLLWrappers:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/CompilerSupportLibraries_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/NetworkOptions:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/MbedTLS_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LibSSH2_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LibGit2_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LibGit2:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/ArgTools:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/nghttp2_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LibCURL_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/MozillaCACerts_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LibCURL:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Downloads:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Tar:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/p7zip_jll:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/UUIDs:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/Pkg:/home/alexis/julia/julia-1.11.1/share/julia/compiled/v1.11/LazyArtifacts:/home/alexis/.julia/compiled/v1.11/MPIPreferences:/home/alexis/.julia/compiled/v1.11/Hwloc_jll:/home/alexis/.julia/artifacts/a610bcf72542cca9e649774fb393a74dc28fd353/lib:/home/alexis/.julia/compiled/v1.11/MPICH_jll:/home/alexis/.julia/artifacts/e85c0a68e07fee0ee7b19c2abc210b1af2f4771a/lib:/home/alexis/.julia/compiled/v1.11/METIS_jll:/home/alexis/.julia/artifacts/e5244f73466343373a87ed1345efac5e3440fa96/lib:/home/alexis/.julia/artifacts/e5244f73466343373a87ed1345efac5e3440fa96/lib/metis/metis_Int32_Real64/lib:/home/alexis/.julia/artifacts/e5244f73466343373a87ed1345efac5e3440fa96/lib/metis/metis_Int64_Real32/lib:/home/alexis/.julia/artifacts/e5244f73466343373a87ed1345efac5e3440fa96/lib/metis/metis_Int64_Real64/lib:/home/alexis/.julia/compiled/v1.11/HSL_jll:/home/alexis/Applications/libHSL/lbt_HSL_jll.jl-2023.11.7/HSL_jll.jl-2023.11.7/override/lib/x86_64-linux-gnu-libgfortran5

alexis@HP-Spectre:~/Bureau/git$ julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.11.1 (2024-10-16)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using JuMP, Ipopt_jll, HSL, HSL_jll, AmplNLWriter

julia> model = Model(() -> AmplNLWriter.Optimizer(Ipopt_jll.amplexe))
A JuMP Model
├ solver: AmplNLWriter
├ objective_sense: FEASIBILITY_SENSE
├ num_variables: 0
├ num_constraints: 0
└ Names registered in the model: none

julia> @variable(model, x[1:2]>=0)
2-element Vector{VariableRef}:
 x[1]
 x[2]

julia> @objective(model, Min, sum(x.^2))
x[1]² + x[2]²

julia> @constraint(model, sin(x[1]) <= -0.5)
sin(x[1]) - -0.5 ≤ 0

julia> set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)

julia> set_optimizer_attribute(model, "linear_solver", "ma27")

julia> optimize!(model)
Ipopt 3.14.16: hsllib=/home/alexis/Applications/libHSL/lbt_HSL_jll.jl-2023.11.7/HSL_jll.jl-2023.11.7/override/lib/x86_64-linux-gnu-libgfortran5/libhsl.so
linear_solver=ma27

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.16, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...: 0
Number of nonzeros in inequality constraint Jacobian.: 1
Number of nonzeros in Lagrangian Hessian.............: 2

Total number of variables............................: 2
                     variables with only lower bounds: 2
                variables with lower and upper bounds: 0
                     variables with only upper bounds: 0
Total number of equality constraints.................: 0
Total number of inequality constraints...............: 1
        inequality constraints with only lower bounds: 0
   inequality constraints with lower and upper bounds: 0
        inequality constraints with only upper bounds: 1

iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
   0 1.9999960e-04 5.10e-01 9.80e-01 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0
   1 1.1526117e-04 5.00e-01 2.62e+01 -1.7 2.62e-01 - 1.00e+00 3.77e-02h 1
   2 1.1923173e-04 5.00e-01 3.62e+04 -1.7 3.73e-01 - 1.00e+00 7.60e-04h 1
   3 1.1924119e-04 5.00e-01 3.05e+09 -1.7 2.59e-01 - 1.00e+00 1.18e-05h 1
   4r 1.1924119e-04 5.00e-01 1.00e+03 -0.3 0.00e+00 - 0.00e+00 5.89e-08R 2
   5r 2.3520927e+01 5.00e-01 1.30e-04 -0.3 4.84e+00 - 1.00e+00 1.00e+00f 1
   6r 5.9645822e+00 5.00e-01 9.50e-01 -1.0 2.41e+00 - 1.00e+00 1.00e+00f 1
   7r 2.8785403e+00 5.00e-01 2.95e-01 -1.7 7.46e-01 - 1.00e+00 1.00e+00f 1
   8r 1.3854380e+00 5.00e-01 1.03e-01 -2.5 5.20e-01 - 1.00e+00 1.00e+00f 1
   9r 7.3420281e-01 5.00e-01 3.46e-02 -3.8 3.20e-01 - 1.00e+00 1.00e+00f 1
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
  10r 4.8913422e-01 5.00e-01 1.50e-09 -3.8 1.57e-01 - 1.00e+00 1.00e+00f 1
  11r 1.2429185e-01 5.00e-01 3.73e-03 -5.7 3.47e-01 - 1.00e+00 1.00e+00f 1
  12r 1.0106452e-01 5.00e-01 1.84e-11 -5.7 3.46e-02 - 1.00e+00 1.00e+00f 1
  13r 2.6157831e-02 5.00e-01 1.97e-04 -8.6 1.56e-01 - 1.00e+00 1.00e+00f 1
  14r 2.4339605e-02 5.00e-01 3.88e-14 -8.6 5.72e-03 - 1.00e+00 1.00e+00f 1
  15r 6.5611573e-03 5.00e-01 3.88e-14 -8.6 7.50e-02 - 1.00e+00 1.00e+00f 1

Number of Iterations....: 15

                                   (scaled) (unscaled)
Objective...............: 1.9148563642281853e-03 1.9148563642281853e-03
Dual infeasibility......: 8.7516499683580481e-02 8.7516499683580481e-02
Constraint violation....: 4.9999998000250589e-01 4.9999998000250589e-01
Variable bound violation: 9.9974940954384683e-09 9.9974940954384683e-09
Complementarity.........: 7.1935703593449981e-08 7.1935703593449981e-08
Overall NLP error.......: 4.9999998000250589e-01 4.9999998000250589e-01

Number of objective function evaluations = 19
Number of objective gradient evaluations = 6
Number of equality constraint evaluations = 0
Number of inequality constraint evaluations = 19
Number of equality constraint Jacobian evaluations = 0
Number of inequality constraint Jacobian evaluations = 18
Number of Lagrangian Hessian evaluations = 16
Total seconds in IPOPT = 0.010

EXIT: Converged to a point of local infeasibility. Problem may be infeasible.

```

---

<div class="post-metadata">

**Author:** ![pulsipher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pulsipher/32/10461_2.png) [@pulsipher](https://discourse.julialang.org/u/pulsipher)\
**Post date:** [November 22, 2024, 3:08am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/7 "2024-11-22T03:08:44Z")

</div>

Thanks for all you help thus far!

> [@amontoison](#):
>
> ```julia-auto
> using Libdl
> 
> libs = Libdl.dllist()
> dirs = unique(dirname(path) for path in libs)
> separator = Sys.iswindows() ? ";" : ":"
> library_paths = join(dirs, separator)
> 
> if Sys.iswindows()
> println("PATH=" * library_paths)
> elseif Sys.isapple()
> println("DYLD_LIBRARY_PATH=" * library_paths)
> else
> println("LD_LIBRARY_PATH=" * library_paths)
> end
> 
> ```

I tried using this, but my issue persists. I could try setting the environment variable before starting Julia like you did, but I am not sure how to collect all the paths. How did you generate that string? Did you join all the strings listed in `Libdl.dllist()`?

---

<div class="post-metadata">

**Author:** ![amontoison](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amontoison/32/218741_2.png) [@amontoison](https://discourse.julialang.org/u/amontoison)\
**Post date:** [November 22, 2024, 4:19am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/8 "2024-11-22T04:19:06Z")

</div>

I just copy-pasted the output of this small script, but I forgot to mention that I had loaded `HSL_jll.jl` before running it:

```julia
using Libdl, HSL_jll

libs = Libdl.dllist()
dirs = unique(dirname(path) for path in libs)
separator = Sys.iswindows() ? ";" : ":"
library_paths = join(dirs, separator)

if Sys.iswindows()
    println("PATH=" * library_paths)
elseif Sys.isapple()
    println("DYLD_LIBRARY_PATH=" * library_paths)
else
    println("LD_LIBRARY_PATH=" * library_paths)
end

```

Then I closed Julia, copy-pasted the new environment variable `LD_LIBRARY_PATH` into my shell, launched Julia, and ran your code:

```shell
shell> LD=LIBRARY_PATH=...
shell> julia

```

```julia
using JuMP, Ipopt_jll, HSL, HSL_jll, AmplNLWriter

model = Model(() -> AmplNLWriter.Optimizer(Ipopt_jll.amplexe))
@variable(model, x[1:2] >= 0)
@objective(model, Min, sum(x.^2))
@constraint(model, sin(x[1]) <= -0.5)

set_optimizer_attribute(model, "hsllib", HSL_jll.libhsl_path)
set_optimizer_attribute(model, "linear_solver", "ma27")

optimize!(model)

```

For your question about how I generate that string, I just joined all the strings listed in `Libdl.dllist()` with a sepator `:` (on Linux) between them.

---

<div class="post-metadata">

**Author:** ![pulsipher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pulsipher/32/10461_2.png) [@pulsipher](https://discourse.julialang.org/u/pulsipher)\
**Post date:** [November 22, 2024, 11:37am UTC](https://discourse.julialang.org/t/cannot-get-hsl-to-work-with-amplnlwriter/122903/9 "2024-11-22T11:37:51Z")

</div>

That did the trick, thanks!