# FFTW x 2 = symbol lookup error

**URL:** <https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528>\
**Category:** General Usage\
**Tags:** fftw, pythoncall\
**Created:** [July 26, 2024, 6:18pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528 "2024-07-26T18:18:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [July 26, 2024, 6:18pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/1 "2024-07-26T18:18:08Z")

</div>

I have [a package](https://github.com/moble/SphericalFunctions.jl) that uses FFTW (via FastTransforms). A user is trying to load it in python (via PythonCall), along with any one of various other python packages that also use FFTW. Apparently, the latter include shared libraries that don’t have `fftw_init_threads` available, so python immediately errors out.

On linux, the error is

```julia
python: symbol lookup error: ~/.julia/artifacts/.../lib/libfasttransforms.so: undefined symbol: fftw_init_threads

```

On macOS, it looks like

```julia
dyld[29279]: missing symbol called
[1] 29279 abort python -c 'import scri; from sxs import julia'

```

It so happens that reversing the import order seems to solve the problem, but I’m worried that’s just delaying some worse incompatibility. Is there any way to actually solve this problem (preferably one that doesn’t involve recompiling the offending python packages)? Some fancy symbol renaming or dlopen craziness?

* * *

I know this isn’t the best MWE, but in case anyone is really motivated:

```bash
mamba create -y -n test_env scri sxs
mamba activate test_env
python -c 'import scri; from sxs import julia'

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 26, 2024, 6:32pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/2 "2024-07-26T18:32:56Z")

</div>

Welcome to [DLL Hell](https://en.wikipedia.org/wiki/DLL_Hell) 😢 . This is a recurrent problem (not just with FFTW) when loading packages dynamically (usually via Python) that link to conflicting versions of shared libraries. I don’t know of a great solution in general that doesn’t require recompiling something.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [July 26, 2024, 7:07pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/3 "2024-07-26T19:07:43Z")

</div>

If you want to continue using PythonCall, usually the solution to this sort of issue is to ensure that the library (fftw in this case) is installed at the same (or ABI-compatible) versions in Julia and Python.

This is easier if you can control the load order (i.e. whether Python or Julia loads the lib first) because then you basically just need to ensure that whichever is loaded second is at an older version.

The other solution is to use Python and Julia in separate processes. That is, don’t use PythonCall and instead launch a separate Python process and talk to it via some sort of IPC.

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [July 26, 2024, 8:21pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/4 "2024-07-26T20:21:55Z")

</div>

> [@stevengj](#):
>
> This is a recurrent problem (not just with FFTW)

To be clear, I didn’t mean to impugn FFTW; it’s just a prime example because it’s so widely used. 🙂

Thanks for the comments, guys.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 26, 2024, 9:02pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/5 "2024-07-26T21:02:53Z")

</div>

If you know where python’s build of FFTW is, you can make Julia use it: [JLL packages · BinaryBuilder.jl](https://docs.binarybuilder.org/stable/jll/#Overriding-specific-products)

```julia
using Preferences
set_preferences!(
    "LocalPreferences.toml",
    "FFTW_jll",
    "libfftw3_path" => "/path/to/libfftw3",
    "libfftw3f_path" => "/path/to/libfftw3f",
)

```

You need to have both `Preferences` and `FFTW_jll` direct dependencies of your active environment.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 26, 2024, 9:25pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/6 "2024-07-26T21:25:38Z")

</div>

> [@giordano](#):
>
> If you know where python’s build of FFTW is, you can make Julia use it: [JLL packages · BinaryBuilder.jl](https://docs.binarybuilder.org/stable/jll/#Overriding-specific-products)

That’s the wrong way around, because FFTW.jl assumes that the library has been [built with multithreading included](https://github.com/JuliaMath/FFTW.jl/blob/f888022d7a1ff78491abf8f33f1055cc52a68f0a/src/FFTW.jl#L32).

(To be fair, that’s a [non-standard build option](https://github.com/JuliaPackaging/Yggdrasil/blob/ce1449a6a391dea61474a9db87c5c6b3b7da8b71/F/FFTW/build_tarballs.jl#L26) of `FFTW_jll`; normally the threads stuff is in a separate library `libfftw3_threads`. There is a reasonable argument to change the Yggdrasil build to work like this and then load both libraries from FFTW.jl. On the other hand, that may run into problems on Windows, which makes life difficult for shared libraries with inter-library dependencies IIRC.)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 26, 2024, 9:34pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/7 "2024-07-26T21:34:34Z")

</div>

> [@stevengj](#):
>
> That’s the wrong way around, because [FFTW.jl](https://juliahub.com/ui/Packages/General/FFTW) assumes that the library has been [built with multithreading included](https://github.com/JuliaMath/FFTW.jl/blob/f888022d7a1ff78491abf8f33f1055cc52a68f0a/src/FFTW.jl#L32).

Then, if there’s no way to point python’s build to Julia’s I don’t think there’s a way around recompiling python’s FFTW package with Julia’s libfftw.

> [@stevengj](#):
>
> On the other hand, that may run into problems on Windows, which makes life difficult for shared libraries with inter-library dependencies IIRC.

What problem?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 26, 2024, 10:36pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/8 "2024-07-26T22:36:14Z")

</div>

> [@giordano](#):
>
> What problem?

It used to be that cross-compiling to Windows with autotools (automake/libtool) made it impossible to create a second shared library `libfftw3_threads.dll` that depends on a first shared library `libfftw3.dll`. I don’t know if this limitation still exists? It is still in the [libtool documentation](https://www.gnu.org/savannah-checkouts/gnu/libtool/manual/html_node/Inter_002dlibrary-dependencies.html):

> Some platforms, such as Windows, do not even allow you this flexibility. In order to build a shared library, it must be entirely self-contained or it must have dependencies known at link time (that is, have references only to symbols that are found in the `.lo` files or the specified `‘-l’` libraries), and you need to specify the `-no-undefined` flag. By default, libtool builds only static libraries on these kinds of platforms.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [July 26, 2024, 10:47pm UTC](https://discourse.julialang.org/t/fftw-x-2-symbol-lookup-error/117528/9 "2024-07-26T22:47:04Z")

</div>

What the linker for Windows doesn’t allow is to have unresolved symbols. But if some symbols in `libA` are marked as undefined but they are provided and exported by `libB`, and `libA` explicitly links to `libB` (i.e. `libA` has `libB` in the list of required libraries, what you can see for example with `objdump -x libA.dll`) then the linker is able to resolve the undefined symbols in `libA` from `libB` and all is good. The problem is when `libA` has undefined symbols which are provided and exported by `libB` _and_ `libA` doesn’t explicitly link to `libB`, a situation which Linux linkers allow.