FFTW x 2 = symbol lookup error

General Usage
,
1

I have a package that uses FFTW (via FastTransforms). A user is trying to load it in python (via PythonCall), along with any one of various other python packages that also use FFTW. Apparently, the latter include shared libraries that don’t have fftw_init_threads available, so python immediately errors out.

On linux, the error is

python: symbol lookup error: ~/.julia/artifacts/.../lib/libfasttransforms.so: undefined symbol: fftw_init_threads

On macOS, it looks like

dyld[29279]: missing symbol called
[1]    29279 abort      python -c 'import scri; from sxs import julia'

It so happens that reversing the import order seems to solve the problem, but I’m worried that’s just delaying some worse incompatibility. Is there any way to actually solve this problem (preferably one that doesn’t involve recompiling the offending python packages)? Some fancy symbol renaming or dlopen craziness?

I know this isn’t the best MWE, but in case anyone is really motivated:

mamba create -y -n test_env scri sxs
mamba activate test_env
python -c 'import scri; from sxs import julia'
1 Like
2

Welcome to DLL Hell :cry: . This is a recurrent problem (not just with FFTW) when loading packages dynamically (usually via Python) that link to conflicting versions of shared libraries. I don’t know of a great solution in general that doesn’t require recompiling something.

4 Likes
3

If you want to continue using PythonCall, usually the solution to this sort of issue is to ensure that the library (fftw in this case) is installed at the same (or ABI-compatible) versions in Julia and Python.

This is easier if you can control the load order (i.e. whether Python or Julia loads the lib first) because then you basically just need to ensure that whichever is loaded second is at an older version.

The other solution is to use Python and Julia in separate processes. That is, don’t use PythonCall and instead launch a separate Python process and talk to it via some sort of IPC.

2 Likes
4

To be clear, I didn’t mean to impugn FFTW; it’s just a prime example because it’s so widely used. :slight_smile:

Thanks for the comments, guys.

5

If you know where python’s build of FFTW is, you can make Julia use it: JLL packages · BinaryBuilder.jl

using Preferences
set_preferences!(
    "LocalPreferences.toml",
    "FFTW_jll",
    "libfftw3_path" => "/path/to/libfftw3",
    "libfftw3f_path" => "/path/to/libfftw3f",
)

You need to have both Preferences and FFTW_jll direct dependencies of your active environment.

6

That’s the wrong way around, because FFTW.jl assumes that the library has been built with multithreading included.

(To be fair, that’s a non-standard build option of FFTW_jll; normally the threads stuff is in a separate library libfftw3_threads. There is a reasonable argument to change the Yggdrasil build to work like this and then load both libraries from FFTW.jl. On the other hand, that may run into problems on Windows, which makes life difficult for shared libraries with inter-library dependencies IIRC.)

1 Like
7

Then, if there’s no way to point python’s build to Julia’s I don’t think there’s a way around recompiling python’s FFTW package with Julia’s libfftw.

What problem?

8

It used to be that cross-compiling to Windows with autotools (automake/libtool) made it impossible to create a second shared library libfftw3_threads.dll that depends on a first shared library libfftw3.dll. I don’t know if this limitation still exists? It is still in the libtool documentation:

Some platforms, such as Windows, do not even allow you this flexibility. In order to build a shared library, it must be entirely self-contained or it must have dependencies known at link time (that is, have references only to symbols that are found in the .lo files or the specified ‘-l’ libraries), and you need to specify the -no-undefined flag. By default, libtool builds only static libraries on these kinds of platforms.

9

What the linker for Windows doesn’t allow is to have unresolved symbols. But if some symbols in libA are marked as undefined but they are provided and exported by libB, and libA explicitly links to libB (i.e. libA has libB in the list of required libraries, what you can see for example with objdump -x libA.dll) then the linker is able to resolve the undefined symbols in libA from libB and all is good. The problem is when libA has undefined symbols which are provided and exported by libB and libA doesn’t explicitly link to libB, a situation which Linux linkers allow.