Cannot install FTS extension in DuckDB

General Usage
,
21

According to Readme · DuckDB.jl (juliahub.com) you’d probably have to delete the cached DuckDB folders in .julia/compiled/v1.10/ (or your version of Julia) before adding the package again, so the package gets recompiled with the version of DuckDB that you have pointed to with your environment variable

I got extensions to work on Windows now by doing the following:

  1. Open PowerShell on Windows and set the environment variable like so:

$Env:JULIA_DUCKDB_LIBRARY=“path/to/duckdb.dll”

  1. Run julia.exe from the same Shell instance
  2. Delete DuckDB and DuckDB_jll folders from .julia/compiled/v1.10/
  3. From the Julia repl running in PowerShell: Remove the DuckDB package from Julia and add it again
  4. Just proceed to set up your DuckDB connection and use “INSTALL fts” without any further modifications

This might also work by setting the environment variable directly in Julia like you already did instead of using PowerShell, but I haven’t tested it myself. If you set the environment variable temporarily in PowerShell things might break with the next update of DuckDB.jl and you might have to repeat the whole process over, though

I don’t know how so set environment variables permanently on Windows without admin rights

If setting the environment variable from within Julia works for you, you could maybe add it to a startup file in the .julia/config folder as decribed here so you won’t ever have to worry about setting it before updating DuckDB.jl (you’d still have to remember to always manually download and replace the matching duckdb.dll, though)

2 Likes
23

I deleted the DuckDB and DuckDB_jll folders and then tried this:

import Pkg
Pkg.activate(;temp=true)
ENV["JULIA_DUCKDB_LIBRARY"] = raw"C:\Users\TGebbels\duckdb.dll"
Pkg.add(["DuckDB"])
using DuckDB
con = DBInterface.connect(DuckDB.DB, "ddb.DuckDB")
DBInterface.execute(con, "INSTALL fts";)
DBInterface.execute(con, "LOAD fts";)

It worked!

23×3 DataFrame
 Row │ extension_name    installed  description
     │ String?           Bool?      String?
─────┼────────────────────────────────────────────────────────────────
...
   7 │ fts                    true  Adds support for Full-Text Searc…
...

Thank you for spelling it out for me!

1 Like
24

What does it mean that this works? Like what does it tell us about what is wrong in the DuckDB.jl package? Is it some sort of version incompatibility with the DuckDB_jll and the extension?

25

This seems to be the explanation of the issue, but I don’t really understand what it means:

This is not as simple as fixing the platform regrettably. Julia uses the mingw toolchain to compile Windows extensions in Yggdrassil, instead of the native MSVC toolchain, see e.g. here. As a result, the Julia binaries are not compatible with standard DuckDB extensions like the ones compiled for e.g. Python. For Julia on Windows we would need to compile all extensions with the same MinGW toolchain which is a large undertaking.

Fortunately for me, the workaround above works for now.

26

I wonder if that means we could simply serve up our own extensions? Or perhaps it would be easier to build the DuckDB_jll differently?

27

I see a few possibilities:

  • Some 3rd non-DuckDB entity starts to build the DuckDB extensions in DuckDB’s packages format for the windows mingw target. That target could be patched to point to that 3rd party’s server.
  • The extensions are build as Julia packages in Yggdrasil. Some kind of message suggesting to install using Pkg could be added when people try to install packages through DuckDB. This seems reasonably do-able since all the infrastructure is in place.
  • DuckDB gives in and makes windows + mingw an official platform – but this seems unlikely: projects usually either choose one of MSVC or mingw for 1st party support and the other may get 3rd party support (see: e.g. Python having full support for MSVC and the unofficial mingw builds having much more spotty support).
  • It may be possible to use MSVC compiled extensions from MINGW DuckDB now that it can be extended using a C interface rather than a C++ one C API extensions by samansmink · Pull Request #12682 · duckdb/duckdb · GitHub . I don’t know a lot about this stuff, but enough to know that there could still be some obstacles to it working smoothly.
28

Actually, if I look at the bandaid fix above (I haven’t tried it yet), the only actual change is it downloads the duckdb.dll file from their GitHub, right? Isn’t a possible shortterm fix to just use those files in DuckDB_jll instead of building our own?

29

The _jll files are all build by BinaryBuilder from recipes in Yggdrasil. I don’t think it would fly just downloading a binary instead.

It might be possible to provide a package libduckdb_upstream or similar which has DuckDB’s release as an artifact. The main problem I see is that Julia’s dependency management does not allow for multiple packages to “provide” some virtual package, so DuckDB.jl would still depend on DuckDB_jll.

30

As far as I understand, their binary blob links to MSVC, but Julia doesn’t work with that compiler, so just fetching more easily that library won’t address the underlining problem of incompatible libraries.