Help debugging failing precompilation of DuckDB.jl [Windows]

General Usage
1

When installing DuckDB.jl on windows, precompilation fails:
The issue is caused by failing precompilation of the DuckDB_jll binary.

Stacktrace 
(@v1.11) pkg> add DuckDB
    Updating registry at `C:\Users\dennis.bal\.julia\registries\General.toml`
   Resolving package versions...
   Installed DuckDB_jll ─ v1.1.2+0
  Downloaded artifact: DuckDB
    Updating `C:\Users\dennis.bal\.julia\environments\v1.11\Project.toml`
  [d2f5444f] + DuckDB v1.1.0
    Updating `C:\Users\dennis.bal\.julia\environments\v1.11\Manifest.toml`
  [c3b6d118] + BitIntegers v0.3.2
  [a10d1c49] + DBInterface v2.6.1
  [d2f5444f] + DuckDB v1.1.0
  [fb4d412d] + FixedPointDecimals v0.5.3
  [842dd82b] + InlineStrings v1.4.2
  [ea10d353] + WeakRefStrings v1.4.2
  [2cbbab25] + DuckDB_jll v1.1.2+0
Precompiling project...
  ✗ DuckDB
  189 dependencies successfully precompiled in 216 seconds. 151 already precompiled.
  1 dependency errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

(@v1.11) pkg> precompile
Precompiling project...
  ✗ DuckDB
  0 dependencies successfully precompiled in 6 seconds. 340 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

DuckDB

Failed to precompile DuckDB [d2f5444f-75bc-4fdf-ac35-56f514c445e1] to "C:\\Users\\dennis.bal\\.julia\\compiled\\v1.11\\DuckDB\\jl_1D62.tmp".
ERROR: LoadError: could not load symbol "duckdb_vector_size":
The specified procedure could not be found.
Stacktrace:
 [1] duckdb_vector_size()
   @ DuckDB C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\api.jl:715
 [2] top-level scope
   @ C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
 [3] include(mod::Module, _path::String)
   @ Base .\Base.jl:557
 [4] include(x::String)
   @ DuckDB C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
 [5] top-level scope
   @ C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:28
 [6] include
   @ .\Base.jl:557 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base .\loading.jl:2806
 [8] top-level scope
   @ stdin:4
in expression starting at C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
in expression starting at C:\Users\dennis.bal\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
in expression starting at stdin:

This issue is reported here. In that discussion, one user stated that installing the binary from the webpage and setting

ENV["JULIA_DUCKDB_LIBRARY"] = raw"C:\path\to\duckdb.dll"

fixed the issue. I can confirm that this indeed fixes the issue.

So to my mind, this means that the binary downloaded by the website is different from the binary built by Yggdrasil, the build-script for which is found here. I am unable to see if that build importantly differs from the official DuckDB build instructions, but I have a hunch that this is where the problem lies.

Is there anyone why can sniff out where the problem is? Given that the binary as downloaded from the webpage works, I suspect that this is an easy fix, and it would be nice if the package did not stay broken in windows forever if the fix is simple.

2 
ERROR: LoadError: could not load symbol "duckdb_vector_size":
The specified procedure could not be found.

Sounds like they’re trying to call a symbol not defined/visible in the library. If that was supposed to be visible it’s usually a bug in the source code of the library or its build system.

3

Right. So given that the same source code likely produced both the binaries downloaded from the website, and the binaries buildt by Yggdrasil, then it seems like a good bet to say that the build is wrong for windows?

Official build instructions for windows 
python scripts/windows_ci.py
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR_PLATFORM=x64 -DENABLE_EXTENSION_AUTOLOADING=1 -DENABLE_EXTENSION_AUTOINSTALL=1 -DDUCKDB_EXTENSION_CONFIGS="${GITHUB_WORKSPACE}/.github/config/bundled_extensions.cmake" -DDISABLE_UNITY=1 -DOVERRIDE_GIT_DESCRIBE="$OVERRIDE_GIT_DESCRIBE"
        cmake --build . --config Release --parallel
Contents of `https://github.com/JuliaPackaging/Yggdrasil/edit/master/D/DuckDB/build_tarballs.jl` 
# Note that this script can accept some limited command-line arguments, run
# `julia build_tarballs.jl --help` to see a usage message.
using BinaryBuilder, Pkg

name = "DuckDB"
version = v"1.1.2"

# Collection of sources required to complete build
sources = [
    GitSource("https://github.com/duckdb/duckdb.git", "f680b7d08f56183391b581077d4baf589e1cc8bd"),
]

# Bash recipe for building across all platforms
script = raw"""
cd $WORKSPACE/srcdir/duckdb/

cmake -B build \
      -DCMAKE_INSTALL_PREFIX=${prefix} \
      -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} \
      -DCMAKE_BUILD_TYPE=Release \
      -DENABLE_SANITIZER=FALSE \
      -DBUILD_EXTENSIONS='autocomplete;icu;parquet;json;fts;tpcds;tpch' \
      -DENABLE_EXTENSION_AUTOLOADING=1 \
      -DENABLE_EXTENSION_AUTOINSTALL=1 \
      -DBUILD_UNITTESTS=FALSE \
      -DBUILD_SHELL=TRUE \
      -DDUCKDB_EXPLICIT_PLATFORM="${target}"
cmake --build build --parallel ${nproc}
cmake --install build

if [[ "${target}" == *-mingw32 ]]; then
    install -Dvm 755 "build/src/libduckdb.${dlext}" -t "${libdir}"
fi
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = expand_cxxstring_abis(supported_platforms())
# Building for PowerPC results in errors inside jemalloc:
#     /tmp/ccmHnfhC.s: Assembler messages:
#     /tmp/ccmHnfhC.s:7829: Error: unrecognized opcode: `pause'
#     make[2]: *** [extension/jemalloc/jemalloc/CMakeFiles/jemalloc.dir/build.make:76: extension/jemalloc/jemalloc/CMakeFiles/jemalloc.dir/src/jemalloc.c.o] Error 1
filter!(p -> arch(p) != "powerpc64le", platforms)

# The products that we will ensure are always built
products = [
    LibraryProduct("libduckdb", :libduckdb),
    ExecutableProduct("duckdb", :duckdb),
]

# Dependencies that must be installed before this package can be built
dependencies = Dependency[
]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; preferred_gcc_version = v"6.1.0", julia_compat="1.6")

I attempted adding a special case for windows in this PR that should run the python command, and also slightly modified the cmake command to closer match the official build instructions for windows. Hope that does it!

1 Like