Is it hard to support Julia UDFs in DuckDB?

Specific Domains Data
21

@asbisen so, @tqml was the mind who made the great macro, which is a big UI improvement imo for the UDFs in duckdb.

you can see an example in this discussion. the issue we both ran into (for the version i merged and this macro), was segmentation fault when one of the arguments has a data type other than Int/Float

the code is available in duckdb fork on his repo

in short it looks like this

y_sum = (a,b) -> a+b
fun = @create_scalar_function "my_sum" [Int64, Int64] Int64 my_sum
register_scalar_function(db, fun)
execute(db, "SELECT my_sum(1,2)")
2 Likes
22

Hey @drizk1 thank you for bringing it up again. I developed a PoC that could work but it has some limitations but it is not-quite ready for merging. Also did not have much time to work on it in the last months.

  • I made a macro and non-macro version. The non macro wrapper has very poor performance because it uses generic types and the dispatch happens at runtime
  • The macro versions need to use eval() on ARM processors (or non-x86 architectures) which seems wrong? But I’m quite new to macro programming on Julia and would love some input here.
  • I could solve at least the string variant. DuckDB has a function assign_string_element() that does exactly what we need.

Link to the implementation: duckdb/tools/juliapkg/src/scalar_function.jl at julia/scalar-function · tqml/duckdb · GitHub

2 Likes