# Direct interface to Polars Rust library

**URL:** <https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283>\
**Category:** Data\
**Tags:** question\
**Created:** [May 23, 2023, 1:56pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283 "2023-05-23T13:56:25Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmbates](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmbates/32/44_2.png) [@dmbates](https://discourse.julialang.org/u/dmbates)\
**Post date:** [May 23, 2023, 1:56pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/1 "2023-05-23T13:56:25Z")

</div>

There has been recent discussion about some benchmarking by the DuckDB folks with good results for DuckDB (“Well they would say that, wouldn’t they?” M. Rice-Davies) and Polars.

Especially now that Polars supports reading and writing the Arrow IPC file format for its DataFrame and Series types, it is reasonable to write a Julia DataFrame with Arrow.jl then read and manipulate it in Polars.

I have been using PyCall.jl and its `pyimport` function as

```julia
pl = pyimport("polars")

```

to access the Python bindings within a Julia session.

I am not that familiar with the internals of Python nor the distinctions between PyCall.jl and PythonCall.jl.

- Would there be an inherent advantage, perhaps related to zero-copy, in using PythonCall.jl to access the Python bindings for Polars?

- Would it be feasible/worthwhile creating a Julia package to access the Rust entry points for Polars? At [pola-rs · GitHub](https://github.com/pola-rs/) there are bindings in Python, R, Node.js and pyo3. Is it that much of a stretch to produce a Julia package to access the polars Rust functions?

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 23, 2023, 2:05pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/2 "2023-05-23T14:05:41Z")

</div>

I know of one direct path – jlrs

> **[GitHub - Taaitaaiger/jlrs: Julia bindings for Rust](https://github.com/Taaitaaiger/jlrs)**
>
> Julia bindings for Rust. Contribute to Taaitaaiger/jlrs development by creating an account on GitHub.

But also your opening paragraph does not really make much sense to me. What does DuckDB have to do with this, specifically? If you want to use DuckDB, just do `using DuckDB, DBInterface` (annoying that we need that second one).

Really I think DataFrames and Polars are close competitors. I’m not sure that a “Polars.jl” is really needed or worth the likely large amount of effort to make it happen.

---

<div class="post-metadata">

**Author:** ![dmbates](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmbates/32/44_2.png) [@dmbates](https://discourse.julialang.org/u/dmbates)\
**Post date:** [May 23, 2023, 2:38pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/3 "2023-05-23T14:38:06Z")

</div>

I was thinking of the benchmarks mentioned in recent posts on [https://discourse.julialang.org/t/the-state-of-dataframes-jl-h2o-benchmark/](https://discourse.julialang.org/t/the-state-of-dataframes-jl-h2o-benchmark/) when I mentioned DuckDB. The latest version of the benchmarks were performed by the DuckDB folks to show the speed of their most recent version. The [benchmark results](https://duckdblabs.github.io/db-benchmark/) also show that Polars is very fast on these benchmarks. The mention of DuckDB was only to provide motivation, through their benchmarks, for interfacing to Polars.

It is certainly possible to use PyCall or PythonCall and the Python bindings for Polars to do the types of summaries shown in the benchmarks. But I don’t know if that route will cause copies of large objects to be made. I believe that the existing Arrow.Table and Arrow.write functions in Arrow.jl can provide for zero-copy exchange of Arrow tables between Julia and Rust, but I don’t know much about the Rust end of that.

The point of interfacing to Polars is not to replace DataFrames.jl but rather to enhance it for cases of large complicated joins and summaries.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [May 23, 2023, 3:15pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/4 "2023-05-23T15:15:12Z")

</div>

> [@dmbates](#):
>
> The point of interfacing to Polars is not to replace DataFrames.jl but rather to enhance it for cases of large complicated joins and summaries.

Note that DataFrames.jl is noticeably slow only in some cases. Typically it is comparable (even if slower). And I could give you benchmarks, when DataFrames.jl is faster than Polars (one relevant case is when you run code on a laptop not on a server having 40-cores; on laptop DataFrames.jl performance is relatively more attractive).

My current thinking is:

- we know in what cases we need to improve DataFrames.jl (it is mostly operations when you have many small groups; we are quite good if there are few large groups); in the cases when we are really slow we plan to improve them
- still it should be expected that in the long run DuckDB/Polars will be faster on the average; the reason is that many more people are willing to work on performance of these frameworks (I opened GSoC/JSoC call for performance improvements for DataFrames.jl this year and it did not get people interested in it)
- if I have to choose DuckDB vs Polars in my work I choose DuckDB as a fallback; the reasons are that  
i. we already have a package DuckDB.jl that allows me to do it  
ii. it is in general faster (not by much but faster)  
iii. it is enough that I know SQL to use it (Polars also has SQL interface, but for DuckDB this is a native interface).

Also, in general, can you please report in DataFrames.jl issue cases when you in practice find DataFrames.jl to be exceedingly slow? Focusing on a concrete classes of cases is the easiest way for us to improve.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 23, 2023, 3:31pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/5 "2023-05-23T15:31:13Z")

</div>

> [@bkamins](#):
>
> one relevant case is when you run code on a laptop not on a server having 40-cores

I guess DF.jl doesn’t automatically use multi-threading, which is fine, I wish the H2O DB benchmark has single thread benchmark.

But I guess they’re coming from big DB perspective, there’s no point in not doing everything multi-threaded. In deed, I think it’s pretty rare when you need to `groupby` many smaller dataframes in parallel, than having just a big “data base”

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [May 23, 2023, 3:40pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/6 "2023-05-23T15:40:10Z")

</div>

Roughly yes. They come from “big database” perspective, and I think it makes sense.

In terms of DataFrames.jl:

- originally it was developed as single-threaded;
- we are slowly adding multi-threading in different places, so things improve, but this is work in progress and did not have a top priority (we put more priority on single-threaded performance, functionality coverage, and API flexibility); but e.g. until Julia fully resolves issues with GC in multi-threading code we will have problems (GC is not an issue for DuckDB and Polars)

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 5:06pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/7 "2023-10-31T17:06:19Z")

</div>

DuckDB has been archived since 2022, and is not compatable with the latests versions of DBeaver.

[https://github.com/kimmolinna/DuckDB.jl](https://github.com/kimmolinna/DuckDB.jl)

I have moved my question to a seperate topic, since it appears to be a use issue.  
[Connecting to DuckDB](https://discourse.julialang.org/t/compatablity-issue-between-versions-of-duckdb/105624)

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 5:08pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/8 "2023-10-31T17:08:13Z")

</div>

Will Dtables be able to address multithreading? What is the GC problem?

I think what’s missing from the Pluto and JuliaHub, and the Julia ecosystem as a whole, is a GUI table editor. With SQLite (or DuckDB if it’s developed). I should be able to connect an editor like DBeaver to enter tables, and map joins, while using the Pluto notebook to analyze data, and for visualization/publishing data.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 31, 2023, 5:22pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/9 "2023-10-31T17:22:04Z")

</div>

> [@brett\_knoss](#):
>
> DuckDB has been archived since 2022, and is not compatable with the latests versions of DBeaver.

DuckDB.jl now lives inside the official DuckDB repo:

> <https://github.com/duckdb/duckdb/tree/main/tools/juliapkg>
>
> //github.com/duckdb/duckdb/tree/main/tools/juliapkg

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 31, 2023, 5:27pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/10 "2023-10-31T17:27:27Z")

</div>

> [@brett\_knoss](#):
>
> What is the GC problem?

If you use allocated objects in columns of your table then and have billions of rows then GC often starts to be the most expensive part of executing data transformations.

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 5:31pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/11 "2023-10-31T17:31:27Z")

</div>

How do I check the version of DuckDB is the same in DBeaver and Julia?

---

<div class="post-metadata">

**Author:** ![rdavis120](https://avatars.discourse-cdn.com/v4/letter/r/b5a626/32.png) [@rdavis120](https://discourse.julialang.org/u/rdavis120)\
**Post date:** [October 31, 2023, 5:47pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/12 "2023-10-31T17:47:33Z")

</div>

In case you hit an issue, I have not been able to work with duckdb since the 0.9.1 release and there is an open [issue](https://github.com/duckdb/duckdb/issues/9419) which others have encountered as well.

Also there is a [Polars.jl](https://github.com/Pangoraw/Polars.jl) being worked on.

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [October 31, 2023, 5:57pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/13 "2023-10-31T17:57:20Z")

</div>

I got DuckDB to work on Julia or DBeaver, but I can’t communicate between the two. What happened to JuliaDB?

Would there be a benefit to making Julia able to call Rust libraries, the same way it can call C or Fortran libraries?

---

<div class="post-metadata">

**Author:** ![brett\_knoss](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brett_knoss/32/13050_2.png) [@brett\_knoss](https://discourse.julialang.org/u/brett_knoss)\
**Post date:** [November 9, 2023, 9:42pm UTC](https://discourse.julialang.org/t/direct-interface-to-polars-rust-library/99283/14 "2023-11-09T21:42:42Z")

</div>

What if i was doing something like employee evaluations, and I want to study the performance of individual employees and staff as a whole. Would rows be better for the former and columns for the later? Would a vector or graph database be better suited to evaluating each as needed?