# Query.jl extremely slow

**URL:** <https://discourse.julialang.org/t/query-jl-extremely-slow/21078>\
**Category:** General Usage\
**Created:** [February 22, 2019, 1:58pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078 "2019-02-22T13:58:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 22, 2019, 1:58pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/1 "2019-02-22T13:58:58Z")

</div>

I’m trying to use `Query.jl` for some basic filtering, but couldn’t get the result even after a very long wait. After reducing the number of columns, I get the output, but still it’s much much slower than plain `DataFrame` filtering and scales badly with the number of columns (exponentially?). The issue is easily reproducible:

```julia
using DataFrames, Query

a = Vector{Union{Missing, Float64}}(rand(1_000_000))
kws = Dict(Symbol("a$i") => copy(a) for i in 1:XX) # <<---- here
df = DataFrame(; kws...)
display(df |> size)
df = df |> @filter(_.a1.hasvalue)
display(df)
display(df |> DataFrame |> size)

```

With `20` in place of `XX` at the specified line I get the result after ~40 seconds, with `30` or more I get no result even after a long time. The memory usage also gets very high, in the tens of gigabytes for larger column counts.

Possibly the issue is related with handling `Missing` as I don’t experience the slowdown with `_.a1 > 0` instead of `_.a1.hasvalue` for example. However I’m not sure, maybe there are other more general reasons.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 23, 2019, 1:14am UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/2 "2019-02-23T01:14:32Z")

</div>

That is a bug in [Tables.jl](https://github.com/JuliaData/Tables.jl), that might be fixed by [Avoid intermediate copy of table for TableTraits sources by davidanthoff · Pull Request #67 · JuliaData/Tables.jl · GitHub](https://github.com/JuliaData/Tables.jl/pull/67). In theory @quinnj has tagged a version that includes that fix, but the version tag also has a bug, so it is stuck right now (see [here](https://github.com/JuliaLang/METADATA.jl/pull/21650)).

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 23, 2019, 10:34am UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/3 "2019-02-23T10:34:07Z")

</div>

So is there a way to test this fix now before the final release of that version? The bug and fix description don’t look related to my issue: it seems that an additional copy should introduce a relatively minor slowdown only. While the code in the first post is slower than expected by many orders of magnitude.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 23, 2019, 3:31pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/4 "2019-02-23T15:31:45Z")

</div>

You can get the `master` branch of Tables.jl via `pkg> add Tables#master`. Just make sure you `free Tables` once a new version has been released.

I found the unnecessary copy part of the code while I was trying to diagnose a slowdown that was similarly extreme to what you found. Removing the copy got rid of that extreme slowdown, and on my system also fixes your problem. There is clearly something else broken somewhere, but removing the copy seems enough to avoid the problematic code path for Query (and the copy shouldn’t happen no matter what).

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [February 23, 2019, 4:12pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/5 "2019-02-23T16:12:56Z")

</div>

I wonder if that PR will fix my [issue](https://github.com/queryverse/Query.jl/issues/232)? Looks like its the same issue…

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 23, 2019, 5:16pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/6 "2019-02-23T17:16:35Z")

</div>

@affans yes, that might also be fixed by it. Would be fantastic if you could report back after trying with the new version!

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [February 23, 2019, 5:32pm UTC](https://discourse.julialang.org/t/query-jl-extremely-slow/21078/7 "2019-02-23T17:32:32Z")

</div>

Thanks for your suggestion, will certainly try it!