# Basic function usage in Query @filter

**URL:** <https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642>\
**Category:** General Usage\
**Tags:** query, queryverse\
**Created:** [May 23, 2024, 7:46pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642 "2024-05-23T19:46:38Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![seadoo](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@seadoo](https://discourse.julialang.org/u/seadoo)\
**Post date:** [May 23, 2024, 7:46pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/1 "2024-05-23T19:46:38Z")

</div>

Is the `@filter` macro from Query.jl only useful for simple things like `_.a > 1` and so on?  
I’ve tried to use `ismissing` which just returns a boolean and it does not return anything (but also does not return any errors). Broadcasting `ismissing` also doesn’t change the outcome (results in an error).

Here is my example with commented output beneath each comparing Query.jl syntax and basic DataFrames.jl syntax (which is successful):

```julia
using DataFrames, Query

example = DataFrame(
	alph = ["a", "b", "c", "d"],
	name = ["apple", "banana", "carrot", "date"],
	prct = [0.2, 1, missing, 10]
)

example |>
@filter(ismissing(_.prct)) |>
collect DataFrame
# Output: [], no rows

example |>
x -> subset(x, :prct => ByRow(ismissing))
# Output:
#	alph name prct
# 1 "c" "carrot" missing

# Edit, LINQ style just for completeness
@from i in example begin
    @where ismissing(i.prct)
    @select i
    @collect DataFrame
end
# Output: []. no rows

```

I also tried anonymous function syntax with no luck. Am I using `@filter` wrong?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [May 23, 2024, 8:11pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/2 "2024-05-23T20:11:26Z")

</div>

There seems to be some lack of documentation: [Docs for missing values · Issue #306 · queryverse/Query.jl · GitHub](https://github.com/queryverse/Query.jl/issues/306)

The docs about this: [Getting Started · Query.jl](https://www.queryverse.org/Query.jl/stable/gettingstarted/#Missing-values-1)

Still, nobody tells us, how it is supposed to work. I tried and found that this is working, but I don’t know if it is the right way:

```julia
julia> using DataValues

julia> example |>
       @filter(_.prct == DataValue(missing))
1x3 query result
alph │ name │ prct
─────┼────────┼─────
c │ carrot │ #NA

```

Let’s ping those who know: @baggepinnen😉

---

<div class="post-metadata">

**Author:** ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)\
**Post date:** [May 23, 2024, 8:12pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/3 "2024-05-23T20:12:37Z")

</div>

It looks like `Query.jl` treats missing values as na:

```julia
julia> example |> @filter(isna(_.prct))
1x3 query result
alph │ name │ prct
─────┼────────┼─────
c │ carrot │ #NA

```

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [May 23, 2024, 8:14pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/4 "2024-05-23T20:14:19Z")

</div>

See [Query.jl - filtering on missing data](https://discourse.julialang.org/t/query-jl-filtering-on-missing-data/14898)

---

<div class="post-metadata">

**Author:** ![seadoo](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@seadoo](https://discourse.julialang.org/u/seadoo)\
**Post date:** [May 23, 2024, 8:14pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/5 "2024-05-23T20:14:29Z")

</div>

Interesting! Yes I kept rereading the doc line about the filter macro, if it returns true then the element is retained, I would not have considered doing col == True, but nice to see that something works!

---

<div class="post-metadata">

**Author:** ![seadoo](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@seadoo](https://discourse.julialang.org/u/seadoo)\
**Post date:** [May 23, 2024, 8:17pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/6 "2024-05-23T20:17:22Z")

</div>

So I guess by that logic none of the `missing` values are `missing` according to Query? What’s strange is that I’ve done a decent amount of manipulation on dataframes containing missing values and the instances of `missing` have always been retained, but perhaps that’s because by collecting back into a DataFrame we never see the intermediate `NA` type?

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [May 23, 2024, 8:36pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/7 "2024-05-23T20:36:39Z")

</div>

Here is a quick check that indeed `ismissing` is called on a `DataValue`:

```julia-repl
julia> using TraceFuns

julia> @trace (example |> @filter(ismissing(_.prct)) |> DataFrame) ismissing
                                       10: ismissing(DataValue{Float64}(0.2)) -- Method ismissing(x) @ Base essentials.jl:1010 of ismissing
                                       10: ismissing(DataValue{Float64}(0.2)) -> false
                                       10: ismissing(DataValue{Float64}(1.0)) -- Method ismissing(x) @ Base essentials.jl:1010 of ismissing
                                       10: ismissing(DataValue{Float64}(1.0)) -> false
                                       10: ismissing(DataValue{Float64}()) -- Method ismissing(x) @ Base essentials.jl:1010 of ismissing
                                       10: ismissing(DataValue{Float64}()) -> false
                                       10: ismissing(DataValue{Float64}(10.0)) -- Method ismissing(x) @ Base essentials.jl:1010 of ismissing
                                       10: ismissing(DataValue{Float64}(10.0)) -> false
0×3 DataFrame
 Row │ alph name prct     
     │ String String Float64? 
─────┴──────────────────────────

```

Imho, there are two issues with this:

1. `Query` operators change the type of data columns and I can just hope that operators forward to `DataValue{T}` as expected
2. Missing values are treated differently – which is one of the motivations for DataValues as far as I understand – an in particular `ismissing` is _not_ forwarded as expected.  
Would it be good practice to define?

```julia
 ismissing(::DataValue) = error("ismissing called on DataValue. This is most likely not what you want, consider calling isna instead")

```

PS: Just found another example where DataValue’s wrapping might be problematic:

```julia-repl
julia> 1.2 isa Real
true

julia> DataValue(1.2) isa Real
false

```

Note that this is pervasive in also effecting dispatch!

---

<div class="post-metadata">

**Author:** ![Satvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/satvik/32/20486_2.png) [@Satvik](https://discourse.julialang.org/u/Satvik)\
**Post date:** [May 23, 2024, 8:41pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/8 "2024-05-23T20:41:28Z")

</div>

It sounds like QueryVerse diverges from the rest of the Julia DataFrames ecosystem in some ways, to provide semantics that work with a broader set of table-like structures. One of those ways is using a replacement for missing. Unfortunately, that results in situations like this, where `ismissing` silently returns false

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 23, 2024, 8:47pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/9 "2024-05-23T20:47:13Z")

</div>

I think most things have been said above, but just to confirm: `Union{Missing,T}` columns in tables are converted to `DataValue{T}` inside the query, and then converted back to `Union{Missing,T}` if you materialize into say a `DataFrame` at the end. So inside the query you need to use things like `isna` etc.

The biggest difference between `Missing` and `DataValue` is probably that traditional predicates stay predicates with `DataValue`, i.e. something like `==` will always return a `Bool`.

There is a bit of docs at [Getting Started · Query.jl](https://www.queryverse.org/Query.jl/stable/gettingstarted/#Missing-values-1) and [GitHub - queryverse/DataValues.jl: Missing values for julia](https://github.com/queryverse/DataValues.jl).

---

<div class="post-metadata">

**Author:** ![seadoo](https://avatars.discourse-cdn.com/v4/letter/s/90ced4/32.png) [@seadoo](https://discourse.julialang.org/u/seadoo)\
**Post date:** [May 23, 2024, 9:04pm UTC](https://discourse.julialang.org/t/basic-function-usage-in-query-filter/114642/10 "2024-05-23T21:04:00Z")

</div>

Thanks! I can’t really tell how to interact with DataValues.jl (even with Query loaded), so it might be beneficial for something like this to be added to the docs for Query, even if it’s as minor as a suggested replacement function for `ismissing`, etc.

I really like the utilization of named tuples as a behind-the-scenes data structure and I really like the way macros work in Query!

However, in the pursuit of consistency, I don’t see where the behavior of `missing` could be improved upon - anything to do with `missing` should always return `missing`, other than `ismissing`, right?

Maybe DataValues.jl solves problems that existed in prior version of Julia that have now been addressed?