This doesn’t mean what you think: the | operator has lower precedence than !=, so your expression is equivalent to a != (0 | b) != 0.
And since 0 | b is just b for any integer b, this means that what you have is effectively a != b != 0.
What do you mean? I get:
julia> subset(df , [:x, :y] => ByRow((a,b)-> a !=0 || b !=0))
3×4 DataFrame
Row │ x y z w
│ Int64 Int64 String String
─────┼──────────────────────────────
1 │ 2 0 A A
2 │ 8 0 C B
3 │ 9 8 D B
which looks correct to me.
It sounds like what you want is a != 0 && b != 0, not ||?