I’ve considered myself a Julian for about 6 months now. I have enough experience to know that a < b & c < d is parsed as a < (b & c) < d and not as most would probably expect. And yet this bug (see comment below) bit me hard today.
This illustrates the problem, although it is very unlike my actual code:
x = collect(1:10)
a = x[x.%3 .== 1 .& iseven.(x)] # wrong
b = x[(x.%3 .== 1) .& iseven.(x)] # parens are required because of precedence of &
println(a) # [3, 4, 9, 10]
println(b) # [4, 10]
I think I was so focused on getting all the broadcast dots in place in my vectorized code that I forgot about the bitwise operator precedence, even though I use it correctly elsewhere. The problematic line looks so RIGHT that I failed to notice the error even after scanning it multiple times.
But the main problem is this: when you make this error, your code fails silently.
This is a massive gotcha, especially for people with a Matlab background who do this routinely. I understand that breaking syntax changes are very unwelcome so close to v1.0, but I really think this needs to be addressed somehow. The current precedences are arguably correct for bitwise operators, but are bugged when used as logical operators. So maybe we need to distinguish between these uses?
There is a longstanding issue (#5187) in which this is discussed in detail. There are many interesting ideas there, but it was eventually closed without a resolution. Is there still hope of doing something about it?
I think that #5187 explains that the precedence of & is right most of the time. And it’s not at all obvious to me why < should have higher precedence.
The main exception, I guess, is your use case—constructing a logical vector for indexing into an array—and, frankly, you should not be doing that anyway. You’ll be constructing a redundant intermediate array (in your case doubly so, because of collect) and you miss out on the benefits of short-circuiting, which can be really bad if the operations are more expensive than % and iseven.
I strongly disagree. Constructing logical masks is a very common and well-supported idiom. It’s not surprising to want to condition a logical mask on more than one comparison. Since all the dot-operations fuse, you’re only constructing one temporary array. And note that logical masks have an advantage over other collections of integers since Julia can simply check the size of the mask instead of checking bounds for each and every element. In fact, it doesn’t take all that many elements for the logical mask to both allocate less (since it doesn’t need to dynamically grow its result) and be faster:
julia> f(x) = [i for i in x if iseven(i) && i % 3 == 1]
g(x) = x[(x.%3 .== 1) .& iseven.(x)]
g (generic function with 1 method)
julia> @btime f($(1:10000));
50.776 μs (15 allocations: 32.73 KiB)
julia> @btime g($(1:10000));
47.223 μs (9 allocations: 18.84 KiB)
I second @DNF: it simply isn’t obvious that < should precede &. In fact, I can only justify an argument to the contrary. Indeed, in all other cases that I can think of, binary operators take precedence over binary relations. For example, how would you read a \le x + y \le b? How would you read a \le xy \le b. In fact, & is simply multiplication on \mathbb{Z}_{2} , so it doesn’t make a whole lot of sense for & and * to have different precedence.
< preceding & might make sense in languages that don’t support multi-sided inequalities, but in Julia it seems highly dubious.
I’ll be damned… I thought that the array was growing in some clever way (e.g. extending the length by 1.x and avoiding bounds checking because the intermediate lengths are known [but I don’t quite understand bounds checking].) I’m certainly surprised it’s more costly than creating an intermediate array fully the length of the original array.
And if the resulting array is much shorter than the original and intermediate arrays, that counts the other way, and also the short circuiting.
Yes, the filtered comprehension does grow the array exponentially (I believe it’s 2x every time). But every time it runs out of space, it needs to allocate that new chunk and go back to copy all the existing elements over before continuing. The exponential growth amortizes the cost, but it’s still a cost. Worse: it’s less predictable in its instructions and memory use.
The logical mask, on the other hand, always reads through the entire array three times. Once to construct the mask, once to count the number of true elements, and once to do the indexing. While this may seem expensive, it’s actually very easy for the computer to guess where the next memory access will be. You can see that it has an even bigger advantage with less-predictable elements:
Back on topic, I’d love to see a solution that makes comparisons with logical masks easier to use and less error-prone. The conclusion of #5187 was to keep the status quo, but that was a long time ago.
Interestingly, dot-broadcast fusion actually gives .&& a sensible meaning and a possible new solution. I wonder if there’d be more support for it now.
Since we were discussing the precedence of &, I would read that as (a \le x + y) \land (x + y \le b), and pray to all higher powers that noone would consider reading that as a \le ((x + y) \land (x + y)) \le b.
That argument is fine for & as a bitwise operator, and there I agree. But my point is that & as a logical operator should have a precedence similar to &&. We want our syntax to be clear and simple, especially for the code constructs we often write. And for me personally and in the public Julia code I’ve looked at, vectorized logic is used much more frequently than bitwise operiations.
I also think the whole “X is just Y in mathematical terms”-argument is dangerous. It can lead you to do crazy things like, I don’t know, choosing * for string concatenation instead of +.
I don’t see why this is crazy. I think that “X is just Y in mathematical terms” should be the default design philosophy, and I think so far that is mostly the case in Julia. The truth is programming languages are plagued with a legacy coming from such unholy monstrosities as COBOL (as an extreme example), and it would be nice to break with that tradition. (Though I don’t mean to imply that this particular suggestion is crazy.)
The dotted .&& and .|| forms make sense to me for this, although it’s a bit odd that they’d just be functions. The dual nature of & and | as bitwise and logical operators is fairly problematic here. At the very least it seems like we should change the precedence of ∨ and ∧ and other operators that could be strictly logical so that a < b ∨ c < d parses as (a < b) ∨ (c < d), and correspondingly for @. a < b ∨ c < d.
I remember reading a joke somewhere that everyone should be allowed to make one breaking change to Julia. These would be mine:
and, or would be the short-circuiting control flow operators. These would replace the current &&, ||.
&&, || would be logical operators implemented as ordinary functions, i.e. they would broadcast as any other function. They would have the same precedence as they have now.
&, | would remain just as they are, i.e. bitwise operators implemented as ordinary functions.
Advantages:
bitwise & logical operators would be separated, each with appropriate precedence
the dotted .&& and .|| forms would be completely natural, not odd at all
and/or are words, so they would be visually similar to other keywords for control flow
Reviving this thread to ask the devs a question. Jeff began addressing this problem by deprecating the precedence of & and | (see near the end of #5187). It seems the intent was to change the precedence of & and | to match && and || later. But the issue appears to have been closed without merging his changes. So is this now officially dead in the water for 0.7/1.0? If so, what happened? I can’t tell from the issue discussion why it was closed in the end, unless the idea is to introduce .&& and .|| post 1.0. Is that it?
That was an experiment and there was a consensus after discussing the result that we all—somewhat surprisingly—didn’t really like it. I don’t think a single person who discussed it on slack or on triage actually liked the change, so we just dropped it. It would also be wildly breaking and disruptive, so unless there was universal love for a change like that, it’s really not worth it. Introducing .&& and .|| definitely will happen although I’m not sure when. It may be easier now that @mbauman’s lazy broadcast infrastructure has been merged.
We needed to wait a cycle for the deprecation of && and || expressions within @. to take effect. That’s now in 0.7, so we’re set to introduce them in 1.x.
Could you please explain the logic behind this schedule?
AFAIK in 0.6 broadcasting over && and || produces an error.
In 0.7 it gives a deprecation warning - it is not clear for me why. It was not allowed earlier so why do we say in the warning using || expressions in @. is deprecated in 0.7 (the code of _dot_ actually only changed that it produces a warning)?
My understanding was that since it produced an error in 0.6 we could safely let it in even in 0.7 (as this is a new feature) or in 1.0 (since in 0.7 we give a warning clearly saying that in the future it will become elementwise - so it would be natural to expect it in 1.0).
julia> false .&& [1,2,3]
ERROR: unsupported or misplaced expression &
julia> @. false && [1, 2, 3]
┌ Warning: using && expressions in @. is deprecated; in the future it will
│ become elementwise. Break the expression into smaller parts instead.
│ caller = ip:0x0
└ @ Core :-1
false
julia> broadcast((x,y)->x&&y, false, [1,2,3])
3-element BitArray{1}:
false
false
false
When && becomes dottable, folks will expect @. to dot it. In 1.0 or 1.1 we can make it behave like the broadcast of the anonymous function.
Thank you for a clarification. I always considered only vectors on LHS and RHS and I missed that one of them could be a scalar. Given this I would call:
a bug .
Anyway - the sooner we can have && and || broadcastable the better.
IMHO in 1.0 should be acceptable as in 0.7 everyone will get a deprecation with a clear message what will be the behavior in the future.
The practical argument for 1.0 is that I expect that with Julia 1.0 new people will consider testing it and usage of &&/|| would be one of the things that you hit when moving from other languages very early on.
If so, what happened? I can’t tell from the issue discussion why it was closed in the end, unless the idea is to introduce 
.