You should not, in general, expect broadcast and map to have the same behavior, they are named differently for a reason.
The basic philosophy (as I understand it) is that map is for general iterable collections while broadcast is for objects with an AbstractArray-like behavior. In light of this, the reason you shouldn’t necessarily expect broadcast to work for Set is because sets are collections with no ordering.
As an example:
x = [1, 2, 3]
y = [4, 5, 6]
x .* y # returns [4, 10, 18]
In this operation ordering was clearly very important: it only makes sense because x and y have a well-defined order. Meanwhile, b .* y (with b the set you defined above) would be undefined because the ordering of b is not defined.