max() returns the maximum of its arguments. That is, max(a, b, c) returns whichever of a, b or c is greatest. It then follows that whatever is returned by max(a, b, c) must be one of a, b, or c. In your case, the answer is b, which is exactly what is returned.
max.(a, b, c) is the elementwise maximum of the three inputs, so it gives a different answer.
This has nothing to do with max itself–it just uses the result of the < function. The lexicographic ordering comes from the definition of < (which by default falls back to the function isless) for abstract vectors. I don’t have a link to the relevant part of the documentation handy, but that’s where you’d want to look.