In order to calculate max(A, A') it must be possible to say which one is ‘greater’, and isless helps with that comparison.
But what is greater of A and A'? I don’t think there is any obvious answer to that question.
My guess is that you probably want to compare element-by-element, and to do that you can use broadcasting:
max.(A, A')
Notice the dot after max. This should return a matrix with pairwise comparisons of all elements.
Take a look at the broadcasting section of the manual for more information: Multi-dimensional Arrays · The Julia Language