countmap uses Dict internals to avoid hashing each column twice (working around julia#24454), and in this problem hashing the columns is the dominant cost for a sufficiently large matrix.
(Indeed, since hashing is the dominant cost, you could probably speed things up further by wrapping the column views in another type that implements a custom hash function. Standard numeric hashing in Julia pays an extra cost so that different types of the same value, e.g. 1.0 vs 1 vs 1.0f0, yield the same hash. For this application you don’t need that, assuming that all of the matrix elements have the same type, i.e. the eltype is concrete. But this kind of voodoo might not be worth the trouble here.)