ArgumentError is documented vaguely because it’s used for quite a few things, like the wrong number or type of arguments. Arity and type can also be handled with method dispatch and absent methods (MethodError), but sometimes things are written with branches in one method.
DomainError more specifically deals with a subset of a Julia type’s instances being invalid in a particular context, like negative reals for sqrt. In isolation, it does technically apply here, but there is an even more specific DimensionMismatch for square matrices.
julia> using LinearAlgebra
julia> det([0 1
2 3]) # determinant defined for square matrices
-2.0
julia> det([0 1 10
2 3 20])
ERROR: DimensionMismatch: matrix is not square: dimensions are (2, 3)
As a general rule, the more specific error is proper, but in practice, check similar functions in Base, the standard library, or popular third-party packages, in that order.