For example, if an algorithm can only be used on square matrices, which error should I throw?

function square_matrix_algorithm(A::AbstractMatrix)
if size(A, 1) != size(A, 2)
throw(ArgumentError("This function can only be used on square matrices!"))
# Or
throw(DomainError(A, "This function can only be used on square matrices!"))
end
...
end

ArgumentError is documented vaguely because itâ€™s used for quite a few things, like the wrong number or type of arguments. Arity and type can also be handled with method dispatch and absent methods (MethodError), but sometimes things are written with branches in one method.

DomainError more specifically deals with a subset of a Julia typeâ€™s instances being invalid in a particular context, like negative reals for sqrt. In isolation, it does technically apply here, but there is an even more specific DimensionMismatch for square matrices.

julia> using LinearAlgebra
julia> det([0 1
2 3]) # determinant defined for square matrices
-2.0
julia> det([0 1 10
2 3 20])
ERROR: DimensionMismatch: matrix is not square: dimensions are (2, 3)

As a general rule, the more specific error is proper, but in practice, check similar functions in Base, the standard library, or popular third-party packages, in that order.

function matmul3x3!(C::AbstractMatrix, tA, tB, A::AbstractMatrix, B::AbstractMatrix,
_add::MulAddMul = MulAddMul())
require_one_based_indexing(C, A, B)
if !(size(A) == size(B) == size(C) == (3,3))
throw(DimensionMismatch(lazy"A has size $(size(A)), B has size $(size(B)), C has size $(size(C))"))
end
...

The == (3,3) right after it in the same line is also a critical trigger of the DimensionMisMatch, and the function name includes 3x3 to indicate its intent. Note that det only takes 1 matrix, so â€śmismatchâ€ť doesnâ€™t inherently compare multiple matrices, it refers to any unsuitable dimensions.

That seems right to me, and Base has the same check.