That won’t work, because it has the wrong units. (The units of the determinant of an m \times m matrix are the matrix elements to the m'th power.) An absolute tolerance has to have the same units as the quantity you are comparing it to.
Also maximum(A) returns the largest element algebraically, whereas I think you want the largest magnitude.
The “right” way to determine whether a matrix is close to singular is to compute its condition number, which usually is computed via the SVD, though you could in principle also use norm(A)*norm(inv(A)) (except that inv might throw). A nearly as good way is to use rank-revealing QR.
But for a 3 \times 3 matrix, checking that det(A) ≤ rtol * maximum(abs ∘ float, A)^size(A,1) should be fairly reasonable, I guess (without thinking about it too hard), for some decent tolerance, e.g. rtol = sqrt(eps(float(eltype(A)))).
(Determinants can also spuriously overflow/underflow because they can grow exponentially with matrix size. But for 3x3 matrices with reasonable-magnitude entries it shouldn’t be an issue.)