Is it possible for you to share the matrix? I think one of the most efficient ways would be to compute the action of the inverse of the shifted matrix (A-\sigma I)^{-1} on a vector x, rather than including the entire matrix inside a LinearMap (this is explained in this post),
If you can share the matrix, I can try to give you an example for this case since I have been doing something similar these past weeks.