Probably you already figured this out, but if you rewrite the code entirely with scalar ops, then your won’t require any memory allocation and deallocation. There may be some loss of performance because of poorer use of memory hierarchy than built-in matrix-multiplication, but I think overall it will be a win since your matrices and vectors are not so large. Explicit loops reduce the readability of the code, but this can be partly hidden using macro packages like devectorize or einsum.
In my finite element code, I use explicit loops and scalar ops for all dense matrix-vector operations on smaller matrices and vectors. I use matrix/vector ops for operations on big matrices like the global stiffness matrix. I’m reasonably happy with the performance.