I have been trying to tell people about this for a while. Julia 1.11 is going to be slower than Julia 1.10 for a number of reasons. There are a lot of big changes landing in this release. Besides the Memory change, more packages have been kicked out of the system image.
If you a performance sensitive user, Julia 1.10 might be your version for a while as the ecosystem adapts to these changes.