Your code does have side-effects! Just maybe not in the way you typically think of it. It uses memory, and lots of it. You’re doing hundreds of millions of allocations, 8GBs worth of it! Memory — and memory bandwidth — is a shared resource. As is the GC.
It’s not just the matrix itself, but each and every single BigInt is independently allocated. Worse, every computation with a BigInt throws it away and allocates a new one. They’re not the fastest datatype in the world, that’s for sure.
Note that the % GC time doesn’t count time that the threads wait for a GC to happen. Every time you allocate a new object, the GC will check the “memory pressure” to see if it needs to do a collection. If it does, it’ll raise a flag and wait for all the threads to hit a “safe point,” at which point marking (multithreaded) and sweeping (single-threaded) is done. That waiting time isn’t accounted for in the metrics that are printed out but may be where some of this is going.