Loop performance issue for a specific data size

ReWoP
Fri, 01 Dec 2023 21:36:31 +0000
General Usage

Hello everyone!

I’m testing a population-based algorithm and have encountered performance issues, particularly when looping with large populations (i.e., using large data matrices).

Everything works fine when using matrix sizes of 15000x1500 and 20000x2000 and smaller. However, upon increasing the matrix size to 25000x2500, I notice a significant performance drop around iteration number 810 (I’m using 1000 max iterations). At this moment, the algorithm starts taking more than 10 times longer per iteration even after reaching the second run of the same algorithm.

At first, I suspected it might be due to suboptimal code on my part, potentially causing issues in the GC. I even tested adding a GC.gc() before initiating the second run, but it didn’t have any effect on the performance.

Strangely, upon running the algorithm with an even larger population of 30000x3000 and 50000x5000, I observed no performance drop throughout the run.

It is not related to lack of memory, as my system has about 50% RAM free during the computation.

I’m using Julia version 1.9.4 and I’ve observed this behavior on my desktop with a AMD CPU and in my laptop with an Intel CPU.

Bellow, I attach code with a minimal reproducible example and some logs from test done with my AMD system.
MRE code (click for more details) Log - Run with 25000x2500 (with performance issues) (click for more details) Log - Run with 30000x3000 (OK) (click for more details) Log - Run with 50000x5000 (OK) (click for more details)
3 posts - 2 participants