zram might help, see: Building Julia on a Raspberry Pi - #5 by Oscar_Smith
And: examples/how_to_enable_ZRAM_Raspberry_Pi.md at master · garyexplains/examples · GitHub
And a larger swap file: How to Change Swap File Size in Ubuntu – TecAdmin
And perhaps disable the option to build the image using multiple threads.
I was able to tune my system to be able to build my system image within 45 minutes… Still a long time. Next step would be to use a pci SSD to store the swap file to see if that can speed things up.