Processus stop for large array of images

General Usage Performance
,
2

With a 338x45526 img, and nbImgToSave = 3000 and lengthImgSplit = 1000, then each element of listOfImg is an array of size 338x1000, and you have 3000 of them. That’s about 1Gb of memory just to hold all those split images, so it’s no wonder your computer is having trouble.

I think the fundamental problem is that you are copying the 338x1000 pieces of img each time you do img[:, i:(1:lengthImgSplit)]. But there may not be any need to do that–as long as you don’t need to modify the elements of listOfImg separately, you can instead just tell Julia to record a view of each part of the image. That doesn’t require copying any data at all, and it should decrease your memory usage by almost 100%.

For example:

function listOfImg2(img::Array{Gray{Normed{UInt8,8}},2})
	lengthImgSplit = 1000
	nbImgToSave = 3000
	listOfImg = [@view(img[:, i:(i + lengthImgSplit)]) for i in 1:nbImgToSave]
	return listOfImg
end

Let’s compare performance:

julia> using BenchmarkTools;

julia> img = rand(Gray{N0f8}, 338, 45526);

julia> @btime listOfImg($img)
  581.718 ms (6004 allocations: 968.56 MiB)

julia> @btime listOfImg2($img);
  11.092 μs (2 allocations: 140.67 KiB)

listOfImg2 is more than 50,000 times faster, and it uses about 7,000 times less memory!

The only thing to be aware of is that with listOfImg2, each element of the result points to the same memory inside the original img. So if you mutate img or mutate any of the elements in the list of slices, then they will all change.

3 Likes