Processus stop for large array of images

General Usage Performance
,
1

Hello, here is my code :

using Images
function listOfImg(img::Array{Gray{Normed{UInt8,8}},2})
	lengthImgSplit = 1000
	nbImgToSave = 3000
	imgSplit = similar(img, size(img, 1), lengthImgSplit)
	listOfImg = Array{typeof(imgSplit),1}(undef, nbImgToSave)
	for i in 1:nbImgToSave
		listOfImg[i] = img[:, i:(i+lengthImgSplit)]
	end
	return listOfImg
end

img = 338×45526 Array{Gray{N0f8},2} with eltype Gray{Normed{UInt8,8}}
My problem is that my computer stopped when nbImgToSave > 3000
When nbImgToSave = 1000, the processus finish in 1.1s, with 2000, it is 2.1s (29.71 k allocations: 647.112 MiB, 18.16% gc time), with 3000, 12.5s (31.71 k allocations: 1.892 GiB, 23.11% gc time) and with 4000 my computer stopped … I think, one part of the problem is because i’ve an old computer (2G of Ram and a SSD of 16G with only 3G free). But i wish to split the entire image (so nbImgToSave ~ 44000) and i want to know if there is a way for this with my material ?

2

With a 338x45526 img, and nbImgToSave = 3000 and lengthImgSplit = 1000, then each element of listOfImg is an array of size 338x1000, and you have 3000 of them. That’s about 1Gb of memory just to hold all those split images, so it’s no wonder your computer is having trouble.

I think the fundamental problem is that you are copying the 338x1000 pieces of img each time you do img[:, i:(1:lengthImgSplit)]. But there may not be any need to do that–as long as you don’t need to modify the elements of listOfImg separately, you can instead just tell Julia to record a view of each part of the image. That doesn’t require copying any data at all, and it should decrease your memory usage by almost 100%.

For example:

function listOfImg2(img::Array{Gray{Normed{UInt8,8}},2})
	lengthImgSplit = 1000
	nbImgToSave = 3000
	listOfImg = [@view(img[:, i:(i + lengthImgSplit)]) for i in 1:nbImgToSave]
	return listOfImg
end

Let’s compare performance:

julia> using BenchmarkTools;

julia> img = rand(Gray{N0f8}, 338, 45526);

julia> @btime listOfImg($img)
  581.718 ms (6004 allocations: 968.56 MiB)

julia> @btime listOfImg2($img);
  11.092 μs (2 allocations: 140.67 KiB)

listOfImg2 is more than 50,000 times faster, and it uses about 7,000 times less memory!

The only thing to be aware of is that with listOfImg2, each element of the result points to the same memory inside the original img. So if you mutate img or mutate any of the elements in the list of slices, then they will all change.

3 Likes
3

Thanks a lot, it solves my problem. I will study more deeply the way julia is running so i can benefit of its performances … You are right, the memory usage decrease by almost 100% :grinning: