# Should I use HDF5 to speed up reading image data repeatedly?

**URL:** <https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576>\
**Category:** Performance\
**Created:** [March 19, 2021, 11:26pm UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576 "2021-03-19T23:26:33Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kimlaberinto](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kimlaberinto/32/21534_2.png) [@kimlaberinto](https://discourse.julialang.org/u/kimlaberinto)\
**Post date:** [March 19, 2021, 11:26pm UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/1 "2021-03-19T23:26:33Z")

</div>

Hi all,

I’m trying to speed-up the following piece of code for an art project.  
I have a collection of png images. All of them together is 25GB+

And in a loop, I need to keep copying an arbitrary pixel in one of the png images (simplified code below), right now it takes running overnight (~9 hours) to complete even with some preloading of images as much as I can.

How can I do this much faster possibly by preprocessing the images into a faster to access format, or possibly accessing the desired pixel without loading the entire image?

EDIT: Right now I’m also thinking about preprocessing it all into HDF5 and maybe that speeds it up

Here’s the basic code:

```julia
for frame = 1:Ntotalframes
    newimg = copy(preloaded_img_of_proper_dims)
    for j = 1:ysize
        for i = 1:xsize
            # Access and copy a single pixel in one of the png images in the 25GB collection
            file_get_to_from, pixel_coords_in_file = getDesiredFilenameAndPixelIndex(i, j, frame)
            newimg[i, j] = getpixelvalue(file_get_to_from, pixel_coords_in_file)
            
            # Basically I'm doing the code below (but with some preloading). How should I speed it up?
            # I can't preload preload entire 25 GB+ images into RAM
            # getpixelvalue = load(file_get_to_from)[pixel_coords_in_file[1], pixel_coords_in_file[2]]
        end
    end
    newimg_file_name = @sprintf("%04d.png", frame);
    save(newimg_file_name, newimg)
end

```

Thank you,  
Kim

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [March 20, 2021, 12:05am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/2 "2021-03-20T00:05:37Z")

</div>

Are you reading one single picture `ysize * xsize` times?

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [March 20, 2021, 12:08am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/3 "2021-03-20T00:08:15Z")

</div>

Can you share more information about the order in which pixels/files are retrieved? Memory-mapping or chunking (with HDF5) would be faster than loading the whole PNG, of course, but if the frame order isn’t completely random, you can keep a large number of images in memory (maybe with [LRUCache.jl](https://github.com/JuliaCollections/LRUCache.jl)) to cut down on storage-thrashing.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [March 20, 2021, 12:20am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/4 "2021-03-20T00:20:38Z")

</div>

You can benchmark your code on a small number of pictures with `Juno.@profiler` in Atom or `@profview` in vscode. There can be bottlenecks in different places, like reading from disk or decoding picture format, but I think the main problem is that nested loop - you should keep it as small as possible, with all data already read from disk outside the loop. Or maybe rearrange cycle order - first find all pixels to change a single file, and then write file in one operation.

---

<div class="post-metadata">

**Author:** ![kimlaberinto](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kimlaberinto/32/21534_2.png) [@kimlaberinto](https://discourse.julialang.org/u/kimlaberinto)\
**Post date:** [March 20, 2021, 12:50am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/5 "2021-03-20T00:50:24Z")

</div>

> Are you reading one single picture ysize \* xsize times?

I’m reading different images from all over the 25GB collection and collecting them into one image.

> `@profview` in vscode.

Thank you! I’ll try this benchmarking macro.

> I think the main problem is that nested loop - you should keep it as small as possible, with all data already read from disk outside the loop.

That is what I was thinking too. I tried preloading all the potential images at the start of the `frame` iteration before the `for j, i` loop, but it still took really long.

---

<div class="post-metadata">

**Author:** ![kimlaberinto](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kimlaberinto/32/21534_2.png) [@kimlaberinto](https://discourse.julialang.org/u/kimlaberinto)\
**Post date:** [March 20, 2021, 12:54am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/6 "2021-03-20T00:54:47Z")

</div>

> Can you share more information about the order in which pixels/files are retrieved?

It’s for an art project, so there’s no consistent way I’m grabbing the images from the 25GB+ collection.  
Technically inside that inner-most loop, it could be any one of the images from the 25GB+ collection.  
(Then I change the parameters, and see what ends up with a result that I like)

> Memory-mapping or chunking (with HDF5) would be faster than loading the whole PNG,

Yeah I’m trying out this method now and seeing how well it performs since accessing a pixel from the 25GB+ is pretty arbitrary and seems like memory-mapping with HDF5 might be the best way to go right now to be able to access anywhere in the 25GB+ image collection fast (hopefully).

> but if the frame order isn’t completely random, you can keep a large number of images in memory (maybe with LRUCache.jl) to cut down on storage-thrashing.

Thank you! I didn’t know this was a possible technique, I’ll keep this in mind for other projects

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [March 20, 2021, 1:03am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/7 "2021-03-20T01:03:54Z")

</div>

With GDAL you can read only a little chunk of an image (PNG or other formats)

---

<div class="post-metadata">

**Author:** ![aramirezreyes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aramirezreyes/32/42573_2.png) [@aramirezreyes](https://discourse.julialang.org/u/aramirezreyes)\
**Post date:** [March 20, 2021, 2:27am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/8 "2021-03-20T02:27:35Z")

</div>

One thing that looks expensive is that you allocate one array for each iteration when you do

```julia
newimg = copy(preloaded_img_of_proper_dims)

```

Have you tried preallocating just one newimg outside of your function and reusing the memory? That could save you some garbage-collection time if the number of frames is large.

Other than the regular things (putting things inside funcions, avoiding globals, etc) I wonder if you can paralellize some portion of the code with coroutines, or using GDAL as @joa-quim suggests.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [March 20, 2021, 2:42am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/9 "2021-03-20T02:42:05Z")

</div>

With HDF5 you can save arrays in chunks. So only one chunk would be read when you accessing one pixel. And that chunk would be cached as long as file is open. So you could try loading the whole collection into one HDF5 file in different datasets with chunking.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [March 20, 2021, 2:47am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/10 "2021-03-20T02:47:20Z")

</div>

I would set chunks length from very small to higher values and check timings for each. Like: 8x8, 16x16, 32x32, and so on. (But don’t use compression for chunks.)

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [March 20, 2021, 4:05am UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/11 "2021-03-20T04:05:52Z")

</div>

Stupid question from me… Clearly getting an efficient file format and choosing your data layout is the way forward - and is good experience for other projects.

However you could be lazy - why not hire a large memory cloud instance?

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [March 29, 2021, 11:52pm UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/12 "2021-03-29T23:52:09Z")

</div>

> [@kimlaberinto](#):
>
> It’s for an art project, so there’s no consistent way I’m grabbing the images from the 25GB+ collection.  
> Technically inside that inner-most loop, it could be any one of the images from the 25GB+ collection.  
> (Then I change the parameters, and see what ends up with a result that I like)

I think you may be able to avoid loading an imagine multiple times per run. In the inner x,y loop, just compute which pixels you need from each image. Try storing the results as a Dict mapping image filenames to arrays of pixel coordinates. Then after you’ve computed which pixels come from which image, you should loop over the image files, reading each image only once, and extract the pixels from each one.

---

<div class="post-metadata">

**Author:** ![uwbanjoman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uwbanjoman/32/9998_2.png) [@uwbanjoman](https://discourse.julialang.org/u/uwbanjoman)\
**Post date:** [July 22, 2021, 6:45pm UTC](https://discourse.julialang.org/t/should-i-use-hdf5-to-speed-up-reading-image-data-repeatedly/57576/13 "2021-07-22T18:45:46Z")

</div>

maybe you should look into the Parquet file format as well, you can just read / write the part you need to change if my understanding is correct.