Operating on large TIFF stacks without loading whole thing into memory?

General Usage
, ,
1

I’m doing some basic image processing on large .tif stacks. For instance, taking the average intensity of each frame.

Currently the step that takes by far the longest is loading the stack into memory using:

stack = load("file.tif")

I feel there must be a better way to do this that doesn’t involve needing to load the whole (Gb’s worth) thing at once.

Thanks

2

OMETIFF.jl supports mmapping to do exactly what you want

1 Like
3

Thanks!

I’m not sure I can see anything in the readme about implementing mmapping.

How is this best done for the simplest case?

4

Check the docs. There’s a kwarg for load that controls whether the full data is loaded from disk or just that which is being accessed
https://tamasnagy.com/OMETIFF.jl/stable/lib/internals/#OMETIFF.load

I say all this having not used it myself.

2 Likes
5

Thanks, I’ll have a play around.

@tlnagy is this entirely specific to OMETIFFs (would probably make sense given the name), or does it extend to general TIFF images?

6

Thanks @ianshmean for linking the docs to OMETIFF.jl

@tlnagy is this entirely specific to OMETIFFs (would probably make sense given the name), or does it extend to general TIFF images?

@ajsc4 There is nothing specific to OME-TIFFs about my memory mapping code here. I simply create a new type that loads in the plane that you need when you fire off a getindex call. As an easy workaround, you can convert your TIFFs to OME-TIFFs using ImageJ or some other tool and OMETIFF.jl should work. Word of warning though, OME-TIFF supports a limited subset of TIFF files.

My longer term plan is to transition to the architecture I’m developing in my TIFF.jl repo which is much more flexible and supports BigTIFFs, compression, and a lot of other TIFF weirdities.

I was planning to refactor OMETIFF.jl and move the mmap code over to TIFF.jl and have OMETIFF.jl simply be a wrapper that adds some features from the metadata. I haven’t had time to do this port myself yet (blame 2020), but I would be happy to accept a PR over at TIFF.jl.

2 Likes
7

An alternative approach might be to wrap your tiff stack into a DiskArray from DiskArrays.jl, which might be useful if memory-mapping does not work because of compression or other difficulties. That way you get an AbstractArray with indexing, broadcasting and reductions optimized for slow random-access. Let me know if you want to try and face problems.

1 Like
8

@ajsc4 As a follow up, the first version of my package TiffImages.jl is out on the general registry. It supports opening most common TIFF types (including BigTIFFs, HDR formats, etc) and can load them out-of-memory so it should do what you need. I’m actively working on making it more versatile.

Here’s a link to the documentation about how to read/write TIFFs: Home · TiffImages.jl

It’s a super fresh package so please open issues on the github page if you run into any limitations. I test it thoroughly on many different TIFF types, but it’s not called Thousands of Incompatible File Formats without reason :sweat_smile:

4 Likes