[ANN] BayeredImages.jl – working with raw color images in Julia

Package Announcements
,
1

BayeredImages.jl (currently unregistered)

This announcement is somewhat premature, but I realized there’s a gap in the Julia image ecosystem that I’m surprised has not been filled already.

The problem

Color cameras use color filter arrays, like the Bayer matrix, or if you’re weird like Fujifilm, the X-Trans matrix. The raw image is really black and white; demosaicing algorithms convert this filtered image to a color image with RGB values at every pixel.

Depending on what kind of image processing you want to do, you might prefer to manipulate the raw image data before demosaicing. A great example of a kind of manipulation which would be useful to do before demosaicing is filtering hot pixels. If they’re not removed before the debayering step, the hot pixel’s data will spill into neighboring pixels, leading to artifacts that are annoying to clean up.

A median filter can handle this (especially if you have a map of hot pixels from the sensor), but applying it to raw CFA data needs care, since you probably only want to take the median with respect to pixels of the same color channel. Or perhaps you want to incorporate data from other channels (since the filter bandpasses overlap), but you’ll probably want to use something other than a simple median for this.

Goals

The goal is to provide a package that

  • has types that represent color filter arrays, including efficient representations for the most common CFAs (like the Bayer matrix), and generic representations for wackier/experimental CFAs.
  • has a type to represent mosaiced images, and correctly handles image transforms which may alter how the CFA is associated with the data (rotation, reflection, cropping).
  • provides demosaicing algorithms, including generic ones that work with arbitrary CFAs (but are superior to ordinary interpolation methods).

Additionally, I’d like to provide versions of common image operations (such as the median filter and the Drizzle algorithm) adapted to work with raw color data, but complex algorithms might be better off placed in a separate package or as a package extension for existing image processing tools. I’ll probably include a few simple examples in this package though; I suspect bad pixel mitigation with median filtering would be handy for anyone interested in this package.

Calling all photographers!

It turns out I actually don’t use a color camera for the vast majority of my imaging. I’m an astrophotographer, and I use a monochome camera. I do have access to some raw image data from cameras I’ve borrowed from my university library, but it’s almost entirely astrophotographic data (time lapses, star trails, etc). I’m writing this package as part of an effort to eventually perform all of my astrophotographic processing in Julia, but I want this package to be more broadly usable for anyone using a color camera.

So if you have raw image data you’d be willing to share for internal testing, I’d be happy to use it! (Sadly, Wikimedia Commons does not host raw image data.) Image I/O is not within the scope of this package, but I’m interested in performing tests with raw data formats I don’t currently have access to (anything that’s not an Adobe DNG or Canon *.CR2 and *.CR3, and especially any color camera not using a standard Bayer matrix, like Fujifilm X-Trans cameras), and with non-astronomical subjects.

2

Hi,

I am from a steganography field, where we have a long tradition (20+ years) of using images stored in RAW format, because the development pipeline matters. If you want large number of raw images for testing, you can download them from here
https://alaska.utt.fr/#material
but you need to ask for access. To my knowledge, this is the biggest dataset of raw images that is available for research.

3

For raw files, there’s tons here with CC license :

I also have Libraw’s binding here, I implemented a very basic demsaicing algo for testing but it would be nice to have something more serious :

4

This sounds neat! And I would definitely agree with your sentiment about separate packages for complex algorithms. Since you mentioned drizzle, I imagine that is something that could quite possibly deserve a package all to itself (probably under the JuliaAstro group for long-term stewardship because of its importance for Hubble data reduction).

For other image processing option, it would be worthwhile to see if any of the functions/tools you envision exist or would fit into JuliaImages packages as well. A natual place for a median filter (to use your other example) would probably be ImageFiltering.jl, though they do seem to have a median function already.

5

I definitely need a drizzle package, and I’m surprised one doesn’t exist yet! I’m constrained on time as a PhD student wrapping up, but I will need to make it happen Eventually™ if someone doesn’t get around to it.

I’d say Bayer drizzle is the prime example of an algorithm that would benefit from the tooling I envision for this package, and the goal would be to have an extension for a future Drizzle.jl package to work with Bayered images directly (the results have way less chromatic correlated noise as opposed to drizzling debayered frames).

6

Thanks! Hopefully this has coverage of some of the stranger color filter arrays that some cameras use. I’ve come to learn that this is relatively common in smartphones (e.g. Quad Bayer and Nonacell).

Coming from my amateur astronomy background, there’s a lot of young people just getting started with astrophotography by taking timelapses or shooting afocally through an eyepiece. I hope they are not left out of serious processing because their sensor’s color filter array can’t be demosaiced!

I actually should have looked for a libraw wrapper, because I’m more interested in making complex manipulation of raw data before demosaicing possible than simply implementing file I/O for the mess of raw formats. I may take a weak dependency on your package to expose some of that functionality in a Julian way (and if I do I’d be happy to file issues/PRs on your codebase to help out as I use it).

Right now though, the package currently supports FITS files with the BAYERPAT keyword with the FITSIO.jl extension. A bit of an esoteric format for raw images, but it’s an open format…

7

I use a Fujifilm X-T3 and can transfer some raw files, if you share me a folder I can upload to (message me).

8

Indeed, drizzle is the way to go for Astroimage stacking. There was a PR to Astroalign.jl which had a fully functional stacking algorithm, including drizzle, hot pixel removal and a raw-dataset to test it with. If you want to give it a try, you should still be able to download that commit and install it locally.
But at that point the Astroalign.jl package did not include Ransac in the code, which caused some issues, which is currently being remedied. I can see, if I can update the drizzle part of the code to work again with the newer version of Astroalign.jl.
Anyway, the Bayer-pattern was implemented for 4-pixel patterns only denoted like “RGGB” or alike.

9

Would a drizzle implementation be better off in Astroalign.jl or a separate package? I don’t want to reinvent the wheel, but I’m assuming it would be better to have that be an independent package that Astroalign.jl could provide an extension for (or take a hard dependency).

For a bit more context, I’ve created Project Esitohi, a Github organization for astrophotographic image processing tools, and that includes an unregistered WIP package, ImageStacking.jl. That package only covers stacking, not registration (but it can stack in parallel and provide more statistics than the mean!). For testing I’ve just relied on my own monochrome data pre-registered through interpolation in Siril - I haven’t started factoring in other parts that are needed for working with drizzled data. Ideally, whatever a drizzle implementation produces should be able to be piped into this package and stacked.

10

Howdy, one of the Astroalign.jl authors here. Yes, I would be inclined to split Rainer’s stacking functionality into a separate package / combine forces with other efforts like yours, Brandon, that we could then call from. Thanks all for your work!

11

In general I also agree, that the stacking can or even should be a different package.
What I have not really evaluated in detail though is that the implementation I referred to above linked the drizzle rather tightly in the align_frame function. Separating drizzle from frame alignment probably means that we either need to come up with a better infrastructure (e.g. passing a “warp!” function as an optional argument for the final warp only) or we have performance issues, since we would need to warp the same image multiple times. Its actually even a bit more involved, since we need access to the drizzle_mask currently being returned by that align_frame function, also to avoid multi-pass operations in the drizzle warp. However, a warp! can probably handle this, as the mask can be set (or even updated?) internally.
I totally agree that is is much cleaner to not requiring align_frame to do the drizzle, but we then need to make it somehow capable of still accepting other ways of warping and “returning” additional results such as the mask.
What do you think, should we give this idea with the optional (or defaulting) final warp!() as an argument?

12

Since this is turning into a discussion about several different packages, I figure it might be good to discuss this in a different thread/platform? I’ll loop in @icweaver as well.

I’m in the Julia Slack under the same username if you’d prefer to work there, or let me know your preferences. In the meantime, I’ve started a draft repo for Drizzle.jl.

13

I now separated the code into the AstroStacker.jl package. It now uses the Ransak version of Astroalign.jl and its own drizzle implementation. I agree that it would be nice to somehow link it to other packages like BayeredImages.jl or move some of the bayering or drizzle code over to this package, but for now that seemed a sensible way to get the stacking working again.