Big changes in Images.jl (v0.6.0)

Hi everyone,

Images.jl, a package for processing and visualizing images, has essentially been rewritten from scratch and version 0.6.0 has been tagged. Because this is a major and disruptive release, I suspect a few words are in order.

Images.jl is one of the oldest packages in the Julia ecosystem, and its code-origins predate the existence of both the package manager and Julia 0.1. From the beginning of my own involvement, my main goal was to ensure that it would be useful for both “computer vision” (largely a world of 2d images) and for big-data 3D/4D/5D image analysis. We were partially able to pull that off, but there were some ugly compromises, most notably encoding important information such as the spatial orientation and “meaning” of the axes of an image using an easily lost, desynchronized, or confused dictionary. This may have been the best we could do at the time. However, the Julia language has evolved considerably, and with the release of Julia 0.5 it became apparent that all the pieces were in place for a better approach.

The new Images makes far better use of Julia’s type system. The dictionary is still available for those who want it (through the ImageMetadata package, which is “bundled” with Images itself), but most people should be able to just use “plain arrays,” more specifically a variety of AbstractArray types defined in various packages. Key advances that made this possible include packages such as AxisArrays (which allows one to give names to axes and encode pixel geometry in a type-system-friendly fashion), MappedArrays (which allows one to extend reinterpret-style functionality to arbitrary AbstractArrays), and the existence of the PermutedDimsArray in Base (for lazy-changes of orientation with images too large to handle in memory). Capitalizing on these and other advances in Julia’s treatment of AbstractArrays is the main motivation for the rewrite.

From a user’s perspective, the first thing you’re likely to notice is a whole slew of deprecation warnings and errors, which I know is a major hassle and for which I apologize. (If you have an impending grant deadline, your best bet will probably be to pin the package at an earlier release. I’d also like to thank Tony Kelman for putting significant work into the dependency chains of our package manager to reduce the likelihood of breakage.) One particular pain-point is for Mac users, who for now likely need to rely on ImageMagick (using ImageMagick) to save and load images. I’m hopeful that QuartzImageIO will update to the new version soon. More than a thousand lines of code were devoted to nothing but issuing deprecation warnings (and where necessary, error messages) that attempt to coach you through rewriting your code; I hope these will at least somewhat reduce your pain in making this transition.

While the main emphasis of this rewrite has been on “core” functionality, there are a few goodies to tempt you (e.g., the new mapwindow for applying an arbitrary function over moving windows of an image). The most important reason to upgrade, however, is that I suspect you’ll find the new version to be considerably more robust and well-behaved than the old.

To learn more, the main resources are:

  • a new NEWS.md file, which describes the major changes in greater detail
  • a completely new set of documentation, now hosted at the JuliaImages umbrella organization
  • an upgrading_tips document on handling errors encountered during upgrading to the new version (as you encounter problems, please contribute updates)
  • a script to make it easier to deal with the recent name-changes in FixedPointNumbers (i.e., UFixed8->N0f8 etc.)

Finally, Images.jl has been transferred to JuliaImages. Going forward, I hope the main emphasis will be on added functionality, either added to Images.jl itself or implemented in separate packages (Images is evolving into more of an “umbrella” package). I also hope that users contribute “demos” to the JuliaImages documentation, similar to that of other image-processing frameworks. We’d be excited to receive contributions to source code, documentation, or the test suite!

Best,
–Tim Holy

33 Likes

I’ve been looking forward to this since I first saw https://github.com/JuliaImages/Images.jl/issues/542 back in October. Thanks for all your hard work! I’m consistently impressed with just how many things Images.jl gets right.

This is absolutely great.
I have been playing around with the new Images “packages” and this seems to be really great. The concepts are now much more orthogonal. It is some learning effort but I think this will pay off quickly. Many many thanks to Tim but also to all others involved. In particular to @mbauman for the AxisArrays package.

4 Likes

@tim.holy: Hi Tim I am currently upgrading our larger code base to the new image infrastructure. There are some issues where I have some question: What is you preferred channel for asking these type of user question? Discourse, Github Issues (on which package?). I will still ask the question here but we can move the discussion if you prefer.

Issue 1: My main type being used now is an ImageMeta of an AxisArray. But my underlaying “core” algorithm should of course handle both types. How can I indicate in a type signature that it can be both? In other words: I want to indicate that an AbstractArray is coming in and that it has a method “pixelspacing”. So AbstractArray would not be ok.

Issue 2: What is the standard way to convert an ArrayAxis to an Array? I was calling some Winston code that simply requires an Array and needed to convert. In the end I used convert(Array,im) which does the job but from the documentation it was not clear if this is the right way.

Issue 3: I am not sure if this is intended behavior. But if I index an ImageMeta of an AxisArray it will be an ImageMeta of an Array (so the AxisArray is lost). getindexim does what I want. However, if I index an AxisArray regularly I get an AxisArray. Thus it is tricky to write code that works on both.

I’d say posting questions here on discourse is fine; if the volume gets too large we could presumably set up an alternative to “Community.” If you’ve found an obvious bug, just report it as an issue with Images.jl or some other package.

I want to indicate that an AbstractArray is coming in and that it has a method “pixelspacing”. So AbstractArray would not be ok.

julia> using Images

julia> pixelspacing(rand(3,5))
(1,1)

That said, if what you’re asking is whether it’s straightforward to support only AxisArray and ImageMeta, there’s an unexported ImageMetaAxis that means “an ImageMeta formed from an AxisArray.” Try something like this:

typealias AxisArrayLike{T,N} Union{AxisArray{T,N}, ImageMetadata.ImageMetaAxis{T,N}}

What is the standard way to convert an ArrayAxis to an Array? I was calling some Winston code that simply requires an Array and needed to convert. In the end I used convert(Array,im)

That seems reasonable. Maybe even better would be to modify Winston to accept something more general than Array.

I am not sure if this is intended behavior. But if I index an ImageMeta of an AxisArray it will be an ImageMeta of an Array (so the AxisArray is lost).

That’s a bug; thanks for catching it. I don’t have time to fix right this second, can you file an issue over at ImageMetadata?

That seems reasonable. Maybe even better would be to modify Winston to accept something more general than Array.

Not sure if this is reasonable. This is the image function that at the end needs a memory block in order to interface with C. So I understand that we want AbstractArray in most cases to support images but there are exceptions

[quote=“tim.holy, post:5, topic:1786”]
julia> using Images

julia> pixelspacing(rand(3,5))
(1,1)
[/quote]

I see. But still not sure how to handle my code in a better way. Here it is:

function dosomething(a::Array, pixelspacing)
...
end
dosomething(a::AbstractImage) = dosomething(data(a), pixelspacing(a))

The intention was to also allow regular arrays going in but that the user should provide a pixelspacing in these cases.

That’s a bug; thanks for catching it. I don’t have time to fix right this second, can you file an issue over at ImageMetadata?

sure will do!

Not sure if this is reasonable. This is the image function that at the end needs a memory block in order to interface with C. So I understand that we want AbstractArray in most cases to support images but there are exceptions

If code needs a contiguous block of memory, most likely it plans to call pointer(A) (or the equivalent, unsafe_convert(Ptr{T}, A)). Why not just call pointer and let it fail if the array type doesn’t implement a relevant method? If the “inner” functions it uses already throw relevant errors in case of problems, placing type restrictions on the “outer” functions unnecessarily blocks users from supplying types that might be perfectly appropriate. In other words, (1) place type-restrictions at their appropriate level, and (2) don’t implement methods for a given type if they don’t make sense.

The intention was to also allow regular arrays going in but that the user should provide a pixelspacing in these cases.

dosomething(a::AxisArrayLike) = dosomething(a, pixelspacing(a)) (using that typealias in my previous post) is basically the new equivalent. Presumably you should very rarely need to call data, most code should just work for AbstractArrays; as demonstrated, you can use typealiases if you really need to impose certain restrictions (or need dispatch specializations).

If this still seems unsatisfying, I might need to more about the internals of dosomething to better understand the issues you’re facing.

If code needs a contiguous block of memory, most likely it plans to call pointer(A) (or the equivalent, unsafe_convert(Ptr{T}, A)). Why not just call pointer and let it fail if the array type doesn’t implement a relevant method? If the “inner” functions it uses already throw relevant errors in case of problems, placing type restrictions on the “outer” functions unnecessarily blocks users from supplying types that might be perfectly appropriate. In other words, (1) place type-restrictions at their appropriate level, and (2) don’t implement methods for a given type if they don’t make sense.

I can follow your argumentation but am not sure if I agree or disagree :slight_smile: For the user it can be pretty confusing getting the error at the lowest level. So it can make sense to make the type restriction at the interface level. But this discussion is really about duck typing vs making type restriction. So both have pros and cons.

If this still seems unsatisfying, I might need to more about the internals of dosomething to better understand the issues you’re facing.

No your suggestion is very good. In the end I also want to learn the new way of structuring Image code and not only solve the concrete issue.

Thanks a lot!

Update: Ron Rock has updated QuartzImageIO, and the mac experience looks to be better than ever. Thanks Ron!

4 Likes

Thread necromancy, but worthwhile, I think.

Tim, Thanks so much for this stalwart work. I’m just finishing off my Thesis, and I’ve been working on (my) creaky image code for a while. Having a solid foundation upon which to build my next project on is really exciting.

1 Like