Augmenting positional labels with their corresponding images for training

I have used Augmentor.jl to perform data augmentation in the past, and it’s a great package. Recently I have been working with image data whose labels are coordinates (i.e., each label is an (x, y) coordinate specifying the position of some object of interest in the image). Augmentor.jl allows me to manipulate images in cool, stochastic ways, but I don’t know of a way to keep track of the augmentation that was just performed in order to augment the label accordingly.

Is there an easy way to do this, or would I have to write my own augmentation code?