Modeling multimodal data in Flux

Specific Domains Machine Learning
1

Hey. I am trying to model some multimodal time-series data. The data has 3 “components” - a time-series component (shape = [timesteps, channels, batchsize]), and 2 separate flattened components (shape = [numfeatures, batchsize]). This is going to be a regression task.

For my particular data the shapes are [50, 3, 64], [4, 64] and [1, 64]. The output is of shape [12, 64].

I’m a little new to Flux (and Julia in general) and was wondering how one would model this?

My dataloader returns a tuple of xs and ys where xs is also a tuple of (timeseries, flattenedmatrix, flattenedmatrix).

for ((tfw, tfs, tfp), tftgt) in traindl
    @show tfw |> size tfs |> size tfp |> size tftgt |> size
    break
end

tfw |> size = (50, 3, 64)
tfs |> size = (4, 64)
tfp |> size = (1, 64)
tftgt |> size = (12, 64)

I am a little new to time-series modeling as well so if there’s any resource to help me model time-series well, that would be cool as well. And also any way to add augmentations to time-series is also welcomed as currently I am not adding any data augmentation.

I also cannot for the life of me figure out how to use the LSTM in Flux. Any idea if Self-Attention would be nice thing to try as well?

Thank you!

2

Not sure about time series modelling, but the Parallel layer is useful to handle multiple input.

3

Do you know if there is a way to track activation means and standard deviations as the data flows through the layers? Do I have to manually do it?

I’m looking for something like register_hook in PyTorch. torch.Tensor.register_hook — PyTorch 2.0 documentation

4

The PyTorch hook functionality is for gradients, not activations. Are you looking for the former? If so, Utilities · Zygote works.