# Modeling multimodal data in Flux

**URL:** <https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [August 27, 2023, 1:18pm UTC](https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257 "2023-08-27T13:18:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![usiam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usiam/32/47120_2.png) [@usiam](https://discourse.julialang.org/u/usiam)\
**Post date:** [August 27, 2023, 1:18pm UTC](https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257/1 "2023-08-27T13:18:57Z")

</div>

Hey. I am trying to model some multimodal time-series data. The data has 3 “components” - a time-series component (`shape = [timesteps, channels, batchsize]`), and 2 separate flattened components (`shape = [numfeatures, batchsize]`). This is going to be a regression task.

For my particular data the shapes are `[50, 3, 64]`, `[4, 64]` and `[1, 64]`. The output is of shape `[12, 64]`.

I’m a little new to Flux (and Julia in general) and was wondering how one would model this?

My dataloader returns a tuple of xs and ys where xs is also a tuple of (timeseries, flattenedmatrix, flattenedmatrix).

```julia
for ((tfw, tfs, tfp), tftgt) in traindl
    @show tfw |> size tfs |> size tfp |> size tftgt |> size
    break
end

tfw |> size = (50, 3, 64)
tfs |> size = (4, 64)
tfp |> size = (1, 64)
tftgt |> size = (12, 64)

```

I am a little new to time-series modeling as well so if there’s any resource to help me model time-series well, that would be cool as well. And also any way to add augmentations to time-series is also welcomed as currently I am not adding any data augmentation.

I also cannot for the life of me figure out how to use the `LSTM` in Flux. Any idea if `Self-Attention` would be nice thing to try as well?

Thank you!

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [August 29, 2023, 7:55am UTC](https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257/2 "2023-08-29T07:55:36Z")

</div>

Not sure about time series modelling, but the [Parallel layer](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.Parallel) is useful to handle multiple input.

---

<div class="post-metadata">

**Author:** ![usiam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/usiam/32/47120_2.png) [@usiam](https://discourse.julialang.org/u/usiam)\
**Post date:** [August 29, 2023, 10:38pm UTC](https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257/3 "2023-08-29T22:38:46Z")

</div>

Do you know if there is a way to track activation means and standard deviations as the data flows through the layers? Do I have to manually do it?

I’m looking for something like `register_hook` in PyTorch. [torch.Tensor.register\_hook — PyTorch 2.0 documentation](https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html)

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 31, 2023, 12:34am UTC](https://discourse.julialang.org/t/modeling-multimodal-data-in-flux/103257/4 "2023-08-31T00:34:14Z")

</div>

The PyTorch hook functionality is for gradients, not activations. Are you looking for the former? If so, [Utilities · Zygote](https://fluxml.ai/Zygote.jl/stable/utils/#Zygote.hook) works.