# Basic example for using Transformers.jl for sequential autoencoding

**URL:** <https://discourse.julialang.org/t/basic-example-for-using-transformers-jl-for-sequential-autoencoding/123663>\
**Category:** Machine Learning\
**Tags:** examples, flux, transformers, sentence-transformer\
**Created:** [December 10, 2024, 3:06pm UTC](https://discourse.julialang.org/t/basic-example-for-using-transformers-jl-for-sequential-autoencoding/123663 "2024-12-10T15:06:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![spolk](https://avatars.discourse-cdn.com/v4/letter/s/e68b1a/32.png) [@spolk](https://discourse.julialang.org/u/spolk)\
**Post date:** [December 10, 2024, 3:06pm UTC](https://discourse.julialang.org/t/basic-example-for-using-transformers-jl-for-sequential-autoencoding/123663/1 "2024-12-10T15:06:19Z")

</div>

I am interested in using `Transformers.jl` for sequence-to-sequence autoencoding and was hoping to get help solving a minimum working example building and training an appropriate transformer-based sequence-to-sequence autoencoder using Transformer.jl on a synthetic dataset of sequences of varying length.

For example, consider the following dataset, which has consists of sequences of random samples from a Gaussian distribution:

```julia
# Set parameters for synthetic sequence data generation
elem_dim = 5; # The number of dimensions in a sequence's element
mean_seq_length = 10; # Desired average length of sequences 
std_seq_length = 3; # Standard deviation in length of sequences 
num_seqs = 100; # Number of sequences to generate

seqs = [
    [randn(elem_dim,1) for j = 1:(std_seq_length*randn() + mean_seq_length)] 
    for i = 1:num_seqs]
]

```

Given data of this format, I am interested in the answers to the following questions:

1. How does the variable `seqs` need to be formatted in order to be input into a transformer using Transformers.jl?
2. What is the smallest, most basic transformer-based architecture for sequence-to-sequence autoencoding for data of this type?

Thank you very much for your time and help.I apologize that this question is quite basic, but I haven’t been able to find a suitable answer online.