# Using Transformers.jl for time series classification?
**URL:** https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509
**Category:** Web Stack
**Created:** [November 20, 2020, 5:12pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509 "2020-11-20T17:12:52Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@lorrp1](https://discourse.julialang.org/u/lorrp1)
#### Post date: [November 20, 2020, 5:12pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/1 "2020-11-20T17:12:52Z")
Hello, has anyone tried using Transformers.jl for time series classification?
is it possible?
@chengchingwen
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [November 20, 2020, 5:24pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/2 "2020-11-20T17:24:01Z")
I think it’s possible. You can the treat time series classification task as a sentence classification task, but I never try it before. No performance guarantee.
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [November 20, 2020, 5:32pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/3 "2020-11-20T17:32:11Z")
There do exist some papers using Transformer model for time series data, like this one: [https://arxiv.org/pdf/2001.08317.pdf](https://arxiv.org/pdf/2001.08317.pdf)
---
### Author:  [@lorrp1](https://discourse.julialang.org/u/lorrp1)
#### Post date: [November 21, 2020, 1:02pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/4 "2020-11-21T13:02:29Z")
hello @chengchingwen
so in the tutorial i could just try changing the length(vocab)
with the number of classes i want?
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [November 21, 2020, 1:42pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/5 "2020-11-21T13:42:47Z")
Not really. It depends on what kinds of time series data you’re working on. If it is real-valued time series, like stock prices, you probably don’t need the embedding layer. Which also means you can’t do cross entropy loss on the output.
---
### Author:  [@lorrp1](https://discourse.julialang.org/u/lorrp1)
#### Post date: [November 22, 2020, 8:56pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/6 "2020-11-22T20:56:44Z")
hello @chengchingwen , ill see if i need one-got encoding anyway im just trying the tutorial now but the loss is not getting lower:
```julia
using Flux,Transformers, CUDA,Transformers.Basic, Transformers.Datasets: batched, Flux: gradient, Flux.Optimise: update!
using Flux: onecold, Flux: onehot
enable_gpu(true)
labels = collect(1:10)
startsym = 11
endsym = 12
unksym = 0
labels = [unksym, startsym, endsym, labels...]
vocab = Vocabulary(labels, unksym)
#function for generate training datas
sample_data() = (d = rand(1:10, 10); (d,d))
#function for adding start & end symbol
preprocess(x) = [startsym, x..., endsym]
sample = preprocess.(sample_data()) # = ([11, 5, 4, 2, 5, 2, 5, 5, 5, 7, 8, 12], [11, 5, 4, 2, 5, 2, 5, 5, 5, 7, 8, 12])
encoded_sample = vocab(sample[1])# = [2, 8, 7, 5, 8, 5, 8, 8, 8, 10, 11, 3]
#define a Word embedding layer which turn word index to word vector
embed = Embed(512, length(vocab)) |> gpu
#define a position embedding layer metioned above
pe = PositionEmbedding(512) |> gpu
#wrapper for get embedding
function embedding(x)
we = embed(x, inv(sqrt(512)))
e = we .+ pe(we)
return e
end
#define 2 layer of transformer
encode_t1 = Transformer(512, 8, 64, 2048) |> gpu
encode_t2 = Transformer(512, 8, 64, 2048) |> gpu
#define 2 layer of transformer decoder
decode_t1 = TransformerDecoder(512, 8, 64, 2048) |> gpu
decode_t2 = TransformerDecoder(512, 8, 64, 2048) |> gpu
#define the layer to get the final output probabilities
linear = Positionwise(Dense(512, length(vocab)), logsoftmax) |> gpu
function encoder_forward(x)
e = embedding(x)
t1 = encode_t1(e)
t2 = encode_t2(t1)
return t2
end
function decoder_forward(x, m)
e = embedding(x)
t1 = decode_t1(e, m)
t2 = decode_t2(t1, m)
p = linear(t2)
return p
end
enc = encoder_forward(encoded_sample)
probs = decoder_forward(encoded_sample, enc)
################################################################################
#trian:
function smooth(et)
sm = fill!(similar(et, Float32), 1e-6/size(embed, 2))
p = sm .* (1 .+ -et)
label = p .+ et .* (1 - convert(Float32, 1e-6))
label
end
#define loss function
function loss(x, y)
label = onehot(vocab, y) #turn the index to one-hot encoding
label = smooth(label) #perform label smoothing
enc = encoder_forward(x)
probs = decoder_forward(y, enc)
l = logkldivergence(label[:, 2:end, :], probs[:, 1:end-1, :])
return l
end
#collect all the parameters
ps = params(embed, pe, encode_t1, encode_t2, decode_t1, decode_t2, linear)
opt = ADAM(1e-4)
#function for created batched data
using Transformers.Datasets: batched
#flux function for update parameters
using Flux: gradient
using Flux.Optimise: update!
#define training loop
function train!()
@info "start training"
for i = 1:2000
data = batched([sample_data() for i = 1:32]) #create 32 random sample and batched
x, y = preprocess.(data[1]), preprocess.(data[2])
x, y = vocab(x), vocab(y)#encode the data
x, y = todevice(x, y) #move to gpu
l = loss(x, y)
grad = gradient(()->l, ps)
if i % 8 == 0
println("loss = $l")
end
update!(opt, ps, grad)
end
end
train!()
################################################################################
#test:
function translate(x)
ix = todevice(vocab(preprocess(x)))
seq = [startsym]
enc = encoder_forward(ix)
len = length(ix)
for i = 1:2len
trg = todevice(vocab(seq))
dec = decoder_forward(trg, enc)
#move back to gpu due to argmax wrong result on CuArrays
ntok = onecold(collect(dec), labels)
push!(seq, ntok[end])
ntok[end] == endsym && break
end
seq[2:end-1]
end
translate([5,5,6,6,1,2,3,4,7, 10])
```
any idea?
thanks
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [November 24, 2020, 12:12pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/7 "2020-11-24T12:12:32Z")
@lorrp1 Ah, sorry about that. The tutorial on the docs is kind of outdated since that was written during the pre-Zygote.jl era. The loss didn’t get lower because actually there’re no gradient value received. The problem is that `l = loss(x,y); grad = gradient(()->l, ps)` no longer produce gradient since it’s not tracker-based AD anymore. You can find the newest example in the [example folder](https://github.com/chengchingwen/Transformers.jl/tree/master/example). I’ll also update the tutorial later.
the fastest fix would be adding `Flux.@nograd smooth` and changing `grad` to `grad = gradient(()->loss(x, y), ps)`
---
### Author:  [@lorrp1](https://discourse.julialang.org/u/lorrp1)
#### Post date: [November 28, 2020, 1:11am UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/8 "2020-11-28T01:11:12Z")
@chengchingwen Hello, it now works fine like you said thank you,
im trying using a label like 0,1 (up down) in a time series to classify if v[x]\>v[x-1] or not after seeing both v[x] and v[x-1].
i need the positional embedding to “sort” the input array but not the word embedding layer so:
im not sure if its correct to make the embedding like this:
```julia
> pe = PositionEmbedding(512) |> gpu
> function embedding(x)
> return pe(x)
> end
```
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [November 28, 2020, 9:37am UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/9 "2020-11-28T09:37:26Z")
I don’t quite understand what you intend to do. Maybe some small example would be better.
---
### Author:  [@lorrp1](https://discourse.julialang.org/u/lorrp1)
#### Post date: [November 29, 2020, 4:48pm UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/10 "2020-11-29T16:48:23Z")
i mean a simple classification of a given array based on pre-defined labels for example here [Getting Started — MLDataUtils.jl v0.1 documentation](https://mldatautilsjl.readthedocs.io/en/latest/introduction/gettingstarted.html)
using a dataset like this:
> julia\> getobs((X, Y), 30)
> ([4.7,3.2,1.6,0.2],“setosa”)
array → label
id need a PositionEmbedding
```julia
pe = PositionEmbedding(512) |> gpu
function embedding(x)
return pe(we)
end
```
then
> ps = params(pe, encode\_t1, encode\_t2, decode\_t1, decode\_t2, linear)
would this work?
---
### Author:  [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)
#### Post date: [December 23, 2020, 5:07am UTC](https://discourse.julialang.org/t/using-transformers-jl-for-time-series-classification/50509/11 "2020-12-23T05:07:00Z")
Because `PositionEmbedding`’s api have some assumption, you need to be careful of the input type.
That’s said we have a sequence data of length n, the easiest way to get the position embedding for it would be just passing the length n to it:
```julia
pe = PositionEmbedding(emb_size)
p = pe(n) # the position embeddings of size (emb_size, n)
```
Otherwise you would need to check if your input type is the same as the regular usage on word inputs.
btw, I don’t regularly check discourse. Pin me if I didn’t show up for a long time.