# Transformers.jl: causal mask on decoder

**URL:** <https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212>\
**Category:** General Usage\
**Tags:** transformers\
**Created:** [February 26, 2023, 2:54am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212 "2023-02-26T02:54:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rrdd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rrdd/32/38481_2.png) [@Rrdd](https://discourse.julialang.org/u/Rrdd)\
**Post date:** [February 26, 2023, 2:54am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212/1 "2023-02-26T02:54:15Z")

</div>

hi, I am new to Transformers.jl and try to follow the tutorial ([Tutorial · Transformers.jl](https://chengchingwen.github.io/Transformers.jl/dev/tutorial/)). I wonder where I can find more details about this call  
t = decoder\_trf(e, m, attention\_mask, cross\_attention\_mask)

In particular, how to modify the above to allow a causal mask to be applied to the decoder input (to avoid peeking ahead). Many thanks!

---

<div class="post-metadata">

**Author:** ![Rrdd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rrdd/32/38481_2.png) [@Rrdd](https://discourse.julialang.org/u/Rrdd)\
**Post date:** [February 28, 2023, 3:06am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212/2 "2023-02-28T03:06:02Z")

</div>

@chengchingwen  
To be more specific, for the lookahead mask, shall I do something along the line of:

t = decoder\_trf(e, m, NeuralAttentionlib.CausalMask(), cross\_attention\_mask)

thanks!

---

<div class="post-metadata">

**Author:** ![chengchingwen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chengchingwen/32/8390_2.png) [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)\
**Post date:** [February 28, 2023, 3:34am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212/3 "2023-02-28T03:34:30Z")

</div>

You don’t need to do that manually. The [`TransformerDecoderBlock`](https://github.com/chengchingwen/Transformers.jl/blob/master/src/layers/layer.jl#L478) constructor create a [`CausalMultiheadQKVAttenOp`](https://github.com/chengchingwen/Transformers.jl/blob/master/src/layers/layer.jl#L293) for the self attention, which does the causal masking already. The basic functionality of `attention_mask` in decoder is for putting something like [`LengthMask`](https://chengchingwen.github.io/NeuralAttentionlib.jl/dev/api/#NeuralAttentionlib.LengthMask) for avoiding padding affect the computation.

---

<div class="post-metadata">

**Author:** ![Rrdd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rrdd/32/38481_2.png) [@Rrdd](https://discourse.julialang.org/u/Rrdd)\
**Post date:** [February 28, 2023, 4:55am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212/4 "2023-02-28T04:55:25Z")

</div>

@chengchingwen that’s convenient and works like a charm!

A follow-up question: In the rare case when I dont want to have this mask, or maybe a special mask that’s not triangular, can this be done?

---

<div class="post-metadata">

**Author:** ![chengchingwen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chengchingwen/32/8390_2.png) [@chengchingwen](https://discourse.julialang.org/u/chengchingwen)\
**Post date:** [February 28, 2023, 7:16am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212/5 "2023-02-28T07:16:22Z")

</div>

Yes, but you would probably need to call the inner-most constructor with `MultiheadQKVAttenOp` and pass your own attention mask from the input. You can find [`NeuralAttentionlib`](https://chengchingwen.github.io/NeuralAttentionlib.jl/dev/api/#Mask) for more kind of masks.