# \#transformers

**URL:** https://discourse.julialang.org/tag/transformers/790.md

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

---

## [Basic example for using Transformers.jl for sequential autoencoding](https://discourse.julialang.org/t/basic-example-for-using-transformers-jl-for-sequential-autoencoding/123663)

<div class="topic-metadata">

**Author:** [@spolk](https://discourse.julialang.org/u/spolk)\
**Replies:** 0\
**Last updated:** [December 10, 2024, 3:06pm UTC](https://discourse.julialang.org/t/basic-example-for-using-transformers-jl-for-sequential-autoencoding/123663 "2024-12-10T15:06:19Z")

</div>

I am interested in using Transformers.jl for sequence-to-sequence autoencoding and was hoping to get help solving a minimum working example building and training an appropriate transformer-based sequence-to-sequence aut…

---

## [Downloading transformer weights](https://discourse.julialang.org/t/downloading-transformer-weights/122893)

<div class="topic-metadata">

**Author:** [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Replies:** 0\
**Last updated:** [November 21, 2024, 12:26pm UTC](https://discourse.julialang.org/t/downloading-transformer-weights/122893 "2024-11-21T12:26:12Z")

</div>

I’d like to download the weighting matrices for an open source transformer model (BERT, GPT2…) and try and do the matrix equations from scatch in Julia. Any thoughts on the best way to do this? ChatGPT suggested eithe…

---

## [Identify and add missing rows dynamically in dataframe](https://discourse.julialang.org/t/identify-and-add-missing-rows-dynamically-in-dataframe/112703)

<div class="topic-metadata">

**Author:** [@Sandy45](https://discourse.julialang.org/u/Sandy45)\
**Replies:** 6\
**Last updated:** [April 9, 2024, 11:25pm UTC](https://discourse.julialang.org/t/identify-and-add-missing-rows-dynamically-in-dataframe/112703 "2024-04-09T23:25:28Z")

</div>

I have scenario where I need to insert the missing data into dataframe dynamically. 3×3 DataFrame Row │ month ac\_hrs crew\_hrs │ String Int64 Int64 ─────┼────────────────────────── 1 │ 202404 2 …

---

## [16 bit float on Transformers.jl](https://discourse.julialang.org/t/16-bit-float-on-transformers-jl/107867)

<div class="topic-metadata">

**Author:** [@RainerHeintzmann](https://discourse.julialang.org/u/RainerHeintzmann)\
**Replies:** 7\
**Last updated:** [December 28, 2023, 7:40pm UTC](https://discourse.julialang.org/t/16-bit-float-on-transformers-jl/107867 "2023-12-28T19:40:13Z")

</div>

I am new to LLMs but thought to give it a shot using Transformers.jl. This worked rather well, after creating an account at HuggingFace and getting permission for Llama-7B from Meta. Yet, I was surprised how much GPU ram…

---

## [Transformers.jl: How to train for masked languge model tasks in Julia?](https://discourse.julialang.org/t/transformers-jl-how-to-train-for-masked-languge-model-tasks-in-julia/107227)

<div class="topic-metadata">

**Author:** [@roeya](https://discourse.julialang.org/u/roeya)\
**Replies:** 4\
**Last updated:** [December 7, 2023, 11:25am UTC](https://discourse.julialang.org/t/transformers-jl-how-to-train-for-masked-languge-model-tasks-in-julia/107227 "2023-12-07T11:25:53Z")

</div>

What am I missing here - seems to be easy enough but I cannot find any documentation on how to do it

---

## [Loading a trained model in Transformers.jl](https://discourse.julialang.org/t/loading-a-trained-model-in-transformers-jl/104210)

<div class="topic-metadata">

**Author:** [@kadir-gunel](https://discourse.julialang.org/u/kadir-gunel)\
**Replies:** 0\
**Last updated:** [September 25, 2023, 8:17am UTC](https://discourse.julialang.org/t/loading-a-trained-model-in-transformers-jl/104210 "2023-09-25T08:17:53Z")

</div>

Hello All, I am having a problem while trying to load a model that is trained from scratch with Transformers.jl. The model it self is a seq2seq model that is very similar to the one in the documentation. Only difference…

---

## [Saving a model built with Transformers.jl](https://discourse.julialang.org/t/saving-a-model-built-with-transformers-jl/96132)

<div class="topic-metadata">

**Author:** [@gemuesi](https://discourse.julialang.org/u/gemuesi)\
**Replies:** 6\
**Last updated:** [March 15, 2023, 1:48pm UTC](https://discourse.julialang.org/t/saving-a-model-built-with-transformers-jl/96132 "2023-03-15T13:48:28Z")

</div>

Hi there, I am pretty new to anything related to this topic in general, but I found Flux to be quite accessible and wanted to have a look at the Transformers.jl package as well. I’ve been playing around with the copy ta…

---

## [Transformers.jl: causal mask on decoder](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212)

<div class="topic-metadata">

**Author:** [@Rrdd](https://discourse.julialang.org/u/Rrdd)\
**Replies:** 4\
**Last updated:** [February 28, 2023, 7:16am UTC](https://discourse.julialang.org/t/transformers-jl-causal-mask-on-decoder/95212 "2023-02-28T07:16:22Z")

</div>

hi, I am new to Transformers.jl and try to follow the tutorial (Tutorial · Transformers.jl). I wonder where I can find more details about this call t = decoder\_trf(e, m, attention\_mask, cross\_attention\_mask) In particu…

---

## [Training sentence transformers in Julia?](https://discourse.julialang.org/t/training-sentence-transformers-in-julia/71460)

<div class="topic-metadata">

**Author:** [@Alex\_Tantos](https://discourse.julialang.org/u/Alex_Tantos)\
**Replies:** 0\
**Last updated:** [November 14, 2021, 3:57pm UTC](https://discourse.julialang.org/t/training-sentence-transformers-in-julia/71460 "2021-11-14T15:57:45Z")

</div>

Hi! Has anyone tried to train a sentence transformer by fine tuning BERT in julia following the method of the 2019 paper on sentence transformers (and the model is called SBERT)? I know this is too general a question, b…

---

## [Transformers for NER classification](https://discourse.julialang.org/t/transformers-for-ner-classification/69300)

<div class="topic-metadata">

**Author:** [@Maxim\_Lubov](https://discourse.julialang.org/u/Maxim_Lubov)\
**Replies:** 9\
**Last updated:** [October 12, 2021, 6:07pm UTC](https://discourse.julialang.org/t/transformers-for-ner-classification/69300 "2021-10-12T18:07:15Z")

</div>

Hello. I am building a model for NER classification. Model is based on pre-trained BERT loaded with Transformers. I have some problems with the loss function. The problem is, that all losses are calculated only for the s…

---

## [Running a pre-trained BERT on twitter data using Flux.jl Transformer.jl](https://discourse.julialang.org/t/running-a-pre-trained-bert-on-twitter-data-using-flux-jl-transformer-jl/67721)

<div class="topic-metadata">

**Author:** [@Storopoli](https://discourse.julialang.org/u/Storopoli)\
**Replies:** 17\
**Last updated:** [September 16, 2021, 9:24am UTC](https://discourse.julialang.org/t/running-a-pre-trained-bert-on-twitter-data-using-flux-jl-transformer-jl/67721 "2021-09-16T09:24:41Z")

</div>

Hi everybody! I have 8k labelled twitter data with either a 0 or 1 (binary classification). MLJ.jl with classical machine learning has not very good performance with the problem. I am trying to fit a pre-trained BERT t…