# All the ways to do one-hot encoding
**URL:** https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807
**Category:** General Usage
**Created:** [July 17, 2021, 10:42am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807 "2021-07-17T10:42:01Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [July 17, 2021, 10:42am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/1 "2021-07-17T10:42:01Z")
There are many ways to do one hot encoding in Julia. I want to list down some ways
| Method | Comment |
| --- | --- |
| Roll your own | It’s not too hard |
| Flux.onehot & Flux.onehotbatch | pretty heavy to depend on Flux so normally avoided unless u r using Flux anyway |
| `DataConvenience.onehot` | can be applied directly to dataframes |
| `FeatureTransform.OneHotEncoder` | They syntax for FeatureTransform is still on the verbose side for my liking imo |
| `MLJ.OneHotEncoder()` | I really want to like MLJ but it’s quite heavy and makes me want to avoid it. What’s with this `machine`? Why do I need a machine to do one hot encoding? |
| `ScikitLearn.jl` | Hmm, need to install a bunch of Python stuff. Thank you no thank you for such a simple thing |
| Keep your data in categorical format | You need to find libraries that accept that |
| `unique(x) .== permutedims(x)` | due to @Mattriks. Great for if you don’t need sparse representation |
That’s it. Anything I’ve missed?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 17, 2021, 10:52am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/2 "2021-07-17T10:52:18Z")
```julia
julia> using StatsBase
julia> x = ["a", "b", "c", "a", "b", "d"]
6-element Vector{String}:
"a"
"b"
"c"
"a"
"b"
"d"
julia> indicatormat(x)
4×6 Matrix{Bool}:
1 0 0 1 0 0
0 1 0 0 1 0
0 0 1 0 0 0
0 0 0 0 0 1
```
but normally I do your option 1 😃 :
```julia
julia> df = DataFrame(x=x)
6×1 DataFrame
Row │ x
│ String
─────┼────────
1 │ a
2 │ b
3 │ c
4 │ a
5 │ b
6 │ d
julia> select(df, [:x => ByRow(isequal(v))=> Symbol(v) for v in unique(df.x)])
6×4 DataFrame
Row │ a b c d
│ Bool Bool Bool Bool
─────┼────────────────────────────
1 │ true false false false
2 │ false true false false
3 │ false false true false
4 │ true false false false
5 │ false true false false
6 │ false false false true
```
(DataConvenience.jl is nice 👍)
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [July 17, 2021, 11:01am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/3 "2021-07-17T11:01:21Z")
Yeah. Nice. I think the issue with DataConvenience.jl is that it’s not designed for production use. So missing categories at production time is harder to handle.
---
### Author:  [@Mattriks](https://discourse.julialang.org/u/Mattriks)
#### Post date: [July 17, 2021, 11:38am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/4 "2021-07-17T11:38:51Z")
```julia
unique(x) .== permutedims(x)
```
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [July 17, 2021, 12:04pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/6 "2021-07-17T12:04:05Z")
StatsModels.jl `DummyCoding`
[https://juliastats.org/StatsModels.jl/stable/contrasts/#StatsModels.DummyCoding](https://juliastats.org/StatsModels.jl/stable/contrasts/#StatsModels.DummyCoding)
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [July 17, 2021, 1:12pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/7 "2021-07-17T13:12:14Z")
> [@Mattriks](#):
>
> `unique(x) .== permutedims(x)`
Mind Blown.
```julia
julia> x = [1, 2, 1, 3, 2];
julia> unique(x) .== permutedims(x)
3×5 BitMatrix:
1 0 1 0 0
0 1 0 0 1
0 0 0 1 0
```
That is so elegant.
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [July 17, 2021, 1:23pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/8 "2021-07-17T13:23:01Z")
😲
---
### Author:  [@lostella](https://discourse.julialang.org/u/lostella)
#### Post date: [July 17, 2021, 3:30pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/9 "2021-07-17T15:30:21Z")
Brilliant! Not sure what `permutedims` does in case `x` is multidimensional, so maybe the following is slightly more general:
```julia
unique(x) .== reshape(x, (1, size(x)…))
```
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [July 17, 2021, 3:44pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/10 "2021-07-17T15:44:31Z")
how do you one-hot encode a multidimensional `x`?
* * *
It’s so nice our broadcast would do outer automatically 😉
---
### Author:  [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)
#### Post date: [July 17, 2021, 5:59pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/11 "2021-07-17T17:59:15Z")
Not in a million years would I have thought of that. 😂 Awesome 👍🏻
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [July 17, 2021, 6:37pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/12 "2021-07-17T18:37:44Z")
Flux’s `OneHotArray` has gotten a lot of work recently and is pretty robust. It’s also [self-contained](https://github.com/FluxML/Flux.jl/blob/master/src/onehot.jl), so we could theoretically pull it into a separate package if there’s enough interest.
---
### Author:  [@e3c6](https://discourse.julialang.org/u/e3c6)
#### Post date: [November 6, 2021, 12:46pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/13 "2021-11-06T12:46:19Z")
[https://github.com/cossio/OneHot.jl](https://github.com/cossio/OneHot.jl)
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [November 8, 2021, 2:52am UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/14 "2021-11-08T02:52:11Z")
a doc or simple instructions in the readme would be helpful.
---
### Author:  [@jamaas](https://discourse.julialang.org/u/jamaas)
#### Post date: [November 11, 2021, 1:43pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/15 "2021-11-11T13:43:34Z")
Ok, here is your challenge. I’m new to this and struggling to get any onehot conversion to work. This is a MWE but here are some very small DNA sequences, what is the best way to “onehot” convert this object, which I think would normally be in a dataframe, for ml? Note there are the standard 4 bases and a couple of unknown “N”'s in the reads. Thx. J
TGTCCGGCTCACCCACATAACCATATATATATATATAG
TATATAATAACCATTAACCAATATATATGGTTATGTGG
CAACATCATTAATTTANAGAGATTTTACTATGGAATAA
TTGTGTGAATNCAGATTTTCAAGGCTCAAAAGAATATT
TTTTGTTGACAGAAAAAAGAATAATCAATATACTGTAT
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [November 11, 2021, 2:05pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/16 "2021-11-11T14:05:06Z")
Not entirely sure how this fits into a DataFrame or what output you’re expecting, but one way:
```julia
julia> s = "TGTCCGGCTCACCCACATAACCATATATATATATATAGTATATAATAACCATTAACCAATATATATGGTTATGTGGCAACATCATTAATTTANAGAGATTTTACTATGGAATAATTGTGTGAATNCAGATTTTCAAGGCTCAAAAGAATATTTTTTGTTGACAGAAAAAAGAATAATCAATATACTGTAT"
"TGTCCGGCTCACCCACATAACCATATATATATATATAGTATATAATAACCATTAACCAATATATATGGTTATGTGGCAACATCATTAATTTANAGAGATTTTACTATGGAATAATTGTGTGAATNCAGATTTTCAAGGCTCAAAAGAATATTTTTTGTTGACAGAAAAAAGAATAATCAATATACTGTAT"
julia> [collect(s) .== x for x ∈ ['A', 'C', 'G', 'T', 'N']]
5-element Vector{BitVector}:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0 … 0, 1, 0, 1, 0, 0, 0, 0, 1, 0]
[0, 0, 0, 1, 1, 0, 0, 1, 0, 1 … 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
[0, 1, 0, 0, 0, 1, 1, 0, 0, 0 … 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]
[1, 0, 1, 0, 0, 0, 0, 0, 1, 0 … 1, 0, 1, 0, 0, 1, 0, 1, 0, 1]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0 … 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
```
(where the first vector is 1 when character is `A`, the second vector is 1 when the character is `C` etc)
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [November 11, 2021, 2:07pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/17 "2021-11-11T14:07:33Z")
I assume there are 5 sequences, so you can e.g. do to also keep track of the sequence ID:
```julia
julia> using DataFrames
julia> seq = reduce(hcat, collect.(["TGTCCGGCTCACCCACATAACCATATATATATATATAG"
"TATATAATAACCATTAACCAATATATATGGTTATGTGG"
"CAACATCATTAATTTANAGAGATTTTACTATGGAATAA"
"TTGTGTGAATNCAGATTTTCAAGGCTCAAAAGAATATT"
"TTTTGTTGACAGAAAAAAGAATAATCAATATACTGTAT"]))
38×5 Matrix{Char}:
'T' 'T' 'C' 'T' 'T'
'G' 'A' 'A' 'T' 'T'
'T' 'T' 'A' 'G' 'T'
'C' 'A' 'C' 'T' 'T'
'C' 'T' 'A' 'G' 'G'
'G' 'A' 'T' 'T' 'T'
'G' 'A' 'C' 'G' 'T'
'C' 'T' 'A' 'A' 'G'
'T' 'A' 'T' 'A' 'A'
'C' 'A' 'T' 'T' 'C'
'A' 'C' 'A' 'N' 'A'
'C' 'C' 'A' 'C' 'G'
'C' 'A' 'T' 'A' 'A'
⋮
'T' 'T' 'T' 'T' 'C'
'A' 'A' 'A' 'C' 'A'
'T' 'T' 'C' 'A' 'A'
'A' 'G' 'T' 'A' 'T'
'T' 'G' 'A' 'A' 'A'
'A' 'T' 'T' 'A' 'T'
'T' 'T' 'G' 'G' 'A'
'A' 'A' 'G' 'A' 'C'
'T' 'T' 'A' 'A' 'T'
'A' 'G' 'A' 'T' 'G'
'T' 'T' 'T' 'A' 'T'
'A' 'G' 'A' 'T' 'A'
'G' 'G' 'A' 'T' 'T'
julia> df = DataFrame(seq, string.("seq", 1:5))
38×5 DataFrame
Row │ seq1 seq2 seq3 seq4 seq5
│ Char Char Char Char Char
─────┼──────────────────────────────
1 │ T T C T T
2 │ G A A T T
3 │ T T A G T
4 │ C A C T T
5 │ C T A G G
6 │ G A T T T
7 │ G A C G T
8 │ C T A A G
9 │ T A T A A
10 │ C A T T C
11 │ A C A N A
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮
28 │ T T C A A
29 │ A G T A T
30 │ T G A A A
31 │ A T T A T
32 │ T T G G A
33 │ A A G A C
34 │ T T A A T
35 │ A G A T G
36 │ T T T A T
37 │ A G A T A
38 │ G G A T T
16 rows omitted
julia> long = stack(df, :)
190×2 DataFrame
Row │ variable value
│ String Char
─────┼─────────────────
1 │ seq1 T
2 │ seq1 G
3 │ seq1 T
4 │ seq1 C
5 │ seq1 C
6 │ seq1 G
7 │ seq1 G
8 │ seq1 C
9 │ seq1 T
10 │ seq1 C
11 │ seq1 A
⋮ │ ⋮ ⋮
180 │ seq5 A
181 │ seq5 T
182 │ seq5 A
183 │ seq5 T
184 │ seq5 A
185 │ seq5 C
186 │ seq5 T
187 │ seq5 G
188 │ seq5 T
189 │ seq5 A
190 │ seq5 T
168 rows omitted
julia> transform(long, :value => (x -> x .== ['T' 'G' 'C' 'A' 'N']) => [:T, :G, :C, :A, :N])
190×7 DataFrame
Row │ variable value T G C A N
│ String Char Bool Bool Bool Bool Bool
─────┼────────────────────────────────────────────────────
1 │ seq1 T true false false false false
2 │ seq1 G false true false false false
3 │ seq1 T true false false false false
4 │ seq1 C false false true false false
5 │ seq1 C false false true false false
6 │ seq1 G false true false false false
7 │ seq1 G false true false false false
8 │ seq1 C false false true false false
9 │ seq1 T true false false false false
10 │ seq1 C false false true false false
11 │ seq1 A false false false true false
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
180 │ seq5 A false false false true false
181 │ seq5 T true false false false false
182 │ seq5 A false false false true false
183 │ seq5 T true false false false false
184 │ seq5 A false false false true false
185 │ seq5 C false false true false false
186 │ seq5 T true false false false false
187 │ seq5 G false true false false false
188 │ seq5 T true false false false false
189 │ seq5 A false false false true false
190 │ seq5 T true false false false false
168 rows omitted
```
---
### Author:  [@jamaas](https://discourse.julialang.org/u/jamaas)
#### Post date: [November 11, 2021, 2:36pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/18 "2021-11-11T14:36:42Z")
Thanks for that. I’ve missed something here. I thought that the one hot coding would convert each individual letter, A, T, C, G into individual four-digit codes such as 1 0 0 0, 0 1 0 0, 0 0 1 0, and 0 0 0 1.
There may be much better ways in Julia, but I was looking for something equivalent to the keras.tensorflow function that is supposed to work like:
```julia
preprocessing.text.one_hot(
input_text = input_object, # dataframe?
n = 4, # number of individual objects to code for, A, T, C, G
filters = 'N', # filter out the N values
lower = False,
split = ' ')
```
BTW, I’ve tried this and not got it to work either, but I can see the logic. I’m happy to be told there are more precise and/or elegant ways to get the final result?
---
### Author:  [@jamaas](https://discourse.julialang.org/u/jamaas)
#### Post date: [November 11, 2021, 2:41pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/19 "2021-11-11T14:41:35Z")
Thanks a bunch, very elegant. As I just wrote in the other note I was expecting the end product to be four individual four-digit codes but my expectation might be incorrect? Also the hard coding 1:5, is that 1:5 bases or 1:5 sequences? In the actual code I’d be running it on thousands of sequences.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [November 11, 2021, 2:47pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/20 "2021-11-11T14:47:53Z")
One hot encoding generally turns a categorical variable into a group of vectors of one - your “four-digit code” essentially works row-wise:
| Sequence | A | C | G | T | N |
| --- | --- | --- | --- | --- | --- |
| ‘A’ | 1 | 0 | 0 | 0 | 0 |
| ‘T’ | 0 | 0 | 0 | 1 | 0 |
etc. - which is exactly what Bogumil shows (in my solution, the “A”, “C”, “G”, “T”, and “N” vector are the columns in the table above)
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [November 11, 2021, 10:37pm UTC](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807/21 "2021-11-11T22:37:11Z")
shouldn’t there be a specialised structure to store these. Since there are only 4 possible values so 2 bits can be used to represent each value.
I am pretty sure I’ve read about specialised data structures that store these efficiently. Just not sure if they are in Julia just yet.
[Next page](https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/64807.md?page=2)