# Character n-grams in array

**URL:** <https://discourse.julialang.org/t/character-n-grams-in-array/50630>\
**Category:** General Usage\
**Created:** [November 23, 2020, 2:01pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630 "2020-11-23T14:01:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![lorb24](https://avatars.discourse-cdn.com/v4/letter/l/ecae2f/32.png) [@lorb24](https://discourse.julialang.org/u/lorb24)\
**Post date:** [November 23, 2020, 2:01pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/1 "2020-11-23T14:01:55Z")

</div>

I’m trying to get the character n-grams for a list of words. For example:

```julia
julia> s = "apple"
julia> s=reduce(vcat, permutedims.(collect.(s)))
['a', 'p', 'p', 'l', 'e']
julia> ngram(s,n) = join([view(s,i:i+n-1) for i=1:length(s)-n+1])
julia> ngram(s,2)
"['a','p'],['p','p'],['p','l'],['l','e']"

```

If I have a list, let’s say:

```julia
julia> s = ["apple"
"orange"
"pear"
"honeycrisp apple"
];

#split_char here is equivalent to 2nd equation in above example
julia> function split_char(s)
    s= collect.(s)
    end
    s
end
julia> split_char(s) 
[['a', 'p', 'p', 'l', 'e'], ['o', 'r', 'a', 'n', 'g', 'e'], ['p', 'e', 'a', 'r'], ['h', 'o', 'n', 'e', 'y', 'c', 'r', 'i', 's', 'p', ' ', 'a', 'p', 'p', 'l', 

```

Then passing it through this n-gram function is where nothing really happens:

```julia
julia> function ngram(s,n)
    s= [view(s[i],i:i+n-1) for i=1:length(s)-n+1]
end

```

My goal is to get this as a result:

```julia
> julia> "['a','p'],['p','p'],['p','l'],['l','e'], ['o','r'],['r','a'],['a','n'],['n','g'],['g','e'], ......]

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 23, 2020, 2:45pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/2 "2020-11-23T14:45:04Z")

</div>

See this discussion: [nlp - Generate ngrams with Julia - Stack Overflow](https://stackoverflow.com/questions/42360957/generate-ngrams-with-julia)

---

<div class="post-metadata">

**Author:** ![lorb24](https://avatars.discourse-cdn.com/v4/letter/l/ecae2f/32.png) [@lorb24](https://discourse.julialang.org/u/lorb24)\
**Post date:** [November 23, 2020, 6:06pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/3 "2020-11-23T18:06:12Z")

</div>

Hey, thanks for the link! I’m actually using the code in the last comment of this thread (also found [here](https://nextjournal.com/jbowles/n-gram-models-part-1)).

My goal is to get separate character-level n-grams for each word in a list (i.e. if I have an array of 4 words, then I’m trying to get a 4-element result of character-level n-grams).

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [November 24, 2020, 4:24pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/4 "2020-11-24T16:24:20Z")

</div>

```julia
julia> function split_char(s)
           s= collect.(s)
       end
split_char (generic function with 1 method)

julia> function ngram(s,n)
           [[view(s[j],i:i+n-1) for i=1:length(s[j])-n+1] for j in 1:length(s) ]
       end
ngram (generic function with 1 method)

julia> s = ["apple"
       "orange"
       "pear"
       "honeycrisp apple"
       ];

julia> ss=split_char(s)
4-element Array{Array{Char,1},1}:
 ['a', 'p', 'p', 'l', 'e']
 ['o', 'r', 'a', 'n', 'g', 'e']
 ['p', 'e', 'a', 'r']
 ['h', 'o', 'n', 'e', 'y', 'c', 'r', 'i', 's', 'p', ' ', 'a', 'p', 'p', 'l', 'e']

julia> ngrams = ngram(ss,2)
4-element Array{Array{SubArray{Char,1,Array{Char,1},Tuple{UnitRange{Int64}},true},1},1}:
 [['a', 'p'], ['p', 'p'], ['p', 'l'], ['l', 'e']]
 [['o', 'r'], ['r', 'a'], ['a', 'n'], ['n', 'g'], ['g', 'e']]
 [['p', 'e'], ['e', 'a'], ['a', 'r']]
 [['h', 'o'], ['o', 'n'], ['n', 'e'], ['e', 'y'], ['y', 'c'], ['c', 'r'], ['r', 'i'], ['i', 's'], ['s', 'p'], ['p', ' '], [' ', 'a'], ['a', 'p'], ['p', 'p'], ['p', 'l'], ['l', 'e']]

```

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [November 24, 2020, 5:41pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/5 "2020-11-24T17:41:21Z")

</div>

> [@oheil](#):
>
> ```julia-auto
> julia> function split_char(s)
> s= collect.(s)
> end
> 
> ```

Some reason for the `s=`?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [November 24, 2020, 5:42pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/6 "2020-11-24T17:42:07Z")

</div>

> [@Henrique\_Becker](#):
>
> Some reason for the `s=` ?

No, just copy&paste from OP 🙂

---

<div class="post-metadata">

**Author:** ![cchderrick](https://avatars.discourse-cdn.com/v4/letter/c/ecd19e/32.png) [@cchderrick](https://discourse.julialang.org/u/cchderrick)\
**Post date:** [November 24, 2020, 8:28pm UTC](https://discourse.julialang.org/t/character-n-grams-in-array/50630/7 "2020-11-24T20:28:37Z")

</div>

This is not what you ask for, but I would still like to showcase the loop fusion and substring features.

```julia
julia> ngram(s::AbstractString,n) = [SubString(s,i:i+n-1) for i=1:length(s)-n+1]
ngram (generic function with 1 method)

julia> s = ["apple"
       "orange"
       "pear"
       "honeycrisp apple"
       ];

julia> ngram.(s,2)
4-element Array{Array{SubString{String},1},1}:
 ["ap", "pp", "pl", "le"]
 ["or", "ra", "an", "ng", "ge"]
 ["pe", "ea", "ar"]
 ["ho", "on", "ne", "ey", "yc", "cr", "ri", "is", "sp", "p ", " a", "ap", "pp", "pl", "le"]

julia> ngram.(s,3)
4-element Array{Array{SubString{String},1},1}:
 ["app", "ppl", "ple"]
 ["ora", "ran", "ang", "nge"]
 ["pea", "ear"]
 ["hon", "one", "ney", "eyc", "ycr", "cri", "ris", "isp", "sp ", "p a", " ap", "app", "ppl", "ple"]

```

And if you really want array of characters:

```julia
julia> function split_char(s)
           s= collect.(s)
           end
split_char (generic function with 1 method)

julia> bigrams = ngram.(s,2)
4-element Array{Array{SubString{String},1},1}:
 ["ap", "pp", "pl", "le"]
 ["or", "ra", "an", "ng", "ge"]
 ["pe", "ea", "ar"]
 ["ho", "on", "ne", "ey", "yc", "cr", "ri", "is", "sp", "p ", " a", "ap", "pp", "pl", "le"]

julia> split_char.(bigrams)
4-element Array{Array{Array{Char,1},1},1}:
 [['a', 'p'], ['p', 'p'], ['p', 'l'], ['l', 'e']]
 [['o', 'r'], ['r', 'a'], ['a', 'n'], ['n', 'g'], ['g', 'e']]
 [['p', 'e'], ['e', 'a'], ['a', 'r']]
 [['h', 'o'], ['o', 'n'], ['n', 'e'], ['e', 'y'], ['y', 'c'], ['c', 'r'], ['r', 'i'], ['i', 's'], ['s', 'p'], ['p', ' '], [' ', 'a'], ['a', 'p'], ['p', 'p'], ['p', 'l'], ['l', 'e']]

```