# NLP Help needed. Probability two words to be next to each other

**URL:** <https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993>\
**Category:** General Usage\
**Tags:** nlp\
**Created:** [March 14, 2020, 11:42pm UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993 "2020-03-14T23:42:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [March 14, 2020, 11:42pm UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/1 "2020-03-14T23:42:44Z")

</div>

Would somebody help me with NLP?  
I am new on this topic.

I have several words: s1,s2,s3.  
I need to find which of these words have a higher probability of staying after the root word.

So I want to compare Probability(“some words”,s1) \<\>Probability(“some words”,s2)…

I guess that I have to use word2vec or GLove, but not sure how to implement it here.

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [March 15, 2020, 2:01am UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/2 "2020-03-15T02:01:18Z")

</div>

Take a look at  
[https://juliatext.github.io/TextAnalysis.jl](https://juliatext.github.io/TextAnalysis.jl)

---

<div class="post-metadata">

**Author:** ![dellison](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dellison](https://discourse.julialang.org/u/dellison)\
**Post date:** [March 15, 2020, 3:09am UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/3 "2020-03-15T03:09:35Z")

</div>

If you’re estimating these probabilities from a corpus, then a simple (if a little old-fashioned) way to calculate these can be from an [n-gram language model](https://web.stanford.edu/~jurafsky/slp3/3.pdf). If you’d like to try it out, I’ve written a little [Julia package](https://github.com/dellison/NGrams.jl) to do create these easily! I haven’t registered it, but you can install it from the REPL:

```julia
julia> ]add https://github.com/dellison/NGrams.jl

```

Here’s a simple example of how you might use it:

```julia
using NGrams

corpus = """
Two roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;

Then took the other, as just as fair,
And having perhaps the better claim,
Because it was grassy and wanted wear;
Though as for that the passing there
Had worn them really about the same,

And both that morning equally lay
In leaves no step had trodden black.
Oh, I kept the first for another day!
Yet knowing how way leads on to way,
I doubted if I should ever come back.

I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.
"""

# get a vector of lowercased tokens, ignoring punctuation
tokens = lowercase.((m -> m.match).(eachmatch(r"\w+", corpus)))

# bigram (2-gram) language model
model = NGrams.LanguageModel(2)

NGrams.observe!(model, tokens)

# calculate probabilities of words coming after "i"
NGrams.prob(model, "i", "could") # p(could|i) = 2/9 = 0.22
NGrams.prob(model, "i", "never") # p(never|i) = 0/9 = 0.0

```

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [March 15, 2020, 3:13am UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/4 "2020-03-15T03:13:23Z")

</div>

Thank you. Yes, it’s easy. I don’t have any corpus.  
The question is about General English Corpus.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [March 15, 2020, 3:14am UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/5 "2020-03-15T03:14:03Z")

</div>

Unfortunately, [https://juliatext.github.io/TextAnalysis.jl](https://juliatext.github.io/TextAnalysis.jl) can’t help here.

---

<div class="post-metadata">

**Author:** ![dellison](https://avatars.discourse-cdn.com/v4/letter/d/ce73a5/32.png) [@dellison](https://discourse.julialang.org/u/dellison)\
**Post date:** [March 15, 2020, 3:20am UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/6 "2020-03-15T03:20:49Z")

</div>

You have to calculate probabilities without having any data?

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [March 15, 2020, 3:47pm UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/7 "2020-03-15T15:47:38Z")

</div>

It,s supposed to use word2vec or GLoVe, but I am not very confident.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 15, 2020, 4:31pm UTC](https://discourse.julialang.org/t/nlp-help-needed-probability-two-words-to-be-next-to-each-other/35993/8 "2020-03-15T16:31:35Z")

</div>

To calculate collocation probabilities using word2vec or GloVE.  
You do

```julia
#usinf pretrained embeddings
input_embedding(w)::Vector = ...
output_embedding(w)::Vector = ...
vocab=...

# rough estimate:
score(w1, w2) = exp(dot(input_embedding(w1)* output_embedding(w2)))

p(w1 around w2) = score(w1, w2)/sum(score(u,v) for u in vocab for v in vocab)

```

If I recall correctly.  
But it’s only for a kinda loose definition of “near”, not for next to.  
And you need pretrained input and output embeddings, and most things (including iirc Embeddings.jl) only give you input embeddings.

You can make a worse approximation by assuming output embeddings are equal to the input embeddings.  
Which is probably okayish