Ngrams

Introduce probabilistic language modeling

let's thinking about the probability of sentence.

P(w1, w2, w3, w4)

= P(w1, w2, w3) * P(w4| w1, w2, w3)

= P(w1, w2) * P(w3| w1, w2) * P(w4| w1, w2, w3)

= P(w1) * P(w2| w1) * P(w3| w1, w2) * P(w4| w1, w2, w3)

Markov Assumption

 P(w1, w2, w3, w4...wn) = P(wn) * P (wn| wk, wk+1...wn)

for example

P(w1, w2, w3, w4)

= P(w1, w2, w3) * P(w4|w3)

= P(w1, w2) * P(w3|w2) * P(w4| w3)

= P(w1) * P(w2| w1) * P(w3|w2) * P(w4|w3)

Then it is easier.

And there are N-grams, 3-grams, 4-grams.