genism单词检测Phrases

转自：https://radimrehurek.com/gensim/models/phrases.html

1.gensim.models.phrases.Phraser 

 //上面链接中的样例代码复制不行，绝了。。。

>>> from gensim.test.utils import datapath
>>> from gensim.models.word2vec import Text8Corpus
>>> from gensim.models.phrases import Phrases, Phraser
>>>
>>> sentences = Text8Corpus(datapath('testcorpus.txt'))#首先加载作为分词的训练集
>>> phrases = Phrases(sentences, min_count=1, threshold=1)#
>>>
>>> bigram = Phraser(phrases)#这样训练出一个二元分词器
>>> sent = [u'trees', u'graph', u'minors']#在这个句子上测试
>>> print(bigram[sent])
[u'trees_graph', u'minors']#前两个word组合为了词组

其中参数min_count/threshold，越高那么词组越少，将单词组合成双字母组的难度就越大。

下面是三词组：

https://blog.csdn.net/weixin_33826609/article/details/91472435

# Build the bigram and trigram models
bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.
trigram = gensim.models.Phrases(bigram[data_words], threshold=100) #以双词组作为训练数据 
 
# Faster way to get a sentence clubbed as a trigram/bigram
bigram_mod = gensim.models.phrases.Phraser(bigram)
trigram_mod = gensim.models.phrases.Phraser(trigram)
 
# See trigram example
print(trigram_mod[bigram_mod[data_words[0]]])