Elastic Search 分词器的介绍和使用

分词器的介绍和使用

什么是分词器?

将用户输入的一段文本，按照一定逻辑，分析成多个词语的一种工具

常用的内置分词器

standard analyzer、simple analyzer、whitespace analyzer、stop analyzer、language analyzer、pattern analyzer

standard analyzer

标准分析器是默认分词器，如果未指定，则使用该分词器。

POST localhost:9200/_analyze

参数：
{
    "analyzer":"standard",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "the",
            "start_offset": 0,
            "end_offset": 3,
            "type": "<ALPHANUM>",
            "position": 0
        },
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 1
        },
        {
            "token": "3",
            "start_offset": 9,
            "end_offset": 10,
            "type": "<NUM>",
            "position": 2
        },
        {
            "token": "points",
            "start_offset": 11,
            "end_offset": 17,
            "type": "<ALPHANUM>",
            "position": 3
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "<ALPHANUM>",
            "position": 4
        },
        {
            "token": "is",
            "start_offset": 26,
            "end_offset": 28,
            "type": "<ALPHANUM>",
            "position": 5
        },
        {
            "token": "curry",
            "start_offset": 29,
            "end_offset": 34,
            "type": "<ALPHANUM>",
            "position": 6
        }
    ]
}

simple analyzer

simple 分析器当它遇到只要不是字母的字符，就将文本解析成 term，而且所有的 term 都是小写的。

POST localhost:9200/_analyze

参数：
{
    "analyzer":"simple",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "the",
            "start_offset": 0,
            "end_offset": 3,
            "type": "word",
            "position": 0
        },
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "word",
            "position": 1
        },
        {
            "token": "points",
            "start_offset": 11,
            "end_offset": 17,
            "type": "word",
            "position": 2
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "word",
            "position": 3
        },
        {
            "token": "is",
            "start_offset": 26,
            "end_offset": 28,
            "type": "word",
            "position": 4
        },
        {
            "token": "curry",
            "start_offset": 29,
            "end_offset": 34,
            "type": "word",
            "position": 5
        }
    ]
}

whitespace analyzer

whitespace 分析器，当它遇到空白字符时，就将文本解析成terms

POST localhost:9200/_analyze

参数：
{
    "analyzer":"whitespace",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "The",
            "start_offset": 0,
            "end_offset": 3,
            "type": "word",
            "position": 0
        },
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "word",
            "position": 1
        },
        {
            "token": "3-points",
            "start_offset": 9,
            "end_offset": 17,
            "type": "word",
            "position": 2
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "word",
            "position": 3
        },
        {
            "token": "is",
            "start_offset": 26,
            "end_offset": 28,
            "type": "word",
            "position": 4
        },
        {
            "token": "Curry!",
            "start_offset": 29,
            "end_offset": 35,
            "type": "word",
            "position": 5
        }
    ]
}

stop analyzer

stop 分析器 和 simple 分析器很像，唯一不同的是，stop 分析器增加了对删除停止词的支持，默认使用了 english 停止词

stopwords 预定义的停止词列表，比如 (the,a,an,this,of,at)等等

POST localhost:9200/_analyze

参数：
{
    "analyzer":"stop",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "word",
            "position": 1
        },
        {
            "token": "points",
            "start_offset": 11,
            "end_offset": 17,
            "type": "word",
            "position": 2
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "word",
            "position": 3
        },
        {
            "token": "curry",
            "start_offset": 29,
            "end_offset": 34,
            "type": "word",
            "position": 5
        }
    ]
}

language analyzer

（特定的语言的分词器，比如说，english，英语分词器)，内置语言：arabic，armenian，basque，bengali，brazilian，bulgarian，catalan，cjk，czech，danish，dutch，english，finnish，french，galician，german，greek，hindi，hungarian，indonesian，irish，italian，latvian，lithuanian，norwegian，persian，portuguese，romanian，russian，sorani，spanish，swedish，turkish，thai

POST localhost:9200/_analyze

参数：
{
    "analyzer":"english",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "<ALPHANUM>",
            "position": 1
        },
        {
            "token": "3",
            "start_offset": 9,
            "end_offset": 10,
            "type": "<NUM>",
            "position": 2
        },
        {
            "token": "point",
            "start_offset": 11,
            "end_offset": 17,
            "type": "<ALPHANUM>",
            "position": 3
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "<ALPHANUM>",
            "position": 4
        },
        {
            "token": "curri",
            "start_offset": 29,
            "end_offset": 34,
            "type": "<ALPHANUM>",
            "position": 6
        }
    ]
}

pattern analyzer

用正则表达式来将文本分割成terms，默认的正则表达式是W+（非单词字符）

POST localhost:9200/_analyze

参数：
{
    "analyzer":"pattern",
    "text":"The best 3-points shooter is Curry!"
}

返回值：
{
    "tokens": [
        {
            "token": "the",
            "start_offset": 0,
            "end_offset": 3,
            "type": "word",
            "position": 0
        },
        {
            "token": "best",
            "start_offset": 4,
            "end_offset": 8,
            "type": "word",
            "position": 1
        },
        {
            "token": "3",
            "start_offset": 9,
            "end_offset": 10,
            "type": "word",
            "position": 2
        },
        {
            "token": "points",
            "start_offset": 11,
            "end_offset": 17,
            "type": "word",
            "position": 3
        },
        {
            "token": "shooter",
            "start_offset": 18,
            "end_offset": 25,
            "type": "word",
            "position": 4
        },
        {
            "token": "is",
            "start_offset": 26,
            "end_offset": 28,
            "type": "word",
            "position": 5
        },
        {
            "token": "curry",
            "start_offset": 29,
            "end_offset": 34,
            "type": "word",
            "position": 6
        }
    ]
}

使用示例

PUT localhost:9200/my_index

参数：
{
    "settings":{
        "analysis":{
            "analyzer":{
                "my_analyzer":{
                    "type":"whitespace"
                }
            }
        }
    },
    "mappings":{
        "properties":{
            "name":{
                "type":"text"
            },
            "team_name":{
                "type":"text"
            },
            "position":{
                "type":"text"
            },
            "play_year":{
                "type":"long"
            },
            "jerse_no":{
                "type":"keyword"
            },
            "title":{
                "type":"text",
                "analyzer":"my_analyzer"
            }
        }
    }
}

返回值：
{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "my_index"
}

PUT localhost:9200/my_index/_doc/1

参数：
{
    "name":"库⾥里里",
    "team_name":"勇⼠士",
    "position":"控球后卫",
    "play_year":10,
    "jerse_no":"30",
    "title":"The best 3-points shooter is Curry!"
}

返回值：
{
    "_index": "my_index",
    "_type": "_doc",
    "_id": "1",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 0,
    "_primary_term": 1
}

POST localhost:9200/my_index/_search

参数：
{
    "query":{
        "match":{
            "title":"Curry!"
        }
    }
}

返回值：
{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.2876821,
        "hits": [
            {
                "_index": "my_index",
                "_type": "_doc",
                "_id": "1",
                "_score": 0.2876821,
                "_source": {
                    "name": "库⾥里里",
                    "team_name": "勇⼠士",
                    "position": "控球后卫",
                    "play_year": 10,
                    "jerse_no": "30",
                    "title": "The best 3-points shooter is Curry!"
                }
            }
        ]
    }
}

常见中文分词器的使用

如果用默认的分词器 standard 进行中文分词，中文会被单独分成一个个汉字

POST localhost:9200/_analyze

参数：
{
    "analyzer": "standard",
    "text": "火箭明年总冠军"
}

返回值：
{
    "tokens": [
        {
            "token": "火",
            "start_offset": 0,
            "end_offset": 1,
            "type": "<IDEOGRAPHIC>",
            "position": 0
        },
        {
            "token": "箭",
            "start_offset": 1,
            "end_offset": 2,
            "type": "<IDEOGRAPHIC>",
            "position": 1
        },
        {
            "token": "明",
            "start_offset": 2,
            "end_offset": 3,
            "type": "<IDEOGRAPHIC>",
            "position": 2
        },
        {
            "token": "年",
            "start_offset": 3,
            "end_offset": 4,
            "type": "<IDEOGRAPHIC>",
            "position": 3
        },
        {
            "token": "总",
            "start_offset": 4,
            "end_offset": 5,
            "type": "<IDEOGRAPHIC>",
            "position": 4
        },
        {
            "token": "冠",
            "start_offset": 5,
            "end_offset": 6,
            "type": "<IDEOGRAPHIC>",
            "position": 5
        },
        {
            "token": "军",
            "start_offset": 6,
            "end_offset": 7,
            "type": "<IDEOGRAPHIC>",
            "position": 6
        }
    ]
}

常见分词器

smartCN分词器：一个简单的中文或中英文混合文本的分词器

IK分词器：更智能更友好的中文分词器

smartCn

安装，进入到bin目录

• Linux 使用命令：sh elasticsearch-plugin install analysis-smartcn
• Windows 使用命令：.elasticsearch-plugin install analysis-smartcn

安装后需要重新启动

POST localhost:9200/_analyze

参数：
{
    "analyzer": "smartcn",
    "text": "火箭明年总冠军"
}

返回值：
{
    "tokens": [
        {
            "token": "火箭",
            "start_offset": 0,
            "end_offset": 2,
            "type": "word",
            "position": 0
        },
        {
            "token": "明年",
            "start_offset": 2,
            "end_offset": 4,
            "type": "word",
            "position": 1
        },
        {
            "token": "总",
            "start_offset": 4,
            "end_offset": 5,
            "type": "word",
            "position": 2
        },
        {
            "token": "冠军",
            "start_offset": 5,
            "end_offset": 7,
            "type": "word",
            "position": 3
        }
    ]
}

IK分词器

下载：https://github.com/medcl/elasticsearch-analysis-ik/releases

安装：解压安装到 plugins 目录

安装后重新启动

POST localhost:9200/_analyze

参数：
{
    "analyzer": "ik_max_word",
    "text": "火箭明年总冠军"
}

返回值：
{
    "tokens": [
        {
            "token": "火箭",
            "start_offset": 0,
            "end_offset": 2,
            "type": "CN_WORD",
            "position": 0
        },
        {
            "token": "明年",
            "start_offset": 2,
            "end_offset": 4,
            "type": "CN_WORD",
            "position": 1
        },
        {
            "token": "总冠军",
            "start_offset": 4,
            "end_offset": 7,
            "type": "CN_WORD",
            "position": 2
        },
        {
            "token": "冠军",
            "start_offset": 5,
            "end_offset": 7,
            "type": "CN_WORD",
            "position": 3
        }
    ]
}