elasticsearch-Mapping

什么是映射

类似于数据库中的表结构定义，主要作用如下：

• 定义Index下字段名（Field Name）
• 定义字段的类型，比如数值型，字符串型、布尔型等
• 定义倒排索引的相关配置，比如是否索引、记录postion等

需要注意的是，在索引中定义太多字段可能会导致索引膨胀，出现内存不足和难以恢复的情况，下面有几个设置：

• index.mapping.total_fields.limit：一个索引中能定义的字段的最大数量，默认是 1000
• index.mapping.depth.limit：字段的最大深度，以内部对象的数量来计算，默认是20
• index.mapping.nested_fields.limit：索引中嵌套字段的最大数量，默认是50

Mapping的数据类型

详见:https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

基本数据类型

属性名字	说明
text	用于全文索引，该类型的字段将通过分词器进行分词，最终用于构建索引
keyword	不分词
long	有符号64-bit integer：-2^63 ~ 2^63 - 1
integer	有符号32-bit integer，-2^31 ~ 2^31 - 1
short	有符号16-bit integer，-32768 ~ 32767
byte	有符号8-bit integer，-128 ~ 127
double	64-bit IEEE 754 浮点数
float	32-bit IEEE 754 浮点数
half_float	16-bit IEEE 754 浮点数
boolean	true,false
date	https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
binary	该类型的字段把值当做经过 base64 编码的字符串，默认不存储，且不可搜索

Mapping范围数据类型

标识一个数据范围而不是一个值  如age:10~20   搜索{"gle":5,"lte":20} 则可以搜索出来数据

支持的数据类型	说明
integer_range
float_range
long_range
double_range
date_range	64-bit 无符号整数，时间戳（单位：毫秒）
ip_range	IPV4 或 IPV6 格式的字符串

可选参数:

relation这只匹配模式

INTERSECTS 默认的匹配模式，只要搜索值与字段值有交集即可匹配到

WITHIN 字段值需要完全包含在搜索值之内，也就是字段值是搜索值的子集才搜索出来

CONTAINS 与WITHIN相反，只搜索字段值包含搜索值的文档

测试

1.添加index

put:127.0.0.1:9200/range_test

{
  "mappings": {
    "_doc": {
      "properties": {
        "count": {
          "type": "integer_range"
        },
        "create_date": {
          "type": "date_range", 
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    }
  }
}

2.添加测试数据

post:127.0.0.1:9200/range_test/_doc/1

{
  "count" : { 
    "gte" : 1,
    "lte" : 100
  },
  "create_date" : { 
    "gte" : "2019-02-1 12:00:00", 
    "lte" : "2019-03-30"
  }
}

3.测试搜索

get:127.0.0.1:9200/range_test/_doc/_search

{
    "query":{
        "term":{
            "count":5
        }
    }
}

5在1~100之间可以搜索出来

{
  "query" : {
    "range" : {
      "create_date" : { 
        "gte" : "2019-02-01",
        "lte" : "2019-03-30",
        "relation" : "within" 
      }
    }
  }
}

Mapping复杂数据类型

数组类型 Array

支持字符串 数值 object对象数组   数组元素必须为相同数据类型

对象类型 Object

{
    "name": "小明",
    "user_info": {
        "student_id": 111,
        "class_info": {
            "class_name": "1年级"
        }
    }
}

被索引形式

{
 "name":"小明",
"user_info.student_id":"111",
"user_info.student_info.class_name":"111"
}

嵌套类型 Nested

能够支持数组元素单独的做索引

查询api:https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

聚合api:https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-reverse-nested-aggregation.html

排序api:https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-reverse-nested-aggregation.html

检索和高亮:https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#nested-inner-hits

Nested和Object区别

put:127.0.0.1:9200/object_test/_doc/1 默认是object类型

{
    "user_name":"小明",
    "subjects":[
        {"subject_name":"地理","id":1},
        {"subject_name":"英语","id":2}
    ]
}

搜索名字为英语id为1的

{
    "query":{
        "bool":{
        "must":[
            {"match":{"subjects.subject_name":"英语"}},
                {"match":{"subjects.id":"1"}}
            ]
            }
    }
}

正常搜索不出来  测试时搜索出来了

因为索引为以下格式

{
 "name":"小明",
"subjects.subject_name":["英语","地理"],
"subjects.subject_id":["1","2"]
}

改为Nested 就不会

地理数据类型

geo_point

几种格式

object对象："location": {"lat": 41.12, "lon": -71.34}

字符串："location": "41.12,-71.34"

geohash："location": "drm3btev3e86"

数组："location": [ -71.34, 41.12 ]

geo_shape

查询api:https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-query.html

专用数据类型

• 记录IP地址 ip
• 实现自动补全 completion
• 记录分词数 token_count
• 记录字符串hash值 murmur3
• Percolator

Mapping设置

一个完整的mapping设置 

  {
      "settings": {
    "analysis": {
      "analyzer": {
        "ik_pinyin_analyzer": {
          "type": "custom",
          "tokenizer": "ik_smart",
          "filter": ["my_pinyin"]#自定义filter
        },
        "pinyin_analyzer": {
          "tokenizer": "shopmall_pinyin"
        },
        "first_py_letter_analyzer": {
          "tokenizer": "first_py_letter"
        },
        "full_pinyin_letter_analyzer": {
          "tokenizer": "full_pinyin_letter"
        },
        "onlyOne_analyzer": {
          "tokenizer": "onlyOne_pinyin"
        }
      },
      "tokenizer": {#自定义分词器 
        "onlyOne_pinyin": {
          "type":"pinyin",
          "keep_separate_first_letter": "false",
          "keep_first_letter":"false"
        },
        "shopmall_pinyin": {
          "keep_joined_full_pinyin": "true",
          "keep_first_letter": "true",
          "keep_separate_first_letter": "false",
          "lowercase": "true",
          "type": "pinyin",
          "limit_first_letter_length": "16",
          "keep_original": "true",
          "keep_full_pinyin": "true",
          "keep_none_chinese_in_joined_full_pinyin": "true"
        },
        "first_py_letter": {
          "type": "pinyin",
          "keep_first_letter": true,
          "keep_full_pinyin": false,
          "keep_original": false,
          "limit_first_letter_length": 16,
          "lowercase": true,
          "trim_whitespace": true,
          "keep_none_chinese_in_first_letter": false,
          "none_chinese_pinyin_tokenize": false,
          "keep_none_chinese": true,
          "keep_none_chinese_in_joined_full_pinyin": true
        },
        "full_pinyin_letter": {
          "type": "pinyin",
          "keep_separate_first_letter": false,
          "keep_full_pinyin": false,
          "keep_original": false,
          "limit_first_letter_length": 16,
          "lowercase": true,
          "keep_first_letter": false,
          "keep_none_chinese_in_first_letter": false,
          "none_chinese_pinyin_tokenize": false,
          "keep_none_chinese": true,
          "keep_joined_full_pinyin": true,
          "keep_none_chinese_in_joined_full_pinyin": true
        }
      },
      "filter": {
        "my_pinyin": {
          "type": "pinyin",
          "keep_joined_full_pinyin": true,
          "keep_separate_first_letter":true
        }
      }
    }

  },
      "mappings": {
    "doc": {#type名字
      "properties": {#mapping的属性
        "productName": {属性名字
          "type": "text",#属性类型
          "analyzer": "ik_pinyin_analyzer",#分词器
          "fields": {#fields 指定自定义分词器 查询时通过productName.keyword_once_pinyin 可以指定
            "keyword_once_pinyin": {
              "type": "text",
              "analyzer": "onlyOne_analyzer"#指定的自定义分词器
            }
          }
        },
        "skuNames": {
          "type": "text",
          "analyzer": "ik_pinyin_analyzer",
          "fields": {
            "keyword_once_pinyin": {
              "type": "text",
              "analyzer": "onlyOne_analyzer"
            }
          }
        },
        "regionCode": {
          "type": "keyword"
        },
        "productNameSuggester": {#es6.x搜索建议实现
          "type": "completion",
          "fields": {
            "pinyin": {
              "type": "completion",
              "analyzer": "pinyin_analyzer"
            },
            "keyword_pinyin": {
              "type": "completion",
              "analyzer": "full_pinyin_letter_analyzer"
            },
            "keyword_first_py": {
              "type": "completion",
              "analyzer": "first_py_letter_analyzer"
            }
          }
        }
          "info": {#es6父子类型设置
          "type": "join",
          "relations": {
            "md_product":[ "sl_customer_character_order_list","ic_product_store_account","sl_customer_product_setting"]
          }
        }
      }
    }
  }
  }

创建mapping

put:http://127.0.0.1:9200/db

{
    "mappings": {
        "product": {//type
            "properties": {
                "productName": {//字段
                    "type": "text"//数据类型
                }
            }
        }
    }
}

mapping参数

参数	说明
analyzer	分词器 默认:standard
boost	字段权重默认1 在通过_all字段查询 根据此字段来权重
dynamic	控制字段新增 true(默认 允许新增) false  strict 不能新增文档
index	控制字段是否索引(可搜索) true 是 false否 可以节省 磁盘空间

参考:https://www.jianshu.com/p/e8a9feea683c

新增mapping字段

Elasticsearch的mapping一旦创建，只能增加字段，而不能修改已经mapping的字段

put http://127.0.0.1:9200/{indexName}/_mapping/{typeName}

{
  "properties": {
    "productSortItemIds": { #字段名字
     "type": "string",#类型
      "store": true, #是否持久化
      "analyzer": "comma", #分词器
      "search_analyzer": "comma" #搜索分词器
    }
  }
}

修改字段类型

es不支持修改字段类型,因为es是根据lucene生成倒排索引 一旦生成不允许修改,如果需要修改 需要使用reindex重建索引 如果修改了类型会导致无法搜索,所以不支持,新增字段不受影响

新增分词器

 post /{index}/_close #关闭索引
 put  /bbc_product/_settings #增加,号分词器
{
  "settings": {
    "analysis": {
      "analyzer": {
          "comma": {
                 "type": "pattern",
                 "pattern":","
         }
      }
    }
  }
}
post /{index}/_open #开启索引

查看当前索引的映射

http://127.0.0.1:9200/blogs2/product/_mapping 不加/_mapping可看整个index设置

{
    "blogs2": {
        "mappings": {
            "product": {
                "properties": {
                    "price": {
                        "type": "long"
                    },
                    "productName": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "remark": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "tags": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    }
                }
            }
        }
    }
}

自定义映射

作用定义数据类型 比如数字映射成text 大于小于范围搜索就会无效 还有明确哪些fullText需要分词哪些不需要分词

确切值(Exact values)和全文本(FullText)

es支持很多种数据类型但是主要分为2大类
确切值就是能够确定的值 比如id 日期  通过=就能查询到我们想要的数据

而全文本是需要进行相似度匹配 返回最佳匹配

Template Mapping

用于设置一个模板,避免新建索引 漏设置一些核心参数,当创建maping会自动将模板索引的设置根据优先级设置到创建的mapping

优先级 当前设置>模板maping order由小到大>默认

#数字字符串被映射成text，日期字符串被映射成日期
PUT ttemplate/_doc/1
{
    "someNumber":"1",
    "someDate":"2019/01/01"
}
GET ttemplate/_mapping


#Create a default template
PUT _template/template_default
{
  "index_patterns": ["*"],
  "order" : 0,
  "version": 1,
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas":1
  }
}


PUT /_template/template_test
{
    "index_patterns" : ["test*"],
    "order" : 1,
    "settings" : {
        "number_of_shards": 1,
        "number_of_replicas" : 2
    },
    "mappings" : {
        "date_detection": false,
        "numeric_detection": true
    }
}

#查看template信息
GET /_template/template_default
GET /_template/temp*


#写入新的数据，index以test开头
PUT testtemplate/_doc/1
{
    "someNumber":"1",
    "someDate":"2019/01/01"
}
GET testtemplate/_mapping
get testtemplate/_settings

PUT testmy
{
    "settings":{
        "number_of_replicas":5
    }
}

put testmy/_doc/1
{
  "key":"value"
}

get testmy/_settings
DELETE testmy
DELETE /_template/template_default
DELETE /_template/template_test



#Dynaminc Mapping 根据类型和字段名
DELETE my_index

PUT my_index/_doc/1
{
  "firstName":"Ruan",
  "isVIP":"true"
}

GET my_index/_mapping
DELETE my_index
PUT my_index
{
  "mappings": {
    "dynamic_templates": [
            {
        "strings_as_boolean": {
          "match_mapping_type":   "string",
          "match":"is*",
          "mapping": {
            "type": "boolean"
          }
        }
      },
      {
        "strings_as_keywords": {
          "match_mapping_type":   "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}


DELETE my_index
#结合路径
PUT my_index
{
  "mappings": {
    "dynamic_templates": [
      {
        "full_name": {
          "path_match":   "name.*",
          "path_unmatch": "*.middle",
          "mapping": {
            "type":       "text",
            "copy_to":    "full_name"
          }
        }
      }
    ]
  }
}


PUT my_index/_doc/1
{
  "name": {
    "first":  "John",
    "middle": "Winston",
    "last":   "Lennon"
  }
}

GET my_index/_search?q=full_name:John