zoukankan      html  css  js  c++  java
  • 【原创】大数据基础之ElasticSearch(2)常用API整理

    Fortunately, Elasticsearch provides a very comprehensive and powerful REST API that you can use to interact with your cluster. Among the few things that can be done with the API are as follows:

    • Check your cluster, node, and index health, status, and statistics
    • Administer your cluster, node, and index data and metadata
    • Perform CRUD (Create, Read, Update, and Delete) and search operations against your indexes
    • Execute advanced search operations such as paging, sorting, filtering, scripting, aggregations, and many others

    es提供了一套容易理解并且强大的rest api接口,通过该接口你可以和集群进行交互,完成各种操作:检查集群状态、管理集群、对索引做CRUD操作、查询索引;

    Rest API Pattern:

    <REST Verb> /<Index>/<Type>/<ID>[?pretty|v]

    ps:Type相当于Category或者Partition的概念;Type未来会被废弃掉;

    The pretty parameter, again, just tells Elasticsearch to return pretty-printed JSON results.

    所有返回json接口都可以增加pretty参数,这样返回的json是格式化的;

    Each of the commands accepts a query string parameter v to turn on verbose output.

    v参数意味着详细输出;

    以下通过CURL请求,关于CURL详见:https://www.cnblogs.com/barneywill/p/10279555.html

    一  集群相关

    1 查看健康情况

    # curl http://$es_server:9200/_cat/health?v
    epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
    1547990539 21:22:19 elasticsearch green 3 3 10 5 0 0 0 0 - 100.0%

    2 查看节点

    # curl http://$es_server:9200/_cat/nodes?v
    ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    server1 29 74 1 0.07 0.10 0.13 mdi * 3iLMMxu
    server2 45 74 1 0.11 0.11 0.13 mdi - vz1k1MB
    server3 47 75 1 0.08 0.07 0.08 mdi - vGUu-b6

    3 查看master

    # curl 'http://$es_server:9200/_cat/master?v'
    id host ip node
    3iLMMxuCTISHPJaVo6I4SA server1 server1 3iLMMxu

    4 查看所有索引

    # curl http://$es_server:9200/_cat/indices?v
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open testdoc GFZhtn6GSMy2pPPj8UK70Q 5 1 1 0 8.9kb 4.4kb

    5 查看节点状态

    # curl -XGET 'http://localhost:9200/_nodes/stats?pretty'

    二 索引相关

    1 建立新索引

    # curl -XPUT 'http://$es_server:9200/testdoc/'
    {"acknowledged":true,"shards_acknowledged":true,"index":"testdoc"}

    2 删除索引

    # curl -XDELETE 'http://$es_server:9200/testdoc/'

    3 查看shards

    # curl http://localhost:9200/_cat/shards

    三 文档相关

    1 插入单个文档

    # curl -XPUT 'http://localhost:9200/testdoc/testtype/1' -d '{"name":"test"}'
    {"_index":"testdoc","_type":"testtype","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}

    如果报错:

    {"error":"Incorrect HTTP method for uri [/testdoc/testtype] and method [PUT], allowed: [POST]","status":405}

    添加header

    -H 'Content-Type: application/json'

    2 查询单个文档

    # curl -XGET 'http://$es_server:9200/testdoc/testtype/1'
    {"_index":"testdoc","_type":"testtype","_id":"1","_version":1,"found":true,"_source":{"name":"test"}}

    3 修改单个文档

    1)使用相同的id和不同的数据再调用一次

    # curl -XPUT 'http://$es_server:9200/testdoc/testtype/1' -d '{"name":"test hello"}'
    {"_index":"testdoc","_type":"testtype","_id":"1","_version":2,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":2}

    2)通过update

    # curl -XPOST 'http://$es_server:9200/testdoc/testtype/1/_update' -d '{"doc":{"name":"test hello again"}}'
    {"_index":"testdoc","_type":"testtype","_id":"1","_version":3,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2,"_primary_term":2}

    4 删除单个文档

    # curl -XDELETE 'http://$es_server:9200/testdoc/testtype/1'

    5 批量文档操作接口

    同时进行两个插入一个修改一个删除

    # curl -XPOST 'http://$es_server:9200/testdoc/testtype/1/_bulk' -d '
    {"index":{"_id":"3"}}
    {"name": "John Doe" }
    {"index":{"_id":"4"}}
    {"name": "Jane Doe" }
    {"update":{"_id":"1"}}
    {"doc": { "name": "John Doe becomes Jane Doe" } }
    {"delete":{"_id":"2"}}'

    6 查询所有文档

    以下两种请求等价

    # curl -XGET 'http://$es_server:9200/testdoc/_search?q=*'
    {"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"testdoc","_type":"testtype","_id":"1","_score":1.0,"_source":{"name":"test hello again"}}]}}

    # curl -XPOST 'http://$es_server:9200/testdoc/_search' -d '{"query":{"match_all":{}}}'

    7 查询count总数

    # curl http://localhost:9200/testdoc/_count

    8 通过条件查询count

    # curl http://localhost:9200/testdoc/_count?q=name:hello

    # curl http://localhost:9200/testdoc/_count?q=name:hello%20AND%20age:10

    注意url传递query时如果有多个field,需要使用AND或OR连接,同时空格替换为编码%20

    9 sql查询

    # curl -XPOST -H 'Content-Type: application/json' 'http://$es_server:9200/_xpack/sql?format=txt' -d '{"query":"select * from testdoc"}'
    name
    ----------------
    test hello again

    四 Setting相关

    1 查看一个索引的setting

    # curl -XGET 'http://localhost:9200/testdoc/_settings'

    2 查看所有setting

    # curl -XGET 'http://localhost:9200/_all/_settings'

    五 Mapping相关

    Mapping(索引结构定义)类似于表结构定义,定义所有的字段、数据类型、是否存储、是否索引、analyzer等;

    Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. For instance, use mappings to define:

    • which string fields should be treated as full text fields.
    • which fields contain numbers, dates, or geolocations.
    • whether the values of all fields in the document should be indexed into the catch-all _all field.
    • the format of date values.
    • custom rules to control the mapping for dynamically added fields.

    1 查看单个索引的mapping

    # curl http://localhost:9200/testdoc/_mapping/testtype
    {"testdoc":{"mappings":{"testtype":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

    2 查看所有的索引的mapping

    # curl http://localhost:9200/_mapping
    # curl http://localhost:9200/_all/_mapping

    3 在已有mapping上添加字段

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/testdoc/_mapping/testtype -d '
    {
      "properties": {
        "email": {
          "type": "keyword"
        }
      }
    }'

    4 设置mapping

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/testdoc -d '
    {
      "mappings": {
        "testtype": { 
          "properties": { 
            "title":    { "type": "text", "analyzer": "standard"}, 
            "name":     { "type": "text"  }, 
            "age":      { "type": "integer" },  
            "created":  {
              "type":   "date", 
              "format": "strict_date_optional_time||epoch_millis"
            }
          }
        }
      }
    }'

    5 更新mapping

    mapping无法更新,只能使用新的mapping创建新的索引,然后重建索引来间接实现mapping更新;

    Other than where documented, existing field mappings cannot be updated. Changing the mapping would mean invalidating already indexed documents. Instead, you should create a new index with the correct mappings and reindex your data into that index. If you only wish to rename a field and not change its mappings, it may make sense to introduce an alias field.

    参考:https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

    六 Analyzer相关

    Analysis is the process of converting text, like the body of any email, into tokens or terms which are added to the inverted index for searching. Analysis is performed by an analyzer which can be either a built-in analyzer or a custom analyzer defined per index.

    analyzer在mapping中配置,比如

    "title": { "type": "text", "analyzer": "standard"},

    测试analyzer

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_analyze?pretty -d '{"tokenizer":"standard","filter":  [ "lowercase", "asciifolding" ],"text":      "Is this chandler?"}'
    {
      "tokens" : [
        {
          "token" : "is",
          "start_offset" : 0,
          "end_offset" : 2,
          "type" : "<ALPHANUM>",
          "position" : 0
        },
        {
          "token" : "this",
          "start_offset" : 3,
          "end_offset" : 7,
          "type" : "<ALPHANUM>",
          "position" : 1
        },
        {
          "token" : "chandler",
          "start_offset" : 8,
          "end_offset" : 16,
          "type" : "<ALPHANUM>",
          "position" : 2
        }
      ]
    }
    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_analyze?pretty -d '{"tokenizer":"standard","text":"联想是全球最大的笔记本厂商"}'
    {
      "tokens" : [
        {
          "token" : "",
          "start_offset" : 0,
          "end_offset" : 1,
          "type" : "<IDEOGRAPHIC>",
          "position" : 0
        },
        {
          "token" : "",
          "start_offset" : 1,
          "end_offset" : 2,
          "type" : "<IDEOGRAPHIC>",
          "position" : 1
        },
        {
          "token" : "",
          "start_offset" : 2,
          "end_offset" : 3,
          "type" : "<IDEOGRAPHIC>",
          "position" : 2
        },
        {
          "token" : "",
          "start_offset" : 3,
          "end_offset" : 4,
          "type" : "<IDEOGRAPHIC>",
          "position" : 3
        },
        {
          "token" : "",
          "start_offset" : 4,
          "end_offset" : 5,
          "type" : "<IDEOGRAPHIC>",
          "position" : 4
        },
        {
          "token" : "",
          "start_offset" : 5,
          "end_offset" : 6,
          "type" : "<IDEOGRAPHIC>",
          "position" : 5
        },
        {
          "token" : "",
          "start_offset" : 6,
          "end_offset" : 7,
          "type" : "<IDEOGRAPHIC>",
          "position" : 6
        },
        {
          "token" : "",
          "start_offset" : 7,
          "end_offset" : 8,
          "type" : "<IDEOGRAPHIC>",
          "position" : 7
        },
        {
          "token" : "",
          "start_offset" : 8,
          "end_offset" : 9,
          "type" : "<IDEOGRAPHIC>",
          "position" : 8
        },
        {
          "token" : "",
          "start_offset" : 9,
          "end_offset" : 10,
          "type" : "<IDEOGRAPHIC>",
          "position" : 9
        },
        {
          "token" : "",
          "start_offset" : 10,
          "end_offset" : 11,
          "type" : "<IDEOGRAPHIC>",
          "position" : 10
        },
        {
          "token" : "",
          "start_offset" : 11,
          "end_offset" : 12,
          "type" : "<IDEOGRAPHIC>",
          "position" : 11
        },
        {
          "token" : "",
          "start_offset" : 12,
          "end_offset" : 13,
          "type" : "<IDEOGRAPHIC>",
          "position" : 12
        }
      ]
    }

    中文分词smarkcn

    $ bin/elasticsearch-plugin install analysis-smartcn

    This plugin can be downloaded for offline install from https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-smartcn/analysis-smartcn-6.6.2.zip.

    The plugin provides the smartcn analyzer and smartcn_tokenizer tokenizer, which are not configurable.

    分词效果:

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_analyze?pretty -d '{"tokenizer":"smartcn_tokenizer","text":"联想是全球最大的笔记本厂商"}'
    {
      "tokens" : [
        {
          "token" : "联想",
          "start_offset" : 0,
          "end_offset" : 2,
          "type" : "word",
          "position" : 0
        },
        {
          "token" : "",
          "start_offset" : 2,
          "end_offset" : 3,
          "type" : "word",
          "position" : 1
        },
        {
          "token" : "全球",
          "start_offset" : 3,
          "end_offset" : 5,
          "type" : "word",
          "position" : 2
        },
        {
          "token" : "",
          "start_offset" : 5,
          "end_offset" : 6,
          "type" : "word",
          "position" : 3
        },
        {
          "token" : "",
          "start_offset" : 6,
          "end_offset" : 7,
          "type" : "word",
          "position" : 4
        },
        {
          "token" : "",
          "start_offset" : 7,
          "end_offset" : 8,
          "type" : "word",
          "position" : 5
        },
        {
          "token" : "笔记本",
          "start_offset" : 8,
          "end_offset" : 11,
          "type" : "word",
          "position" : 6
        },
        {
          "token" : "厂商",
          "start_offset" : 11,
          "end_offset" : 13,
          "type" : "word",
          "position" : 7
        }
      ]
    }

    参考:https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html

    中文分词ik

    $ bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.6.2/elasticsearch-analysis-ik-6.6.2.zip

    The IK Analysis plugin integrates Lucene IK analyzer (http://code.google.com/p/ik-analyzer/) into elasticsearch, support customized dictionary.
    Analyzer: ik_smart , ik_max_word , Tokenizer: ik_smart , ik_max_word

    分词效果:

    ik_smark

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_analyze?pretty -d '{"tokenizer":"ik_smart","text":"联想是全球最大的笔记本厂商"}'         
    {
      "tokens" : [
        {
          "token" : "联想",
          "start_offset" : 0,
          "end_offset" : 2,
          "type" : "CN_WORD",
          "position" : 0
        },
        {
          "token" : "",
          "start_offset" : 2,
          "end_offset" : 3,
          "type" : "CN_CHAR",
          "position" : 1
        },
        {
          "token" : "全球",
          "start_offset" : 3,
          "end_offset" : 5,
          "type" : "CN_WORD",
          "position" : 2
        },
        {
          "token" : "最大",
          "start_offset" : 5,
          "end_offset" : 7,
          "type" : "CN_WORD",
          "position" : 3
        },
        {
          "token" : "",
          "start_offset" : 7,
          "end_offset" : 8,
          "type" : "CN_CHAR",
          "position" : 4
        },
        {
          "token" : "笔记本",
          "start_offset" : 8,
          "end_offset" : 11,
          "type" : "CN_WORD",
          "position" : 5
        },
        {
          "token" : "厂商",
          "start_offset" : 11,
          "end_offset" : 13,
          "type" : "CN_WORD",
          "position" : 6
        }
      ]
    }

    ik_max_word

    # curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_analyze?pretty -d '{"tokenizer":"ik_max_word","text":"联想是全球最大的笔记本厂商"}'
    {
      "tokens" : [
        {
          "token" : "联想",
          "start_offset" : 0,
          "end_offset" : 2,
          "type" : "CN_WORD",
          "position" : 0
        },
        {
          "token" : "",
          "start_offset" : 2,
          "end_offset" : 3,
          "type" : "CN_CHAR",
          "position" : 1
        },
        {
          "token" : "全球",
          "start_offset" : 3,
          "end_offset" : 5,
          "type" : "CN_WORD",
          "position" : 2
        },
        {
          "token" : "最大",
          "start_offset" : 5,
          "end_offset" : 7,
          "type" : "CN_WORD",
          "position" : 3
        },
        {
          "token" : "",
          "start_offset" : 7,
          "end_offset" : 8,
          "type" : "CN_CHAR",
          "position" : 4
        },
        {
          "token" : "笔记本",
          "start_offset" : 8,
          "end_offset" : 11,
          "type" : "CN_WORD",
          "position" : 5
        },
        {
          "token" : "笔记",
          "start_offset" : 8,
          "end_offset" : 10,
          "type" : "CN_WORD",
          "position" : 6
        },
        {
          "token" : "本厂",
          "start_offset" : 10,
          "end_offset" : 12,
          "type" : "CN_WORD",
          "position" : 7
        },
        {
          "token" : "厂商",
          "start_offset" : 11,
          "end_offset" : 13,
          "type" : "CN_WORD",
          "position" : 8
        }
      ]
    }

    参考:https://github.com/medcl/elasticsearch-analysis-ik

    七 复杂查询

    1 查询接口主要参数

    q
    The query string.

    stored_fields
    The selective stored fields of the document to return for each hit, comma delimited. Not specifying any value will cause no fields to return.

    sort
    Sorting to perform. Can either be in the form of fieldName, or fieldName:asc/fieldName:desc. The fieldName can either be an actual field within the document, or the special _score name to indicate sorting based on scores. There can be several sort parameters (order is important).

    from
    The starting from index of the hits to return. Defaults to 0.

    size
    The number of hits to return. Defaults to 10.

    timeout
    A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout.

    default_operator
    The default operator to be used, can be AND or OR. Defaults to OR.

    其中sort有很多种实现,比如 _geo_distance 可以用来实现地理位置远近排序,另外还可以通过filter来实现地理位置圈定,详见:https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html

  • 相关阅读:
    Bootstrap里的文件分别代表什么意思及其引用方法
    selenium alert JS弹窗问题处理
    Selenium爬取元素定位
    mysql 启动提示:错误2系统找不到指定文件
    团体程序设计天梯赛-练习集(一)(string.find()、string.erase()、string.insert()、map容器、L1-002 (*)、L1-005、L1-063、L1-058、L1-054 (*)、L1-003 )
    PTA基础编程题目集(四)(直接插入排序、希尔排序、大数阶乘、6-10 阶乘计算升级版、6-11 求自定类型元素序列的中位数)
    PTA基础编程题目集(三)(7-29 删除字符串中的子串 、7-26 单词长度、6-7 统计某类完全平方数)
    PTA基础编程题目集(二)(输入带空格字符串,字符数组、7-28 猴子选大王(*)、7-30 字符串的冒泡排序、7-31 字符串循环左移 、7-1 厘米换算英尺英寸、7-35 有理数均值(*))
    PTA基础编程题目集(一)(pow函数用法,7-13 日K蜡烛图、7-18 二分法求多项式单根(*))
    水题
  • 原文地址:https://www.cnblogs.com/barneywill/p/10296419.html
Copyright © 2011-2022 走看看