zoukankan      html  css  js  c++  java
  • python elasticsearch加入@timestamp字段设置及时区设置

    es不常用,最近业务需要用到,因为kibana显示数据需要基于时间轴,临时学习一波,废话不多说,看以下代码:

    # -*- coding: utf-8 -*-
    # @Time    : 2021-04-13 09:51
    # @Author  : xxx
    # @FileName: demo_es.py
    # @Software: PyCharm
    
    from elasticsearch import Elasticsearch
    import datetime
    import time
    import random
    import string
    import uuid
    
    ES_URL = 'http://172.255.12.15:9200/'
    
    def gen_data():
        """
        just generate test data
        :return: test data
        """
        field_1 = time.time()
        field_2 = random.choice(string.ascii_letters)
        data_local = locals()
        # 时间戳字符串, iso格式, 不增加isoformat()处理时为datetime类型
        # data_local['@timestamp'] = datetime.datetime.now().isoformat() # 其实这有个问题,存入到es后,kibana显示的是当前时间+8,所以做了以下处理
        # data_local['@timestamp'] = ((datetime.datetime.now() - 
                            datetime.timedelta(hours=8))).isoformat() # 能解决kibana显示问题,但实际入库的timestamp时间早8小时
    
        data_local['@timestamp'] = datetime.datetime.now().isoformat() + "+0800" # 手动+0800,相当于加入时区,我们处于东八区,kibana显示正确
    
        return data_local
    
    
    def gen_doc_id():
        return str(uuid.uuid4())
    
    
    def insert_2_es(data):
        """
        insert data to es
        :param data:
        :return:
        """
        timestamp = datetime.date.today().strftime("%Y-%m-%d")
        index = "test-" + timestamp
        es_client = Elasticsearch(ES_URL)
    
        if not es_client.indices.exists(index):
            # setting mappings for index, 如果入库的话,后台有相应的序列化校验机制,不用设置mapping一样可以
            mapping = '''
            {
                "mappings": {
                      "_default_": {
                        "_all": {
                          "enabled": true,
                          "norms": false
                        },
                        "dynamic_templates": [
                          {
                            "message_field": {
                              "path_match": "message",
                              "match_mapping_type": "string",
                              "mapping": {
                                "norms": false,
                                "type": "text"
                              }
                            }
                          },
                          {
                            "string_fields": {
                              "match": "*",
                              "match_mapping_type": "string",
                              "mapping": {
                                "fields": {
                                  "keyword": {
                                    "type": "keyword"
                                  }
                                },
                                "norms": false,
                                "type": "text"
                              }
                            }
                          }
                        ],
                        "properties": {
                          "@timestamp": {
                            "type": "date",
                            "include_in_all": true
                          },
                          "@version": {
                            "type": "keyword",
                            "include_in_all": true
                          }
                        }
                      }
                }
            }
        '''
            es_client.indices.create(index,ignore=400, body=mapping)
            print("create index successfully, index: {}".format(index))
        doc_id = gen_doc_id()
        es_client.index(index=index, doc_type='_doc', id=doc_id, body=data, op_type='create')
        print("insert to es successfully, doc_id: {}".format(doc_id))
    
    
    
    if __name__ == "__main__":
        data = gen_data()
        print(data)
        print(gen_doc_id())
        insert_2_es(data)
    

    以下是kibana显示结果:

    加入时区处理后

  • 相关阅读:
    HQL查询.
    Apache solr(二).
    Apache solr(一).
    Node.js入门以及第一个helloworld程序.
    SQL优化一(SQL使用技巧)
    NodeJS安装第一个工程.
    [LeetCode] 545. Boundary of Binary Tree 二叉树的边界
    [LeetCode] 41. First Missing Positive 首个缺失的正数
    [LeetCode] 749. Contain Virus 包含病毒
    [LeetCode] 803. Bricks Falling When Hit 打击砖块掉落
  • 原文地址:https://www.cnblogs.com/davis12/p/14652200.html
Copyright © 2011-2022 走看看