4.安装fluentd用于收集集群内部应用日志

作者

微信:tangy8080
电子邮箱:914661180@qq.com
更新时间：2019-06-13 11:02:14 星期四

欢迎您订阅和分享我的订阅号,订阅号内会不定期分享一些我自己学习过程中的编写的文章
如您在阅读过程中发现文章错误,可添加我的微信 tangy8080 进行反馈.感谢您的支持。

文章主题

• 在大多数情况下,我们需要集中管理应用的日志.但是我们又不能强制要求开发者直接对日志进行统一输出
  对开发者来说这可能是侵入式的,为了统一输出日志,可能导致业务收到影响.
  在这种情况下我们可以自己采集日志,采集工具比较多,这里我们选择fluentd,用于收集集群内的日志
• 集群外部日志的收集将在下一节涉及
• 日志收集原则:在任意时刻(由Es高性能索引速度支持)可以实时的查看到某一节点(由physics.hostname支持)的某一服务(由tag支持)的日志

前置条件

• 已经完成了本章节的第一，第二节

安装

开始安装fluentd-elasticsearch

由于我们对fluentd-elasticsearch的定制比较多,所以我们选择使用源码来安装fluentd-elasticsearch

#克隆源码库
cd /usr/local/src/
git clone https://github.com/kiwigrid/helm-charts
cd helm-charts/charts/fluentd-elasticsearch/

#为所有输出添加hostname,physics.hostname,tag字段,方便查找日志
vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml
#在containers.input.conf节点下添加如下配置
    <filter **>
      @id filter_hostname
      @type record_transformer
      <record>
        hostname "#{Socket.gethostname}"
        physics.hostname "#{ENV['K8S_NODE_NAME']}"
        tag ${tag}
      </record>
    </filter>

• 在filter中添加了hostname,该值为k8s中的podName
• physics.hostname标识节点的名称,该值为k8s中节点的名称,在daemonset.yaml中定义默认定义了K8S_NODE_NAME名称

#执行安装
cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch

helm install --name fluentd-elasticsearch --set elasticsearch.host=elasticsearch-client,hostLogDir.dockerContainers=/data/k8s/docker/data/containers .

• hostLogDir.dockerContainers 用于收集docker产生的日志,请按实际情况更改
• stable/fluentd-elasticsearch is deprecated as we move to our own repo (https://kiwigrid.github.io) which will be puplished on hub.helm.sh soon. The chart source can be found here: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch

[按需]卸载fluentd-elasticsearch

helm del --purge fluentd-elasticsearch

采集思路解析

• 我们在每个节点上运行一个fluentd代理（采用DaemonSet）,用于采集部署在k8s容器环境中程序和部署在物理主机中程序的日志。
• 对于容器内部应用.它的生命周期是不固定的,所以我们可以把日志放在宿主机上,（也可以使用sidecar容器，https://kubernetes.io/docs/concepts/cluster-administration/logging/）,出于性能原因,这里会选择只运行一个fluentd代理)
  
• pos_file需要放在何处? 需要放在宿主机上的一个稳定位置（通常和设定和path一个目录）.它记录了采集进度,假如容器销毁重启后,它依然可以从上次进度的位置开始采集
  
• 日志轮转问题, 如果应用本身不支持日志轮转.我们需要考虑日志轮转问题.避免日志越来越大

采集实例配置

实例一:运行在物理机器中的Nginx日志

配置挂载目录

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/values.yaml

#nginx日志不在/var/log目录下,在文件中最后添加Nginx日志文件挂载
extraVolumes:
   - name: nginxlog
     hostPath:
      path: /usr/local/nginx/logs
          
extraVolumeMounts:
   - name: nginxlog
     mountPath: /var/log/nginx
     readOnly: true

配置采集数据源

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml

    # service.honeysuckle-log-consumer
    <source>
      @id honeysuckle-log-consumer.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>d{4}-d{2}-d{2} d{2}:d{2}:d{2},d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>[.*]) - (?<msg>[sS]*)$/ 
        time_key time
      </parse>
      path /var/log/businesslogs/honeysuckle-log-consumer/Log.txt
      pos_file /var/log/businesslogs/honeysuckle-log-consumer/Log.txt.pos
      tag service.honeysuckle-log-consumer
    </source>
    
    
    # service.honeysuckle-configmanager-service
    <source>
      @id honeysuckle-configmanager-service.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>d{4}-d{2}-d{2} d{2}:d{2}:d{2},d{3}) (?<thread>.*) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>[.*]) - (?<msg>[sS]*)$/ 
        time_key time
      </parse>
      path /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt
      pos_file /var/log/businesslogs/honeysuckle-configmanager-service/Log.txt.pos
      tag service.honeysuckle-configmanager-service
    </source>
    
    # Nginx Access Log Source
    <source>
      @id nginx.accesslog
      @type tail
      path /var/log/nginx/access.log
      pos_file /var/log/access.log.pos
      format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)"(?:s+(?<http_x_forwarded_for>[^ ]+))?)?$/
      time_format %d/%b/%Y:%H:%M:%S %z
      tag nginx
     </source>

• nginx默认的日志格式format可以在这里找到
  https://docs.fluentd.org/parser/nginx

实例二:运行在容器中的Helloword程序

该项目用于模拟一个服务,它每隔3秒向 /app/App_Data/Logs/Log.txt 写入一条日志
该项目采用Log4net写入日志,应用本身支持日志轮转,最新的日志都在Log.txt中
log4net.config的配置如下:

<?xml version="1.0" encoding="utf-8" ?>
<log4net>
  <!--文本文件appender-->
  <appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender" >
    <file value="App_Data/Logs/Log.txt" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="10" />
    <maximumFileSize value="1024KB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
    </layout>
  </appender>

  <root>
    <appender-ref ref="RollingFileAppender" />
    <level value="ALL" />
  </root>
</log4net>

日志输出的样例类似于:

2019-06-17 07:29:52,038 [5] INFO  honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - electronicinvoice_log_queue.BasicConsume 已创建.
2019-06-17 07:31:51,510 [WorkPool-Session#1:Connection(21956434-0138-45f3-be65-848ca544cad3,amqp://honeysuckle.site:5672)] ERROR honeysuckle.log.consumer.HoneysuckleWebModule [(null)] - Error in EventingBasicConsumer.Received
Elasticsearch.Net.UnexpectedElasticsearchClientException: The operation was canceled. ---> System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   at Nest.LowLevelDispatch.BulkDispatch[TResponse](IRequest`1 p, SerializableData`1 body)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch)
   at honeysuckle.log.consumer.HoneysuckleWebModule.<>c__DisplayClass10_0.<CreateNewChannel>b__0(Object sender, BasicDeliverEventArgs e) in /src/src/honeysuckle.log.consumer/HoneysuckleWebModule.cs:line 141

基于日志输出格式,我们采用正则表达式插件进行日志解析（表达式在下面的source中可以看到）
可以使用fluentular工具进行表达式的正确性测试

该项目托管在:

http://admin@gitblit.honeysuckle.site/r/public/helloworld.git

如有测试需要.欢迎clone

配置采集数据源

vim /usr/local/src/helm-charts/charts/fluentd-elasticsearch/templates/configmaps.yaml

    # service.helloworld.log Log Source
    <source>
      @id helloworld.log
      @type tail
      <parse>
        @type regexp
        expression /^(?<time>d{4}-d{2}-d{2} d{2}:d{2}:d{2},d{3}) (?<thread>[d+]) (?<level>[a-zA-Z]+) (?<logger>.*) (?<property>[.*]) - (?<msg>.*)$/
        time_key time
      </parse>
      path /var/log/businesslogs/helloworld/Log.txt
      pos_file /var/log/businesslogs/helloworld/Log.txt.pos
      tag service.helloworld
    </source>

查看fluentd是否采集到了数据

待所有组件全部runing之后,可以通过curl查看fluentd是否采集到了数据

curl 'http://10.254.193.78:9200/_cat/indices?v'
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-2019.06.11 _v7q6b4DSt6mLJ_GLDyFUQ   5   1       4112            0      4.6mb          2.2mb
yellow open   logstash-2019.06.12 mz3sF15LTbqQwZiXMubdgg   5   1      29394            0     27.9mb         15.2mb
green  open   logstash-2019.06.13 rtjvfGVWSM-5rLvq72jxlA   5   1        971            0      2.4mb          1.1mb

在Kibana中检查日志收集情况

• 检查点一:日志是否根据tag正确分类
  

• 检查点二：日志否则附加了hostname和physics.hostname属性
  

处理升级问题

随着业务的需要,我们后期可能会加入其他的服务.这个时候我们可以在配置文件（configmaps.yaml）中添加好采集规则之后,进行升级

cd /usr/local/src/helm-charts/charts/fluentd-elasticsearch
helm upgrade fluentd-elasticsearch .

一些问题的处理方式记录

• failed to read data from plugin storage file path="/var/log/kernel.pos/worker0/storage.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/kernel.pos/worker0/storage.json

rm /var/log/kernel.pos/worker0/storage.json

引用链接

https://github.com/helm/charts/tree/master/stable/kibana#configuration