loki 是一个方便的类似prometheus 的log 系统，mtail 是一个方便的日志提取工具，
可以暴露为http 服务——支持导出prometheus metrics
环境准备
- docker-compose 文件
version: "3"
services:
nginx-log:
build: ./
ports:
- "8090:80"
- "3903:3903"
volumes:
- "./examples/linecount.mtail:/progs/linecount.mtail"
- $PWD:/etc/promtail
loki:
image: grafana/loki:master
ports:
- "3100:3100"
volumes:
- $PWD:/etc/loki
command: -config.file=/etc/loki/loki-local-config.yaml
promtail:
image: grafana/promtail:make-images-static-26a87c9
volumes:
- $PWD:/etc/promtail
- ./log:/var/log
command:
-config.file=/etc/promtail/promtail-docker-config.yaml
grafana:
image: grafana/grafana:master
ports:
- "3000:3000"
environment:
- "GF_EXPLORE_ENABLED=true"
prometheus:
image: prom/prometheus
volumes:
- "./prometheus.yml:/etc/prometheus/prometheus.yml"
ports:
- "9090:9090"
- dockerfile 说明
dockerfile 主要是一个简单的openresty 服务，集成了mtail+loki,dockerfile 使用multi stage 处理
FROM dalongrong/mtail as builder
FROM grafana/promtail:make-images-static-26a87c9 as promtail
FROM openresty/openresty:alpine
ENV TINI_VERSION v0.18.0
RUN apk add --update
&& apk add --no-cache tini
ADD entrypoint.sh /entrypoint.sh
ADD mtail.sh /mtail.sh
ADD promtail.sh /promtail.sh
COPY nginx.conf usr/local/openresty/nginx/conf/
COPY --from=builder /usr/bin/mtail /usr/bin/
COPY --from=promtail /usr/bin/promtail /usr/bin/
EXPOSE 80 3903
ENTRYPOINT ["/sbin/tini","-s", "--", "/entrypoint.sh"]
- 启动脚本(entrypoint)
#!/bin/sh
sh mtail.sh
sh promtail.sh
exec /usr/local/openresty/bin/openresty -g "daemon off;"
mtail.sh:
#!/bin/sh
nohup /usr/bin/mtail -logtostderr -progs /progs/linecount.mtail -logs /var/log/error.log &
promtail.sh:
#!/bin/sh
nohup /usr/bin/promtail -config.file=/etc/promtail/promtail-docker-config2.yaml &
- prometheus server 配置
prometheus.yml 文件
scrape_configs:
- job_name: openresty-metrics
metrics_path: /metrics
static_configs:
- targets: ['nginx-log:3903']
- loki 配置
主要是agent 以及server
promtail-docker-config2.yaml:
server:
http_listen_port: 0
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
client:
url: http://loki:3100/api/prom/push
scrape_configs:
- job_name: system
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
job: varlogs-nginx
__path__: /var/log
- mtail 配置
# Copyright 2011 Google Inc. All Rights Reserved.
# This file is available under the Apache license.
counter line_count
/$/ {
line_count++
}
启动&&测试
- 构建镜像
docker-compose build
- 启动
docker-compose up -d
- grafana 配置
很简单，需要配置的有datasource（loki，prometheus） - 测试
模拟访问故障 http://localhost:8090/demoapp
mtail 界面
prometheus server
集成metrics&&log 分析
说明
grafana 的log 功能很方便，这样我们可以同时支持分析，以及问题排查了，而且不需要使用太多的工具就能解决问题
参考资料
https://www.cnblogs.com/rongfengliang/p/10112500.html
https://github.com/grafana/loki#getting-started
https://github.com/rongfengliang/grafana-loki-demo
https://github.com/google/mtail/blob/master/docs/Building.md
https://github.com/rongfengliang/mtail-nginx-docker-compose-demo