grafana agent 集成tempo 试用

很简单就是几个工具椽到一起，方便学习了解grafana agent 对于tempo的支持

预备说明

grafana agent 对于tempo的支持，我们需要使用opentelemetry-collector，关于opentelemetry-collector 的使用
以及功能可以参考相关资料

环境准备

• 系统参考架构

• docker-compose

 

version: "3"

services:

  tempo:

    image: grafana/tempo:latest

    command: -config.file=/etc/tempo.yaml

    volumes:

    - ./etc/tempo.yaml:/etc/tempo.yaml

    ports:

      - "14268:14268"  # jaeger ingest

      - "3300:3300"

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  tempo-query:

    image: grafana/tempo-query:latest

    command: ["--grpc-storage-plugin.configuration-file=/etc/tempo-query.yaml"]

    volumes:

      - ./etc/tempo-query.yaml:/etc/tempo-query.yaml

    ports:

      - "16686:16686"  # jaeger-ui

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  otel-collector:

    image: otel/opentelemetry-collector-dev:latest

    volumes:

    - ./etc/otel-collector-config.yaml:/etc/otel-collector/config.yaml

    command: --config=/etc/otel-collector/config.yaml

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  grafana-agent:

    image: grafana/agent:v0.9.1

    volumes:

      - ./etc/agent2.yaml:/etc/agent-config/agent.yaml

    command:  -config.file=/etc/agent-config/agent.yaml   --prometheus.wal-directory=/tmp/agent/wal

    ports:

      -  "12345:12345"

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  victoriametrics:

    image: victoriametrics/victoria-metrics

    volumes:

      - ./example-data/victoriametrics:/victoriametrics

    ports:

      - "8428:8428"

    command: 

      - '-storageDataPath=/victoriametrics'

      - '-retentionPeriod=1'

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  synthetic-load-generator:

    image: omnition/synthetic-load-generator:1.0.25

    volumes:

      - ./etc/load-generator.json:/etc/load-generator.json

    environment:

      - TOPOLOGY_FILE=/etc/load-generator.json

      - JAEGER_COLLECTOR_URL=http://grafana-agent:14268

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  grafana:

    image: grafana/grafana:7.3.6

    volumes:

      - ./example-data/datasources:/etc/grafana/provisioning/datasources

      - ./example-data/dashboards-provisioning:/etc/grafana/provisioning/dashboard

    ports:

      - "3000:3000"

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

  loki:

    image: grafana/loki:2.1.0

    command: -config.file=/etc/loki/local-config.yaml

    ports:

      - "3100:3100"                                

    environment:

      - JAEGER_AGENT_HOST=tempo

      - JAEGER_ENDPOINT=http://tempo:14268/api/traces # send traces to Tempo

      - JAEGER_SAMPLER_TYPE=const

      - JAEGER_SAMPLER_PARAM=1

    logging:

      driver: loki

      options:

        loki-url: 'http://localhost:3100/api/prom/push'

• 依赖组件说明
  loki log 处理，grafana dashborad 展示，synthetic-load-generator 生成模拟数据，victoriametrics prometheus
  remote storage，opentelemetry-collector agent 集成tempo 的一个依赖，grafane agent 我们学习的主角，tempo
  agent集成tempo的依赖
• 配置说明
  主要介绍关于核心配置的，具体可以参考demo
  tempo 配置,主要是关于tempo 几个核心组件的配置，我们暴露了一些常用trace 的服务

 

auth_enabled: false

server:

  http_listen_port: 3300

distributor:

  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.

    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can

      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver

        thrift_http:                   #

        grpc:                          # for a production deployment you should only enable the receivers you need!

        thrift_binary:

        thrift_compact:

    zipkin:

    otlp:

      protocols:

        http:

        grpc:

    opencensus:

​

ingester:

  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it

  traces_per_block: 100                # cut the head block when it his this number of traces or ...

  max_block_duration: 5m               #   this much time passes

​

compactor:

  compaction:

    compaction_window: 1h              # blocks in this time window will be compacted together

    max_compaction_objects: 1000000    # maximum size of compacted blocks

    block_retention: 1h

    compacted_block_retention: 10m

​

storage:

  trace:

    backend: local                     # backend configuration to use

    wal:

      path: /tmp/tempo/wal             # where to store the the wal locally

      bloom_filter_false_positive: .05 # bloom filter false positive rate.  lower values create larger filters but fewer false positives

      index_downsample: 10             # number of traces per index record

    local:

      path: /tmp/tempo/blocks

    pool:

      max_workers: 100                 # the worker pool mainly drives querying, but is also used for polling the blocklist

      queue_depth: 10000

agent 配置,agent 是我们的核心部分，集成了metrics，log，trace

server:

  log_level: debug

  http_listen_port: 12345

​

# metrics 部分

prometheus:

  global:

    scrape_interval: 5s

  configs:

    - name: default

      scrape_configs:

        - job_name: agent

          static_configs:

            - targets: ['127.0.0.1:12345']

        - job_name: victoriametrics

          static_configs:

            - targets: ['victoriametrics:8428']

        - job_name: tempo

          static_configs:

            - targets: ['tempo:3300']

              labels:

                cluster: 'tempo'

                container: 'tempo'

      remote_write:

        - url: http://victoriametrics:8428/api/v1/write

#  log

loki:

  positions:

    filename: /tmp/positions.yaml

  scrape_configs:

    - job_name: varlogs

      static_configs:

        - targets: [localhost]

          labels:

            job: varlogs

            __path__: /var/log/*log

  clients:

    - url: http://loki:3100/loki/api/v1/push

# 集成temp 的trace

tempo:

  receivers:

    jaeger:

      protocols:

        thrift_http: 

  attributes:

    actions:

    - action: upsert

      key: env

      value: prod

## 需要依赖opentelemetry-collector

  push_config:

    endpoint: otel-collector:55680 # 

    insecure: true

    batch:

      timeout: 5s

      send_batch_size: 100

opentelemetry-collector
主要是opentelemetry-collector 标配组件的配置，receivers，exporters，pipeline （services）

 

receivers:

  otlp:

    protocols:

      grpc:

exporters:

  jaeger:

    endpoint: tempo:14250

    insecure: true

service:

  pipelines:

    traces:

      receivers: [otlp]

      exporters: [jaeger]

运行说明

• 首先安装docker loki driver

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

• 启动

docker-compose up -d

• 效果

说明

grafana 的agent 是一个不错的选择，集成的功能很多

参考资料

https://github.com/rongfengliang/grafana-tempo-learning
https://github.com/grafana/agent/blob/master/docs/configuration-reference.md
https://github.com/grafana
https://github.com/grafana/tempo
https://github.com/rongfengliang/grafana-tempo-learning
https://github.com/open-telemetry/opentelemetry-collector
https://grafana.com/docs/tempo/latest/getting-started/
https://grafana.com/docs/grafana/latest/datasources/loki/
https://github.com/open-telemetry/opentelemetry-specification/blob/master/experimental/trace/zpages.md
https://grafana.com/docs/loki/latest/