基于consul服务发现监控prometheus数据采集

业务服务使用micrometer + prometheus + grafana模式进行监控，如果服务机器列表发生变化，需要改动prometheus.yml机器域名或者IP，如果发生机器迁移或者扩缩容则容易遗忘，因此，我们可以通过prometheus与consul集成来实现动态监测机器列表变动来获取对应的监控服务机器信息。

1、业务服务目前基本是基于consul做的服务发现，可以直接在promethus.yml里配置，注意通过drop过滤非业务服务，并进行标签的重命名；

 - job_name: 'consul-prometheus'
    metrics_path: /actuator/prometheus
    scheme: http
    consul_sd_configs:
    -  server: 'consul_ip:8500'
       services: [] 
    relabel_configs:
       - source_labels:
           -  __meta_consul_service
         regex: 'consul|nginx|comment'
         action: drop
       - source_labels:
           -  __meta_consul_service
         target_label: group

2、将nginx服务注册到consul上，在promethus.yml里配置，注意通过nginx服务，并进行标签的重命名；

- job_name: 'nginx-consul-prometheus'
  metrics_path: /metrics
  consul_sd_configs:
  - server: 'consul_ip:8500'
    services: 
    - nginx 
 relabel_configs:
  - source_labels: ["__meta_consul_service"]
    target_label: "group"

nginx服务机器上需安装consul客户端，模板文件nginx.json，启动consul客户端时记得要额外加载该配置文件

{
  "service":{
  "id": "nginx",
  "name": "nginx",
  "address": "{{hostIp['stdout']}}",
  "port": 9145,
  "meta": {
    "sogou_host": "{{hostIp['stdout']}}"
  },
  "tags": ["nginx"],
  "checks": [
  {
   "http": "http://{{hostIp['stdout']}}:9145/metrics",
   "interval": "30s"
  }
  ]
 }
}

 3、ansible脚本如下，将nginx服务注册consul：

- name: register hostIp
  shell: /sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:" |head -n 1
  register: hostIp
  ignore_errors: True
  tags:
    - consul

- name: register nginx_metrics
  shell: netstat -ntlp | grep 9145
  register: nginx_metrics
  ignore_errors: True
  failed_when: nginx_metrics.rc != 1 and nginx_metrics.rc != 0

- name: Copy consul.json
  template: src=consul.hcl.j2 dest=/search/odin/consul/conf/consul.hcl

- name: Copy nginx.json
  template: src=nginx.json.j2 dest=/search/odin/consul/conf/nginx.json
  when: nginx_metrics.rc == 0
  tags:
    - consul

- name: Copy systemd script
  template: src=consul-promethus.service dest=/etc/systemd/system/consul-promethus.service
  tags:
    - consul

- name: chown /search/odin
  file:
    path: /search/odin/consul
    owner: odin
    group: odin
    mode: 0755
    recurse: yes
  tags:
    - consul

- name: Service consul started
  systemd:
    daemon_reload: yes
    enabled: yes
    name: consul-promethus
    state: restarted
  tags:
    - consul

4、安装consul太麻烦，可以直接使用api将服务注册到consul server，可以忽略上面的了

curl http://consulip:8500/v1/agent/service/register -X PUT -i -H "Content-Type:application/json" -d '{
 "ID": "id",  
 "Name": "name",
 "Tags": [
   "nginx"
 ],
 "Address": "ip",
 "Port": 9145,
 "Check": {
   "DeregisterCriticalServiceAfter": "30s",
   "HTTP": "http://ip:9145/metrics",
   "Interval": "30s"
 }
}'

。。。