简单的应用服务器性能监控

由于项目上线，紧急应急需求，话说回来，公司CTO无能，如此重大的项目，居然没有早早规划应用系统的监控准备工作，还他妈的要求每10分钟汇报一次，愚蠢的很。。。暂且不吐槽了。

项目主要要求监控服务器的CPU利用率，磁盘利用率，以及tomcat的平均连接数（可能一台服务器上有多个tomcat）。

关于上述指标，我采用shell脚本实现相关数据的抓取，但是呢，上面领导的无能，总不能导致我们项目实际负责人也跟着无能的让团队受苦吧，就算不能百分百解放团队成员，至少能做到一些自动化，也是技术提升生产力，释放人类双手的一种价值所在吧。我的方案，就是通过shell抓取实际待监控的数据，然后用python程序作为agent，装在所有带监控的服务器上，以服务的形式存在在。另外，在其中一台上面，安装一个watchdog的程序，每隔一段时间，就“汪汪”一下，向主人提醒，有新的数据到来了。。。

另外，为了方便查看数据，统计的数据配合EChart.js展示，将上述监控数据在这个watchdog程序里面汇总，然后通过rsync将结果文件同步到一个网站（nginx）上，通过网络访问该监控数据web页面，做到随时随地的看监控。虽然不是高大上的系统，至少也算是一种简单有效的方案，我崇尚简单就是美！

下面简单看看，这个解决方案的架构图：

如上面已经介绍过，具体的性能数据的获取，是通过shell实现的。当watchdog这个定时任务执行的时候，将会向所有的待监测服务器发送指令（python程序实现的，基于TCP发送指令），待监控的机器上的python编写的agent程序在收到指令后，即执行shell脚本，实现数据的监控获取。然后以TCP的方式返回给watchdog程序。watchdog程序，将所有的检测数据，按照一定的格式，当然是方便EChat的展示了，写入一个文件，最终通过rsync推送到nginx的web服务器上，这样一来，需要看这个数据的人，就可以通过访问网址的形式，得到相关的监测结果。

思路和实现过程都不是很复杂，就不做过多的描述。下面，将相关的shell脚本以及python的agent程序以及watchdog程序，附在这里，供有需要的参考！

1、 shell脚本

#!/bin/bash

function cpuSts(){
   content=`top -n 1| grep "Cpu(s)"`
   ##echo $content
   cpup=`echo ${content#*:}`
   cpuu=`echo ${cpup%us*}`
   cpuss=`echo ${cpup#*,}`
   cpus=`echo ${cpuss%sy*}`
   us=`echo "$cpuu" | sed 's/^[ 	]*//g'`
   sy=`echo "$cpus" | sed 's/^[ 	]*//g'`
   #echo $us
   ius=`echo $us |awk -F ' ' '{print $2}'`
   #echo $ius
   #echo $sy
   isy=`echo $sy |awk -F ' ' '{print $2}'`
   #echo $isy

   fius=`echo $ius |awk -F '%' '{print $1}'`
   fisy=`echo $isy |awk -F '%' '{print $1}'`
   total=`echo $fius+$fisy | bc`
   #echo "us:$ius, sy:$isy, total:0$total%"
   echo "$total%"
}
cpus=`cpuSts`
#echo "cpu:$cpus"

function diskSts() {
   content=`df -h | grep "/dev" | grep -v "tmpfs" | awk '{print $5}'`
   echo "$content"
}
disks=`diskSts`
#echo "disk:$disks"

function tomcatSts() {
   pids=`ps -ef|grep tomcat | grep "org.apache.catalina.startup.Bootstrap start" |grep -v "grep" | awk '{print $2}'`
   cnt=0
   idx=0
   for pid in $pids; do
       #echo "$pid"
       tct=`netstat -alptn | grep $pid/java | grep ESTABLISHED | wc -l`
       let cnt+=tct
       let idx+=1
   done
   echo "establishedCount:$cnt,tomcatCount:$idx"
}
tomcats=`tomcatSts`
echo "cpu:$cpus,disk:$disks,$tomcats"

2. watchdog的python编写的cron的任务代码（TCP客户端）

#!/usr/bin/env python
# -*- coding:utf-8 -*-

# #给server端发送命令
import socket, sys, traceback
import threading
import time
import subprocess
host1 = '10.130.207.210'
host2 = '10.130.207.211'
port = 23153
curDate =  str(time.strftime('%Y-%m-%d',time.localtime(time.time())))
filepath = "/home/tomcat/weibao/robot/%s.txt"%(curDate)
filepathjs = "/home/tomcat/weibao/robot/%s.js"%(curDate)

def trans(info):
   keypair = info.split(",")
   res=""
   est=0
   tmc=1
   connc=0
   for item in keypair:
       key,val = item.split(":")
       if key == "cpu" or key == "disk":
           lent = len(val)
           val = val[0:lent-1]
           val = float(val)
       if key == "establishedCount":
           est = val
       elif key == "tomcatCount":
           if val > 0:
               tmc = val
       else:
           res += '"%s":"%s",' % (key,val)

   connc = int(est) / int(tmc)
   res += '"estcon":"%s"' % connc
   if res[-1] == ",":
       return res[0:len(res)-1]
   else:
       return res

def client(host):
   s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   try:
      s.connect((host, port))
   except Exception, e:
      msg = traceback.format_exc()
      print '连接错误:', msg

   s.send("weibaost")
   buff = s.recv(4096)
   if not len(buff):
      print 'buff is empty !'
   else :
     # sys.stdout.write(buff)
      print 'currentDate:%s'%(curDate)
      print 'host:%s,%s'%(host,str(buff))

      curTime = str(time.strftime('%H:%M',time.localtime(time.time())))
      with open(filepath,"a") as f:
          f.write('{"time":"%s","host":"%s","qa":"12",%s},'%(curTime,host,trans(str(buff))))

def filetrans():
    with open(filepath,"r") as f:
        content = f.read()
        print content
        content = content[0:len(content)-1]
        print content
        content = "json=[" +content + "]"
        print content
    with open(filepathjs,"w")as f:
        f.write(content)

threads = []
t1 = threading.Thread(target=client,args=(host1,))
threads.append(t1)
t2 = threading.Thread(target=client,args=(host2,))
threads.append(t2)
if __name__ == "__main__":
   for t in threads:
   #    print 'thread start....'
       t.start()
       t.join()

   filetrans()
   handler = subprocess.Popen("/home/tomcat/rsyncwb.sh", shell=True, stdout=subprocess.PIPE)
   output = handler.stdout.readlines()
   print output
   with open("/home/tomcat/listenLog.log","a") as f:
       f.write(str(output))# client(host1)

上述代码中rsyncwb.sh就是执行rsync指令，实现将最终处理好的数据推送到nginx的web服务器上的某个指定路径下，方便客户通过浏览器访问

3. python的agent程序（TCP服务端）

#!/usr/bin/env python
# -*- coding:utf-8 -*-

# #执行客户端发送过来的命令，并把执行结果返回给客户端
import socket, traceback, subprocess

host = '0.0.0.0'
port = 23153

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

s.bind((host, port))
s.listen(10)

while 1:
    try:
        client_socket, client_addr = s.accept()
    except Exception, e:
        traceback.print_exc()
        continue

    try:
        print 'From host:', client_socket.getpeername()

        command = client_socket.recv(4096)
        if not len(command):
            break
        print client_socket.getpeername()[0] + ':' + str(command)
        scommand = str(command)
        if scommand == "weibaost":
            # 执行客户端传递过来的命令
            handler = subprocess.Popen("/home/tomcat/weibaoSt.sh", shell=True, stdout=subprocess.PIPE)
            output = handler.stdout.readlines()
            if output is None:
                output = []
            for one_line in output:
                client_socket.sendall(one_line)
               # client_socket.sendall("
")
        else:
            print "Not supported command!!!"

    except Exception, e:
        traceback.print_exc()
        client_socket.sendall("command error")

    try:
        client_socket.close()
    except Exception, e:
        traceback.print_exc()

上述代码中的weibaoSt.sh，就是第一小节贴出的shell代码内容。

最后，EChart展示相关的，请查考百度的官网，自己研究，不是这里的重点。这里，将关键的配置和展示部分的代码贴出来：

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>机器人系统监控</title>
<link href="css/statistics.css" rel="stylesheet">

<!-- 引入 ECharts 文件 -->
<script src="js/echarts.js"></script>
<script src="js/jquery-2.1.1.min.js"></script>

</head>

<body>
    
    
    <div class="col-sm-9 col-sm-offset-3 col-lg-10 col-lg-offset-2 main main-plane title-box"  id="reportTab" style="margin-left:12%;100%">
        
        <div class="chart-container">
        <!--问题开始 -->
            <div style="text-align:center;;margin-left: 4%">
                <h2>问答数</h2>
                <div>
                    <!-- 为ECharts准备一个具备大小（宽高）的Dom -->
                    <div id="r01" style="1200px;height:500px;"></div>
                </div>
            </div>
        <!--问题结束-->
        
        
      
        </div> 

         <div class="chart-container">
        <!--CPU开始 -->
            <div style="text-align:center;;margin-left: 4%">
                <h2>CPU</h2>
                <div>
                    <!-- 为ECharts准备一个具备大小（宽高）的Dom -->
                    <div id="r02" style="1200px;height:500px;"></div>
                </div>
            </div>
        <!--CPU结束-->
        
        
      
        </div> 

         <div class="chart-container">
        <!--DISK开始 -->
            <div style="text-align:center;;margin-left: 4%">
                <h2>DISK</h2>
                <div>
                    <!-- 为ECharts准备一个具备大小（宽高）的Dom -->
                    <div id="r03" style="1200px;height:500px;"></div>
                </div>
            </div>
        <!--DISK结束-->
        
        
      
        </div> 

         <div class="chart-container">
        <!--TOMCAT开始 -->
            <div style="text-align:center;;margin-left: 4%">
                <h2>TOMCAT</h2>
                <div>
                    <!-- 为ECharts准备一个具备大小（宽高）的Dom -->
                    <div id="r04" style="1200px;height:500px;"></div>
                </div>
            </div>
        <!--TOMCAT结束-->
        
        </div> 
        
    </div>
    <script src="js/example1.js"></script>

</body>

</html>

$(function() {

    Date.prototype.Format = function (fmt) { //author: meizz 
    var o = {
        "M+": this.getMonth() + 1, //月份 
        "d+": this.getDate(), //日 
        "h+": this.getHours(), //小时 
        "m+": this.getMinutes(), //分 
        "s+": this.getSeconds(), //秒 
        "q+": Math.floor((this.getMonth() + 3) / 3), //季度 
        "S": this.getMilliseconds() //毫秒 
    };
    if (/(y+)/.test(fmt)) fmt = fmt.replace(RegExp.$1, (this.getFullYear() + "").substr(4 - RegExp.$1.length));
    for (var k in o)
    if (new RegExp("(" + k + ")").test(fmt)) fmt = fmt.replace(RegExp.$1, (RegExp.$1.length == 1) ? (o[k]) : (("00" + o[k]).substr(("" + o[k]).length)));
    return fmt;
    }


    var jsName = new Date().Format("yyyy-MM-dd")+".js";
    var jsUrl = "http://ecuat.taikang.com/whcms/weibao/robot/"+jsName;
    $.getScript(jsUrl,function(){
        loadData();
    });

    /********************************问题统计 start ********************************************/

    var R01Chart = echarts.init(document.getElementById('r01'));
    // 显示标题，图例和空的坐标轴
    R01Chart.setOption({
         tooltip : {
        trigger: 'axis'
    },
    legend: {
        
    },
    toolbox: {
        show : true,
        feature : {
            mark : {show: true},
            dataView : {show: true, readOnly: false},
            magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']},
            restore : {show: true},
            saveAsImage : {show: true}
        }
    },
    calculable : true,
    xAxis : [
        {
            type : 'category',
            boundaryGap : false,
            data : []
        }
    ],
    yAxis : [
        {
            type : 'value'
        }
    ],
    series : [
        
    ]
    });
    //R01Chart.showLoading(); // 数据加载完之前先显示一段简单的loading动画

    /********************************问题统计 end ********************************************/


    /********************************CPU start ********************************************/

    var R02Chart = echarts.init(document.getElementById('r02'));
    // 显示标题，图例和空的坐标轴
    R02Chart.setOption({
         tooltip : {
        trigger: 'axis'
    },
    legend: {
        
    },
    toolbox: {
        show : true,
        feature : {
            mark : {show: true},
            dataView : {show: true, readOnly: false},
            magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']},
            restore : {show: true},
            saveAsImage : {show: true}
        }
    },
    calculable : true,
    xAxis : [
        {
            type : 'category',
            boundaryGap : false,
            data : []
        }
    ],
    yAxis : [
        {
            type : 'value'
        }
    ],
    series : [
        
    ]
    });


    /********************************CPU end ********************************************/


    /********************************DISK start ********************************************/


var R03Chart = echarts.init(document.getElementById('r03'));
    // 显示标题，图例和空的坐标轴
    R03Chart.setOption({
         tooltip : {
        trigger: 'axis'
    },
    legend: {
        
    },
    toolbox: {
        show : true,
        feature : {
            mark : {show: true},
            dataView : {show: true, readOnly: false},
            magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']},
            restore : {show: true},
            saveAsImage : {show: true}
        }
    },
    calculable : true,
    xAxis : [
        {
            type : 'category',
            boundaryGap : false,
            data : []
        }
    ],
    yAxis : [
        {
            type : 'value'
        }
    ],
    series : [
        
    ]
    });


    /********************************DISK end ********************************************/


    /********************************TOMCAT start ********************************************/


var R04Chart = echarts.init(document.getElementById('r04'));
    // 显示标题，图例和空的坐标轴
    R04Chart.setOption({
         tooltip : {
        trigger: 'axis'
    },
    legend: {
        
    },
    toolbox: {
        show : true,
        feature : {
            mark : {show: true},
            dataView : {show: true, readOnly: false},
            magicType : {show: true, type: ['line', 'bar', 'stack', 'tiled']},
            restore : {show: true},
            saveAsImage : {show: true}
        }
    },
    calculable : true,
    xAxis : [
        {
            type : 'category',
            boundaryGap : true,
            data : []
        }
    ],
    yAxis : [
        {
            type : 'value'
        }
    ],
    series : [
        
    ]
    });


    /********************************TOMCAT end ********************************************/

    function loadData() {

        var xAxisData = new Set();
        var IPDATA = new Object();
        var CPUDATA = new Object();
        var DISKDATA = new Object();
        var TOMCATDATA = new Object();

        var legendSet = new Set();
        $.each(json,function(i,n){
            var IP = n.host;
            var QA = n.qa;
            var CPU = n.cpu;
            var DISK = n.disk;
            var TOMCAT = n.estcon;
            var TIME = n.time;

        if(!IPDATA.hasOwnProperty(IP)){
            IPDATA[IP]=new Object();
            IPDATA[IP].name=IP;
            IPDATA[IP].type='line';
            //IPDATA[IP].stack= '总量';
            IPDATA[IP].data=new Array();
            IPDATA[IP].xb=0;

            CPUDATA[IP]=new Object();
            CPUDATA[IP].name=IP;
            CPUDATA[IP].type='line';
            //CPUDATA[IP].stack= '总量';
            CPUDATA[IP].data=new Array();
            CPUDATA[IP].xb=0;

            DISKDATA[IP]=new Object();
            DISKDATA[IP].name=IP;
            DISKDATA[IP].type='line';
            //DISKDATA[IP].stack= '总量';
            DISKDATA[IP].data=new Array();
            DISKDATA[IP].xb=0;

            TOMCATDATA[IP]=new Object();
            TOMCATDATA[IP].name=IP;
            TOMCATDATA[IP].type='line';
            //TOMCATDATA[IP].stack= '总量';
            TOMCATDATA[IP].data=new Array();
            TOMCATDATA[IP].xb=0;
        }

            var kk=IPDATA[IP].xb;
            IPDATA[IP].data[kk]=QA;
            IPDATA[IP].xb=kk+1;

            CPUDATA[IP].data[kk]=CPU;
            CPUDATA[IP].xb=kk+1;

            DISKDATA[IP].data[kk]=DISK;
            DISKDATA[IP].xb=kk+1;

            TOMCATDATA[IP].data[kk]=TOMCAT;
            TOMCATDATA[IP].xb=kk+1;

            xAxisData.add(TIME);
            legendSet.add(IP);
            



        });

            var legendArr = new Array();
            var ii = 0;
            legendSet.forEach(function (item) {
                legendArr[ii] = item;
                ii++;
            });

            var tt=0;
            var series=new Array();
            for(item in  IPDATA){
                series[tt]=IPDATA[item];
                tt++;
            }

            var tt2=0;
            var series2=new Array();
            for(item in  CPUDATA){
                series2[tt2]=CPUDATA[item];
                tt2++;
            }

            var tt3=0;
            var series3=new Array();
            for(item in  DISKDATA){
                series3[tt3]=DISKDATA[item];
                tt3++;
            }

            var tt4=0;
            var series4=new Array();
            for(item in  TOMCATDATA){
                series4[tt4]=TOMCATDATA[item];
                tt4++;
            }

            var xAxisDataArr = new Array();
            var iii = 0;
            xAxisData.forEach(function (item) {
                xAxisDataArr[iii] = item;
                iii++;
            });

    console.log(series);
    console.log(series2);
    console.log(series3);
    console.log(series4);
        
        /*******问题数据加载*****/
            R01Chart.setOption({
                legend : {
                    data : legendArr
                },
                xAxis : [
                {
                    data : xAxisDataArr
                }    
                ],
                series : series
            });

        /*******CPU数据加载*****/
            R02Chart.setOption({
                legend : {
                    data : legendArr
                },
                xAxis : [
                {
                    data : xAxisDataArr
                }    
                ],
                series : series2
            });

        /*******DISK数据加载*****/
            R03Chart.setOption({
                legend : {
                    data : legendArr
                },
                xAxis : [
                {
                    data : xAxisDataArr
                }    
                ],
                series : series3
            });

        /*******TOMCAT数据加载*****/
            R04Chart.setOption({
                legend : {
                    data : legendArr
                },
                xAxis : [
                {
                    data : xAxisDataArr
                }    
                ],
                series : series4
            });

    }
});

另外，附带一下相关的测试数据，以js为后缀名，时间为文件名的数据文件：

json=[{"time":"08:58","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"08:58","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"08:59","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"08:59","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:00","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:00","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:01","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:01","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:02","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:02","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:03","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:03","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:04","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:04","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:05","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:05","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:06","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:06","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:07","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:07","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:08","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:08","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:09","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:09","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:10","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:10","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:11","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:11","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:12","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:12","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:13","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:13","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:14","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:14","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:15","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:15","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:16","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:16","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:17","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:17","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:18","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:18","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:19","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:19","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:20","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:20","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:21","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:21","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:22","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:22","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:23","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"30"},{"time":"09:23","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"19"},{"time":"09:24","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"09:24","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"20"},{"time":"09:25","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"09:25","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"20"},{"time":"09:26","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:26","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:27","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:27","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:28","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"26"},{"time":"09:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:28","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:29","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:29","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:30","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:30","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:31","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:31","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:32","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:32","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:33","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:33","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:34","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:34","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:35","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:35","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"22"},{"time":"09:36","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:36","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:37","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:37","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:38","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:38","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:39","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"09:39","host":"10.130.207.211","qa":"12","cpu":"3.7","disk":"31.0","estcon":"21"},{"time":"09:40","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:40","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:41","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:41","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:42","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:42","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:43","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:43","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:44","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:44","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:45","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:45","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:46","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:46","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:47","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:47","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:48","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:48","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:49","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:49","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"09:50","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:50","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:51","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:51","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:52","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:52","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:53","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:53","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:54","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:54","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:55","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:55","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"09:56","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:56","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:57","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:57","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:58","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:58","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"09:59","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"09:59","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:00","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:00","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:01","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:01","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:02","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:02","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:03","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:03","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:04","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:04","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:05","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:05","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:06","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:06","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:07","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:07","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:08","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:08","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:09","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:09","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:10","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:10","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:11","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:11","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:12","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:12","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:13","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:13","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:14","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:14","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:15","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:15","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:16","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:16","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:17","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:17","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:18","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:18","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:19","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:19","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:20","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:20","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"21"},{"time":"10:21","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:21","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"22"},{"time":"10:22","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:22","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"23"},{"time":"10:23","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:23","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:24","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:24","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:25","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:25","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:26","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:26","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:27","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:27","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:28","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:28","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"26"},{"time":"10:29","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:29","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:30","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:30","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:31","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"5"},{"time":"10:31","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:32","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"10:32","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:33","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"34"},{"time":"10:33","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:34","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:34","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:35","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:35","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:36","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"36"},{"time":"10:36","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:37","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:37","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:38","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"35"},{"time":"10:38","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:39","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"33"},{"time":"10:39","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"25"},{"time":"10:40","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:40","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:41","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:41","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:42","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:42","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:43","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"31"},{"time":"10:43","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:44","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:44","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:45","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:45","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:46","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:46","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:47","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:47","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"},{"time":"10:48","host":"10.130.207.210","qa":"12","cpu":"3.8","disk":"45.0","estcon":"32"},{"time":"10:48","host":"10.130.207.211","qa":"12","cpu":"3.8","disk":"31.0","estcon":"24"}]

上面的测试案例中，有个指标，问答数，是hardcode写的，因为脚本还没有完成，这个博文只是说明，这个简单有效的监控方案，是应急的时候的行之有效的方案。下面给出一个tomcat连接数的监控截图：

监控是互联网系统的非常重要的内容，需要非常的重视，即便你再牛逼，也不能轻视这个方面的投入，最后的防线预警机！