· Monit 简介
Monit是一个轻量级(500KB)跨平台的用来监控Unix/linux系统的开源工具。部署简单，并且不依赖任何第三方程序、插件或者库。
Monit可以监控服务器进程、文件、文件系统、网络状态（HTTP/SMTP等协议）、远程主机、服务器资源变化等等。 并且可以设定资源变化后需要做的动作，比如服务失败后自动重启，邮件告警等等。
Monit内置了WEB UI，可以一目了然地了解监控项的情况。Monit是监控本机服务的工具，M/Monit是其配套产品用以对Monit统一管理，但是M/Monit并不是开源的，需要购买。
本文我们介绍在CentOS7.3上部署Monit用来监控进程状态，并实现服务失败后自动重启以及邮件告警通知的功能。监控其他资源的方法在默认配置文件中都有举例这里不一一介绍。
Monit官网：https://mmonit.com/monit
Monit手册：https://mmonit.com/monit/documentation/monit.html
· Monit 安装
# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# yum install -y monit
·Monit 常用命令
monit -t # 配置文件检测
monit # 启动monit daemon
monit -c /var/monit/monitrc # 启动monit daemon时指定配置文件
monit reload # 重新加载配置文件
monit status # 查看所有监控项务状态
monit status nginx # 查看nginx服务状态
monit stop all # 停止所有服务，这里需要注意的是，如果开启了自动重启功能，停止某个被监控的服务必须用monit stop xxx，若用系统命令停止服务，Monit会自动再把服务起来。
monit stop nginx # 停止nginx服务
monit start all # 启动所有服务
monit start nginx # 启动nginx服务
monit -V # 查看版本
·Monit 配置文件
/etc/monit.conf 主配置文件
/etc/monit.d/ 各项服务单独配置文件路径，在主配置文件中将其include进来。
/etc/monit.conf 举例说明：
配置文件关键字：'if', 'and', 'with(in)', 'has', 'us(ing|e)', 'on(ly)', 'then', 'for', 'of'
###############################################################################
## Monit control file
###############################################################################
##
## Comments begin with a '#' and extend through the end of the line. Keywords
## are case insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'.
##
## Below you will find examples of some frequently used statements. For
## information about the control file and a complete list of statements and
## options, please have a look in the Monit manual.
##
##
###############################################################################
## Global section
###############################################################################
##
## Start Monit in the background (run as a daemon):
#
# 设置检测周期30s
set daemon 30 # check services at 30 seconds intervals
# with start delay 240 # optional: delay the first check by 4-minutes (by
# # default Monit check immediately after Monit start)
#
#
## Set syslog logging. If you want to log to a standalone log file instead,
## specify the full path to the log file
#
#设置log路径，这里默认记录到syslog
set logfile syslog
#
#
## Set the location of the Monit lock file which stores the process id of the
## running Monit instance. By default this file is stored in $HOME/.monit.pid
#
# set pidfile /var/run/monit.pid
#
## Set the location of the Monit id file which stores the unique id for the
## Monit instance. The id is generated and stored on first Monit start. By
## default the file is placed in $HOME/.monit.id.
#
# set idfile /var/.monit.id
#
## Set the location of the Monit state file which saves monitoring states
## on each cycle. By default the file is placed in $HOME/.monit.state. If
## the state file is stored on a persistent filesystem, Monit will recover
## the monitoring state across reboots. If it is on temporary filesystem, the
## state will be lost on reboot which may be convenient in some situations.
#
# set statefile /var/.monit.state
#
## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using a comma separator. If the first mail server fails, Monit
# will use the second mail server in the list and so on. By default Monit uses
# port 25 - it is possible to override this with the PORT option.
#
# set mailserver mail.bar.baz, # primary mailserver
# backup.bar.baz port 10025, # backup mailserver on port 10025
# localhost # fallback relay
#
# 设置邮件服务器用来发送邮件告警通知
set mailserver mail.abcd.so
#
## By default Monit will drop alert events if no mail servers are available.
## If you want to keep the alerts for later delivery retry, you can use the
## EVENTQUEUE statement. The base directory where undelivered alerts will be
## stored is specified by the BASEDIR option. You can limit the queue size
## by using the SLOTS option (if omitted, the queue is limited by space
## available in the back end filesystem).
#
# set eventqueue
# basedir /var/monit # set the base directory where events will be stored
# slots 100 # optionally limit the queue size
#
#
## Send status and events to M/Monit (for more informations about M/Monit
## see http://mmonit.com/). By default Monit registers credentials with
## M/Monit so M/Monit can smoothly communicate back to Monit and you don't
## have to register Monit credentials manually in M/Monit. It is possible to
## disable credential registration using the commented out option below.
## Though, if safety is a concern we recommend instead using https when
## communicating with M/Monit and send credentials encrypted.
#
# set mmonit http://monit:monit@192.168.1.10:8080/collector
# # and register without credentials # Don't register credentials
#
#
## Monit by default uses the following format for alerts if the the mail-format
## statement is missing::
## --8<--
# 设置邮件告警通知格式
set mail-format {
from: monit@$HOST
subject: monit alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
Your faithful employee,
Monit
}
## --8<--
##
## You can override this message format or parts of it, such as subject
## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.
## are expanded at runtime. For example, to override the sender, use:
#
# set mail-format { from: monit@foo.bar }
#
#
## You can set alert recipients whom will receive alerts if/when a
## service defined in this file has errors. Alerts may be restricted on
## events by using a filter as in the second example below.
#
# 设置邮件告警通知人，Monit默认会通知monit进程本身的变化情况，如果不想收到monit进程自身的通知，加上but not on {instance}配置
set alert weian@abcd.so but not on { instance } # receive all alerts
#
## Do not alert when Monit starts, stops or performs a user initiated action.
## This filter is recommended to avoid getting alerts for trivial cases.
#
# set alert weian@abcd.so
#
#
## Monit has an embedded HTTP interface which can be used to view status of
## services monitored and manage services from a web interface. The HTTP
## interface is also required if you want to issue Monit commands from the
## command line, such as 'monit status' or 'monit restart service' The reason
## for this is that the Monit client uses the HTTP interface to send these
## commands to a running Monit daemon. See the Monit Wiki if you want to
## enable SSL for the HTTP interface.
#
# 设置UI界面访问信息
set httpd port 2812 and
use address 10.2.2.28 # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
allow admin:monit # require user 'admin' with password 'monit'
###############################################################################
## Services
###############################################################################
##
## Check general system resources such as load average, cpu and memory
## usage. Each test specifies a resource, conditions and the action to be
## performed should a test fail.
#
# check system $HOST
# if loadavg (1min) > 4 then alert
# if loadavg (5min) > 2 then alert
# if cpu usage > 95% for 10 cycles then alert
# if memory usage > 75% then alert
# if swap usage > 25% then alert
#
#
## Check if a file exists, checksum, permissions, uid and gid. In addition
## to alert recipients in the global section, customized alert can be sent to
## additional recipients by specifying a local alert handler. The service may
## be grouped using the GROUP option. More than one group can be specified by
## repeating the 'group name' statement.
#
# check file apache_bin with path /usr/local/apache/bin/httpd
# if failed checksum and
# expect the sum 8f7f419955cefa0b33a2ba316cba3659 then unmonitor
# if failed permission 755 then unmonitor
# if failed uid root then unmonitor
# if failed gid root then unmonitor
# alert security@foo.bar on {
# checksum, permission, uid, gid, unmonitor
# } with the mail-format { subject: Alarm! }
# group server
#
#
## Check that a process is running, in this case Apache, and that it respond
## to HTTP and HTTPS requests. Check its resource usage such as cpu and memory,
## and number of children. If the process is not running, Monit will restart
## it by default. In case the service is restarted very often and the
## problem remains, it is possible to disable monitoring using the TIMEOUT
## statement. This service depends on another service (apache_bin) which
## is defined above.
#
# check process apache with pidfile /usr/local/apache/logs/httpd.pid
# start program = "/etc/init.d/httpd start" with timeout 60 seconds
# stop program = "/etc/init.d/httpd stop"
# if cpu > 60% for 2 cycles then alert
# if cpu > 80% for 5 cycles then restart
# if totalmem > 200.0 MB for 5 cycles then restart
# if children > 250 then restart
# if loadavg(5min) greater than 10 for 8 cycles then stop
# if failed host www.tildeslash.com port 80 protocol http
# and request "/somefile.html"
# then restart
# if failed port 443 type tcpssl protocol http
# with timeout 15 seconds
# then restart
# if 3 restarts within 5 cycles then unmonitor
# depends on apache_bin
# group server
#
# 监控进程可以通过上面监控pid文件的方式，当没有pid文件时，可以通过MATCHING正则表达式来匹配进程。
# 测试一个进程是否匹配来自命令行使用的模式monit procmatch "regex-pattern"，这将列出匹配或不匹配的所有进程，regex模式。
# 我们这里监控了包含shop-pad-server字段的进程，并指明了启动以及停止的命令，这样在进程因故断掉后，Monit会自动重启进程。
# 同时若进程ID变动，会发送邮件通知到之前指定的收件人。
check process shop-pad-server with MATCHING shop-pad-server
start program = "/usr/bin/nohup /home/azureuser/pad-server/run.sh > /home/azureuser/pad-server/nohup.out 2>&1 &"
stop program = "/usr/bin/ps -ef | /usr/bin/grep shop-pad| /usr/bin/grep -v grep | /usr/bin/awk '{print $2}' | xargs kill"
if changed pid then alert
# ## Check filesystem permissions, uid, gid, space and inode usage. Other services, ## such as databases, may depend on this resource and an automatically graceful ## stop may be cascaded to them before the filesystem will become full and data ## lost. # # check filesystem datafs with path /dev/sdb1 # start program = "/bin/mount /data" # stop program = "/bin/umount /data" # if failed permission 660 then unmonitor # if failed uid root then unmonitor # if failed gid disk then unmonitor # if space usage > 80% for 5 times within 15 cycles then alert # if space usage > 99% then stop # if inode usage > 30000 then alert # if inode usage > 99% then stop # group server # # ## Check a file's timestamp. In this example, we test if a file is older ## than 15 minutes and assume something is wrong if its not updated. Also, ## if the file size exceed a given limit, execute a script # # check file database with path /data/mydatabase.db # if failed permission 700 then alert # if failed uid data then alert # if failed gid data then alert # if timestamp > 15 minutes then alert # if size > 100 MB then exec "/my/cleanup/script" as uid dba and gid dba # # ## Check directory permission, uid and gid. An event is triggered if the ## directory does not belong to the user with uid 0 and gid 0. In addition, ## the permissions have to match the octal description of 755 (see chmod(1)). # # check directory bin with path /bin # if failed permission 755 then unmonitor # if failed uid 0 then unmonitor # if failed gid 0 then unmonitor # # ## Check a remote host availability by issuing a ping test and check the ## content of a response from a web server. Up to three pings are sent and ## connection to a port and an application level network check is performed. # # check host myserver with address 192.168.1.1 # if failed ping then alert # if failed port 3306 protocol mysql with timeout 15 seconds then alert # if failed port 80 protocol http # and request /some/path with content = "a string" # then alert # # ## Check a network link status (up/down), link capacity changes, saturation ## and bandwidth usage. # # check network public with interface eth0 # if failed link then alert # if changed link then alert # if saturation > 90% then alert # if download > 10 MB/s then alert # if total upload > 1 GB in last hour then alert # # ## Check custom program status output. # # check program myscript with path /usr/local/bin/myscript.sh # if status != 0 then alert # # ############################################################################### ## Includes ############################################################################### ## ## It is possible to include additional configuration parts from other files or ## directories. # # include /etc/monit.d/* # # 导入其他单项服务的监控配置 # Include all files from /etc/monit.d/ include /etc/monit.d/*
· 通知周期设置
Monit默认情况下如果一个服务失败只发送一个通知：
alert foo@bar
如果您希望在服务保持处于失败状态时每十个周期通知一次，您可以使用：
alert foo@bar with reminder on 10 cycles
同样，如果您想在每个失败的周期获得通知，您可以使用：
alert foo@bar with reminder on 1 cycle
要禁止某些用户和服务的警报，可以在服务检查的局部配置里添加语句：
noalert mail-address
· 服务检测周期设置
可以使用every语句修改服务检查计划。
1.轮询周期倍数
EVERY [number] CYCLES
2.Cron-style
EVERY [cron]
# [cron]
# * * * * *
# 分 时 日 月 周
3.与Cron-style相反（do-not-check）
NOT EVERY [cron]
示例：
示例1：每两个周期检查一次
check process nginx with pidfile /var/run/nginx.pid
every 2 cycles
示例2：在上午8点到下午7点之间检查每个工作日
check program checkOracleDatabase
with path /var/monit/programs/checkoracle.pl
every "* 8-19 * * 1-5"
示例3：在星期日0AM到3AM之间不要在备份窗口中运行检查，否则运行具有常规轮询周期频率的检查。
check process mysqld with pidfile /var/run/mysqld.pid
not every "* 0-3 * * 0"
注意不要使用特定的分钟，因为Monit可能不会在那分钟运行。
· WEB UI界面
参考：
http://www.cnblogs.com/52fhy/p/6412547.html
http://blog.csdn.net/senlin1202/article/details/54291176
https://mmonit.com/monit/documentation/monit.html