nginx error.log 中出现大量如下错误信息:
[root@localhost nginx]# grep -aP '^20.* [crit]' error.log
2017/03/14 12:06:31 [crit] 3549#0: accept4() failed (24: Too many open files)
[root@localhost nginx]# grep -aP '^20.* [alert]' error.log
2017/03/14 16:04:27 [alert] 3551#0: *84168270 socket() failed (24: Too many open files) while connecting to upstream, client: 1.1.1.1, server:...
由于系统limits open files 限制导致以上错误,所以:
root@open-mk-push-1:~# cat /etc/security/limits.conf * soft nofile 655360 * hard nofile 655360 root soft nofile 655360 root hard nofile 655360 * soft core unlimited * hard core unlimited root soft core unlimited root@open-mk-push-1:~# ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 512240 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 512240 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
但以上只是系统的open files 限制设置为655350;而nginx的open files 是否继承系统open files 设置还需要重新启动nginx 进程。可以通过以下命令查看:
root@open-mk-push-1:~# cat /proc/`ps -ef | grep nginx|grep -v grep|head -1|awk '{print $2}'`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 512240 512240 processes Max open files 1024 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 512240 512240 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
可以看到nginx进程的Max open files的Soft Limit和Hard Limit分别只有1024和4096,这是一个初始的值可以说是很低了,故之前出现的Too many open files就可以理解了。
总结:ulimit -a && cat /proc/`ps -ef | grep nginx|grep -v grep| head -1 | awk '{print $2}'`/limits 必须保持一致,否则会导致大量连接失败,此时需要重启nginx 进程 。
由于我们nginx是通过supervisor纳管的,所以得重启supervisor这个服务才能使nginx的max open files与系统值保持一致。