客户反应很慢,程序玩不了,登录服务器,查看,发现cpu/io正常,内存也还有,但是负载很离谱的飙高到了30多,查原因:
因为用了redis,最近redis数据增长量很大,而且优化过,怀疑是redis问题,查看redis info信息,发现内存使用量在迅速升高又迅速下降,如此重复:
看到:
# Persistence loading:1 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1517792011 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok loading_start_time:1517792011 loading_total_bytes:316285045 loading_loaded_bytes:262143999 loading_loaded_perc:82.88 loading_eta_seconds:7
发现redis一直是loading状态,看/data/redis目录,有不少tmp文件了,应该是内存跑满了,导致redis无法bgsave,所以一直卡着,不断重复:
敲命令关掉redis bgsave:
config set save ""
发现报错:
(error) ERR Only CONFIG GET is allowed during loading
赶紧关闭其他服务,释放内存,php貌似也有内存没有完全释放,内存使用量大大提升
然后查看info信息:
# Persistence loading:0 rdb_changes_since_last_save:797 rdb_bgsave_in_progress:0 rdb_last_save_time:1517792293 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok
终于恢复正常:
tips:随着业务量上来,redis和数据库和程序要分开