Prometheus断电后启动异常 Error on ingesting samples

Prometheus断电后启动异常 Error on ingesting samples that are too old or are too far into the future

 

国庆假期回来后，发现 Prometheus 服务启动后没数据，查看日志

level=warn ts=2020-10-09T02:29:06.700Z caller=scrape.go:1216 component="scrape manager" scrape_pool=service.server target=http://192.168.1.22:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=929
level=warn ts=2020-10-09T02:29:06.700Z caller=scrape.go:987 component="scrape manager" scrape_pool=service.server target=http://192.168.1.22:9100/metrics msg="appending scrape report failed" err="out of bounds"

日志的时间戳与当前时间戳提前了8个小时： level=warn ts=2020-10-09T02:29:06.700Z

可能是国庆服务器长时间断电导致的 tsdb 记录日期太旧。

解决方法：

prometheus 版本：2.16.0
操作系统：CentOS 7.6

# 创建一个新的 tsdb 数据目录
mkdir /root/prometheus-2.16.0/tsdb_data/

# 启动时指定 tsdb 数据目录为刚刚创建的
/root/prometheus-2.16.0/prometheus 
--config.file="/root/prometheus-2.16.0/prometheus.yml" 
--storage.tsdb.path="/root/prometheus-2.16.0/tsdb_data" 
--web.enable-lifecycle 
--web.enable-admin-api &