prometheus 服务器重启后之"Error on ingesting samples that are too old or are too far into the future"问题解决

背景描述：aws中的prometheus实例状态检查出现异常，重启实例后实例正常，此时发现grafana中主机为空，经查看日志发现如下：

level=warn ts=2021-08-16T03:20:04.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:04.961Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1507 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=491
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1203 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=959
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=941
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1507 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=558
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1203 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Appending scrape report failed" err="out of bounds"

解决方法：

备份prometheus数据库，新建data目录，重启prometheus和grafana服务
[root@prometheus /opt/prometheus]# mv data/ data_bak
[root@prometheus /opt/prometheus]# mkdir -pv data && chown -R prometheus.prometheus data
[root@prometheus /opt/prometheus]# systemctl restart prometheus
[root@prometheus /opt/prometheus]# systemctl restart grafana-server

web界面访问prometheus正常，grafana访问正常