ELK集群完全重启升级v7.2至7.8

目录

• Disable shard allocation
• Stop indexing and perform a synced flush
• Temporarily stop the tasks associated with active machine learning jobs and datafeeds (Optional)
• Shutdown all nodes
• Upgrade all nodes
• Upgrade any plugins
• Start each upgraded node
• Upgrade kibana and metricbeat
  • kibana升级
  • metricbeat升级
• Wait for all nodes to join the cluster and report a status of yellow
• Reenable allocation
• GET _cat/recoveryRestart machine learning jobs

Disable shard allocation

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}

curl -u elastic:密码 -XPUT "https://test-phy-elastic-140-91:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{ "persistent": { "cluster.routing.allocation.enable": "primaries" }}'

Stop indexing and perform a synced flush

POST _flush/synced
curl -u elastic:密码 -XPOST "https://test-phy-elastic-140-91:9200/_flush/synced"

Temporarily stop the tasks associated with active machine learning jobs and datafeeds (Optional)

POST _ml/set_upgrade_mode?enabled=true
curl -u elastic:密码 -XPOST "https://test-phy-elastic-140-91:9200/_ml/set_upgrade_mode?enabled=true"

Shutdown all nodes

先关闭从节点：
test-phy-elastic-140-93:
cd /opt/elastic-node6 && sh shutdown.sh
test-phy-elastic-140-92:
cd /opt/elastic-node4 && sh shutdown.sh
test-phy-elastic-140-91:
cd /opt/elastic-node2 && sh shutdown.sh
再关闭主节点：
test-phy-elastic-140-93:
cd /opt/elastic-node5 && sh shutdown.sh
test-phy-elastic-140-92:
cd /opt/elastic-node3 && sh shutdown.sh
test-phy-elastic-140-91:
cd /opt/elastic-coordinating && sh shutdown.sh
cd /opt/elastic-node1 && sh shutdown.sh

Upgrade all nodes

注：备份旧版本目录（官方升级说法是将旧目录配置同步到新目录下，由于涉及到数据盘路径，我们本次将新版本目录下文件同步到旧的下面）(不同步data目录和logs目录)

1.上传新版本tar包到/opt/bak目录下
2.操作如下：
test-phy-elastic-140-93:
cd /opt
mkdir /opt/elk7.2/
cd /opt/elastic-node5/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node5
rm -rf /opt/elastic-node5/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node5/
cd /opt/elastic-node5/
cp -a /opt/elk7.2/elastic-node5/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/
--------------------------------------------------------------------------------------------------------------
cd /opt/elastic-node6/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node6
rm -rf /opt/elastic-node6/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node6/
cd /opt/elastic-node6/
cp -a /opt/elk7.2/elastic-node6/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/

————————————————————————————————————————————————————————
test-phy-elastic-140-92:
cd /opt
mkdir /opt/elk7.2/
cd /opt/elastic-node3/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node3
rm -rf /opt/elastic-node3/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node3/
cd /opt/elastic-node3/
cp -a /opt/elk7.2/elastic-node3/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/
--------------------------------------------------------------------------------------------------------------
cd /opt/elastic-node4/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node4
rm -rf /opt/elastic-node4/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node4/
cd /opt/elastic-node4/
cp -a /opt/elk7.2/elastic-node4/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/

————————————————————————————————————————————————————————
test-phy-elastic-140-91:
cd /opt
mkdir /opt/elk7.2/
cd /opt/elastic-node2/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node2
rm -rf /opt/elastic-node2/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node2/
cd /opt/elastic-node2/
cp -a /opt/elk7.2/elastic-node2/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/
-------------------------------------------------------------------------------------------------------------
cd /opt/elastic-node1/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-node1
rm -rf /opt/elastic-node1/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-node1/
cd /opt/elastic-node1/
cp -a /opt/elk7.2/elastic-node1/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/
-------------------------------------------------------------------------------------------------------------
cd /opt/elastic-coordinating/
rsync -a --exclude data --exclude logs * /opt/elk7.2/elastic-coordinating
rm -rf /opt/elastic-coordinating/{bin,config,jdk,lib,modules,plugins,README.textile}
tar xf /opt/bak/elasticsearch-7.8.0-linux-x86_64.tar.gz -C /opt/
rm -rf /opt/elasticsearch-7.8.0/logs
cp -a /opt/elasticsearch-7.8.0/* /opt/elastic-coordinating/
cd /opt/elastic-coordinating/
cp -a /opt/elk7.2/elastic-coordinating/config/{certs,elasticsearch.yml,jvm.options,elasticsearch.keystore} config/

Upgrade any plugins

./bin/elasticsearch-plugin list 查看并无插件

Start each upgraded node

按照先主后从135246的顺序依次启动各节点及协调节点，查看日志bw-elastic.log
curl -u elastic:密码 -XGET "https://test-phy-elastic-140-91:9200/_cat/nodes?h=ip,name,version"

Upgrade kibana and metricbeat

kibana升级

重启集群后因为版本问题，官网上虽说es版本需要大于等于kibana版本，但是kibana7.2可能不会正常显示，重启kibana亦无法使用，此时需要完成kibana升级至7.8

cd /opt/kibana && sh shutdown.sh
cat /dev/null > nohup.out
cd /opt
mv /opt/kibana /opt/elk7.2/
tar xf /opt/bak/kibana-7.8.0-linux-x86_64.tar.gz -C /opt
mv /opt/kibana-7.8.0-linux-x86_64/ /opt/kibana
cd /opt/kibana
cp -a /opt/elk7.2/kibana/config/* /opt/kibana/config/
cp -a /opt/elk7.2/kibana/data /opt/kibana/
cp -a /opt/elk7.2/kibana/*.sh /opt/kibana/
sh startup.sh
tailf nohup.out
web访问：https://172.17.140.91:5601

配置文件可去掉注释部分：
server.host: "test-phy-elastic-140-91"
elasticsearch.hosts: ["https://test-phy-elastic-140-91:9200"]
server.ssl.enabled: true
server.ssl.certificate: config/certs/kibana.crt
server.ssl.key: config/certs/kibana.key
elasticsearch.ssl.certificateAuthorities: [ "config/certs/cacert.pem" ]
xpack.security.encryptionKey: "xxxxxxxxx_at_least_32_characters"
xpack.security.sessionTimeout: 36000000
xpack.monitoring.elasticsearch.hosts: ["https://test-phy-elastic-140-91:9200"]
xpack.monitoring.elasticsearch.ssl.certificateAuthorities: config/certs/cacert.pem

kibana升级后出现自身监控不可用的问题，是因为旧版本的部分配置参数被弃用，可通过./bin/kibana启动，会输出以下说明：
1.Setting [elasticsearch.username] to "elastic" is deprecated. You should use the "kibana_system" user instead.
2."xpack.monitoring.elasticsearch" is deprecated and has been replaced by "monitoring.ui.elasticsearch"
3."xpack.monitoring" is deprecated and has been replaced by "monitoring". However both key are present, ignoring "xpack.monitoring"
4.Setting [monitoring.username] to "elastic" is deprecated. You should use the "kibana_system" user instead
5."xpack.security.sessionTimeout" is deprecated and has been replaced by "xpack.security.session.idleTimeout"
6.set xpack.encryptedSavedObjects.encryptionKey

测试环境示例：

server.port: 5601
server.host: "lw-test-13-21"
elasticsearch.hosts: ["https://192.168.13.21:9201"]
server.ssl.enabled: true
server.ssl.certificate: config/certs/elastic-node1.crt
server.ssl.key: config/certs/elastic-node1.key
elasticsearch.ssl.certificateAuthorities: [ "config/certs/cacert.pem" ]
xpack.encryptedSavedObjects.encryptionKey: "xxxxxxxxxxxx_at_least_32_characters"
xpack.security.session.idleTimeout: 36000000
xpack.monitoring.ui.ealsticsearch: ["https://192.168.13.21:9201"]
monitoring.elasticsearch.ssl.certificateAuthorities: config/certs/cacert.pem

metricbeat升级

cd /opt/metricbeat/ && sh shutdown.sh
cat /dev/null > nohup.out
cd /opt
mv /opt/metricbeat /opt/elk7.2/
tar xf /opt/bak/metricbeat-7.8.0-linux-x86_64.tar.gz -C /opt
mv /opt/metricbeat-7.8.0-linux-x86_64 /opt/metricbeat
cp -a /opt/elk7.2/metricbeat/metricbeat.yml /opt/metricbeat
cp -a /opt/elk7.2/metricbeat/modules.d/elasticsearch-xpack.yml
cp -a /opt/elk7.2/metricbeat/{certs,data} /opt/metricbeat/
sh startup.sh
tailf nohup.out

配置文件可去掉注释部分：
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.kibana:
host: "https://test-phy-elastic-140-91:5601"
username: "metricbeat_internal"
password: "${ES_PWD}"
output.elasticsearch:
hosts: ["test-phy-elastic-140-91:9201"]
protocol: "https"
username: "metricbeat_internal"
password: "${ES_PWD}"
ssl.certificate_authorities: ["certs/cacert.pem"]
ssl.certificate: "certs/test-phy-elastic-140-91.crt"
ssl.key: "certs/test-phy-elastic-140-91.key"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~

metricbeat升级后出现访问http403问题,ca证书无法得到信任
可通过追加es的cacert.pem的内容至服务器/etc/pki/tls/certs/ca-bundle.crt文件末尾，就不会再报该错误了，其他错误可通过提示解决

修改 modules.d/elasticsearch-xpack.yml
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-elasticsearch.html
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
- enrich # 增加该指标集 可参考https://www.elastic.co/guide/en/beats/metricbeat/7.8/metricbeat-module-elasticsearch.html
period: 10s
hosts: ["https://test-phy-elastic-140-92:9201"]
username: "metricbeat_internal" # 修改用户和密码
password: "mima"
xpack.enabled: true
ssl.certificate_authorities: ["certs/cacert.pem"]
ssl.certificate: "certs/test-phy-elastic-140-92.crt"
ssl.key: "certs/test-phy-elastic-140-92.key"

该用户不仅具有监控采集的角色，需要更新kibana_user为kibana_admin，增加enrich_user角色。

Wait for all nodes to join the cluster and report a status of yellow

使用curl -u elastic:密码 -XGET "https://test-phy-elastic-140-91:9200/_cat/health" 查看集群整体状态

Reenable allocation

GET _cat/health
GET _cat/nodes

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}

curl -u elastic:密码 -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
'

GET _cat/recoveryRestart machine learning jobs

POST _ml/set_upgrade_mode?enabled=false
curl -u elastic:密码 -XPOST "https://test-phy-elastic-140-91:9200/_ml/set_upgrade_mode?enabled=false"