zoukankan      html  css  js  c++  java
  • ceph集群故障运维--持续更新

    一.PG处于异常状态active+undersized+degraded

    部署环境: 自己搭建的3节点集群,集群共5个OSD,部署Ceph的RadosGW的服务时,副本默认设置为3,集群存放数据量少。集群状态处于如下,无法自己恢复:

    [root@k8s-01 ceph]# ceph -s 
      cluster:
        id:     dd1a1ab2-0f34-4936-bc09-87bd40ef5ca0
        health: HEALTH_WARN
                Degraded data redundancy: 183/4019 objects degraded (4.553%), 15 pgs degraded, 16 pgs undersized
     
      services:
        mon: 3 daemons, quorum k8s-01,k8s-02,k8s-03
        mgr: k8s-01(active)
        mds: cephfs-2/2/2 up  {0=k8s-01=up:active,1=k8s-03=up:active}, 1 up:standby
        osd: 5 osds: 5 up, 5 in
        rgw: 3 daemons active
     
      data:
        pools:   6 pools, 288 pgs
        objects: 1.92 k objects, 1020 MiB
        usage:   7.5 GiB used, 342 GiB / 350 GiB avail
        pgs:     183/4019 objects degraded (4.553%)
                 272 active+clean
                 15  active+undersized+degraded
                 1   active+undersized
    

    分析上面两个状态显示的具体含义:

    • undersized
      PG当前的Acting Set小于存储池副本数

    • degraded
      降级状态,Peering完成后,PG检测到任意一个PG实例存在不一致(需要被同步/修复)的对象,或者当前ActingSet小于存储池副本数。

    以我们的集群为例说明,这个状态降级的集群可以正常读写数据,undersized是当前存活的PG副本数为2,小于副本数3.将其做此标记,表明存数据副本数不足。

    针对我们的集群,我们查看下集群的详细信息:

    [root@k8s-01 ceph]# ceph health detail 
    HEALTH_WARN Degraded data redundancy: 183/4017 objects degraded (4.556%), 15 pgs degraded, 16 pgs undersized
    PG_DEGRADED Degraded data redundancy: 183/4017 objects degraded (4.556%), 15 pgs degraded, 16 pgs undersized
        pg 4.0 is stuck undersized for 782.927699, current state active+undersized+degraded, last acting [0,3]
        pg 4.1 is stuck undersized for 782.930776, current state active+undersized+degraded, last acting [1,4]
        pg 4.2 is stuck undersized for 782.930141, current state active+undersized, last acting [1,3]
        pg 4.3 is stuck undersized for 782.929428, current state active+undersized+degraded, last acting [0,4]
        pg 4.4 is stuck undersized for 782.931202, current state active+undersized+degraded, last acting [1,3]
        pg 4.5 is stuck undersized for 782.931627, current state active+undersized+degraded, last acting [3,0]
        pg 4.6 is stuck undersized for 782.931584, current state active+undersized+degraded, last acting [0,4]
        pg 4.7 is stuck undersized for 782.928895, current state active+undersized+degraded, last acting [1,3]
        pg 6.0 is stuck undersized for 766.814237, current state active+undersized+degraded, last acting [0,4]
        pg 6.1 is stuck undersized for 766.818367, current state active+undersized+degraded, last acting [1,3]
        pg 6.2 is stuck undersized for 766.819767, current state active+undersized+degraded, last acting [1,3]
        pg 6.3 is stuck undersized for 766.822230, current state active+undersized+degraded, last acting [4,2]
        pg 6.4 is stuck undersized for 766.821061, current state active+undersized+degraded, last acting [1,4]
        pg 6.5 is stuck undersized for 766.826778, current state active+undersized+degraded, last acting [3,0]
        pg 6.6 is stuck undersized for 766.818134, current state active+undersized+degraded, last acting [1,3]
        pg 6.7 is stuck undersized for 766.826165, current state active+undersized+degraded, last acting [3,2]
    [root@k8s-01 ceph]# 
    

    分析:这两种状态一般同时出现,表明有些PG没有满足设定的replicas数量要求。由上面的信息我们分析PG 4.0显示只有两个拷贝,分别在OSD.0和OSD.3上,分析其原因可能是我们的集群副本数无法满足。我们的集群主机是以host为故障域,所以我们设置副本数为3时,需要保证每个节点都必须存在OSD,结果查看集群的osd tree发现只有两个节点有OSD,另外一个节点不存在OSD,这里我们设置存储池的副本为2,查看集群是否正常。

    # 查看ceph的OSD磁盘大小
    [root@k8s-01 ceph]# ceph osd df 
    ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE VAR  PGS 
     0   hdd 0.09769  1.00000 100 GiB 1.5 GiB  98 GiB 1.50 0.70 118 
     1   hdd 0.09769  1.00000 100 GiB 1.4 GiB  99 GiB 1.42 0.66 123 
     2   hdd 0.04880  1.00000  50 GiB 1.5 GiB  48 GiB 3.00 1.40  47 
     3   hdd 0.04880  1.00000  50 GiB 1.6 GiB  48 GiB 3.14 1.46 151 
     4   hdd 0.04880  1.00000  50 GiB 1.5 GiB  48 GiB 3.04 1.42 137 
                        TOTAL 350 GiB 7.5 GiB 342 GiB 2.14          
    MIN/MAX VAR: 0.66/1.46  STDDEV: 0.83
    # 查看ceph的OSD tree
    [root@k8s-01 ceph]# ceph osd tree
    ID CLASS WEIGHT  TYPE NAME       STATUS REWEIGHT PRI-AFF 
    -1       0.34177 root default                            
    -3       0.24417     host k8s-01                         
     0   hdd 0.09769         osd.0       up  1.00000 1.00000 
     1   hdd 0.09769         osd.1       up  1.00000 1.00000 
     2   hdd 0.04880         osd.2       up  1.00000 1.00000 
    -5       0.09760     host k8s-02                         
     3   hdd 0.04880         osd.3       up  1.00000 1.00000 
     4   hdd 0.04880         osd.4       up  1.00000 1.00000 
    
    • 设置存储池的副本数为2
    [root@k8s-01 ceph]# ceph osd  pool set default.rgw.log size 2
    [root@k8s-01 ceph]# ceph osd dump 
    epoch 130
    fsid dd1a1ab2-0f34-4936-bc09-87bd40ef5ca0
    created 2019-06-03 19:22:38.483687
    modified 2019-07-08 18:14:50.342952
    flags sortbitwise,recovery_deletes,purged_snapdirs
    crush_version 11
    full_ratio 0.95
    backfillfull_ratio 0.9
    nearfull_ratio 0.85
    require_min_compat_client jewel
    min_compat_client jewel
    require_osd_release mimic
    pool 1 'datapool' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 48 flags hashpspool stripe_width 0 application cephfs
    pool 2 'metapool' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 52 flags hashpspool stripe_width 0 application cephfs
    pool 3 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 121 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
    pool 4 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 127 flags hashpspool stripe_width 0 application rgw
    pool 5 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 126 flags hashpspool stripe_width 0 application rgw
    pool 6 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 129 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
    max_osd 5
    
    • 查看集群状态
    [root@k8s-01 ceph]# ceph -s 
      cluster:
        id:     dd1a1ab2-0f34-4936-bc09-87bd40ef5ca0
        health: HEALTH_OK
     
      services:
        mon: 3 daemons, quorum k8s-01,k8s-02,k8s-03
        mgr: k8s-01(active)
        mds: cephfs-2/2/2 up  {0=k8s-01=up:active,1=k8s-03=up:active}, 1 up:standby
        osd: 5 osds: 5 up, 5 in
        rgw: 3 daemons active
     
      data:
        pools:   6 pools, 288 pgs
        objects: 1.92 k objects, 1019 MiB
        usage:   7.5 GiB used, 342 GiB / 350 GiB avail
        pgs:     288 active+clean
     
      io:
        client:   339 B/s rd, 11 KiB/s wr, 0 op/s rd, 1 op/s wr
    

    至此我们发现集群状态已经恢复。

    二.cephfs挂载点执行df -h卡死

    • 查看卡死的客户端ID
    [root@k8s-02 ~]# ceph daemon mds.k8s-02 session ls
    {
            "id": 44272,
            "num_leases": 0,
            "num_caps": 1,
            "state": "open",
            "request_load_avg": 0,
            "uptime": 230.958432,
            "replay_requests": 0,
            "completed_requests": 0,
            "reconnecting": false,
            "inst": "client.44272 192.168.50.11:0/2167123153",
            "client_metadata": {
                "features": "00000000000001ff",
                "ceph_sha1": "cbff874f9007f1869bfd3821b7e33b2a6ffd4988",
                "ceph_version": "ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable)",
                "entity_id": "admin",
                "hostname": "k8s-02",
                "mount_point": "/mnt/ceph",
                "pid": "1904636",
    
    
    • 执行evict
    # 这里evict后面是客户端的ID
    [root@k8s-02 ~]# cep daemon mds.k8s-02 session evict 44272
    
    • umount -l 卡死的挂载点
    [root@k8s-02 ~]# umount -l /mnt/ceph
    [root@k8s-02 ~]# ceph-fuse /mnt/ceph
    
  • 相关阅读:
    为zabbix穿上一件漂亮的外衣
    CentOS7 Ceph分布式集群部署
    SSH 免秘钥登录
    zabbix监控Tomcat/JVM 实例性能
    zabbix 监控 IPMI
    2装饰者模式
    1代理模式
    3单例模式
    2抽象工厂模式
    1工厂模式
  • 原文地址:https://www.cnblogs.com/yuhaohao/p/13099494.html
Copyright © 2011-2022 走看看