标签（空格分隔）： ceph，ceph运维，pg
集群环境：
[root@node3 ~]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[root@node3 ~]# ceph -v
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)
集群当前布局：
[root@node3 ceph-6]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08844 root default
-3 0.02948 host node1
0 hdd 0.00980 osd.0 up 1.00000 1.00000
3 hdd 0.00980 osd.3 up 1.00000 1.00000
-5 0.02948 host node2
1 hdd 0.00980 osd.1 up 1.00000 1.00000
4 hdd 0.00980 osd.4 up 1.00000 1.00000
-7 0.02948 host node3
2 hdd 0.00980 osd.2 up 1.00000 1.00000
5 hdd 0.00980 osd.5 up 1.00000 1.00000
为每个主机再添加一个osd：
为了重现too few pgs的错误，同时为了创建指定数据位置osd，下面创建bluestore的osd，数据存储在/dev/sdd1上。在每个主机上执行下面的步骤：
第一步：创建bluestore类型的osd：
[root@node2 ~]# ceph-disk prepare --bluestore /dev/sdd2 --block.db /dev/sdd1
set_data_partition: incorrect partition UUID: cafecafe-9b03-4f30-b4c6-b4b80ceff106, expected ['4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', '4fbd7e29-8ae0-4982-bf9d-5a8d867af560', '4fbd7e29-9d25-41b8-afd0-35865ceff05d']
prepare_device: OSD will not be hot-swappable if block.db is not the same device as the osd data
prepare_device: Block.db /dev/sdd1 was not prepared with ceph-disk. Symlinking directly.
meta-data=/dev/sdd2 isize=2048 agcount=4, agsize=648895 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2595579, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
第二步：激活该osd：
[root@node2 ~]# ceph-disk activate /dev/sdd2
creating /var/lib/ceph/tmp/mnt.mR3qCJ/keyring
added entity osd.8 auth auth(auid = 18446744073709551615 key=AQBNqOVZt/iUBBAArkrWrZi9N0zxhHhYfhanyw== with 0 caps)
got monmap epoch 1
Removed symlink /etc/systemd/system/ceph-osd.target.wants/ceph-osd@8.service.
Created symlink from /etc/systemd/system/ceph-osd.target.wants/ceph-osd@8.service to /usr/lib/systemd/system/ceph-osd@.service.
最后查看集群布局，发现共有9个osd：
[root@node3 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08844 root default
-3 0.02948 host node1
0 hdd 0.00980 osd.0 up 1.00000 1.00000
3 hdd 0.00980 osd.3 up 1.00000 1.00000
7 hdd 0.00989 osd.7 up 1.00000 1.00000
-5 0.02948 host node2
1 hdd 0.00980 osd.1 up 1.00000 1.00000
4 hdd 0.00980 osd.4 up 1.00000 1.00000
8 hdd 0.00989 osd.8 up 1.00000 1.00000
-7 0.02948 host node3
2 hdd 0.00980 osd.2 up 1.00000 1.00000
5 hdd 0.00980 osd.5 up 1.00000 1.00000
6 hdd 0.00989 osd.6 up 1.00000 1.00000
重现too few pgs错误：
创建一个pg数较小的存储池：
[root@node3 ~]# ceph osd pool create rbd 64 64
pool 'rbd' created
[root@node3 ~]# rados lspools
rbd
[root@node3 ~]# ceph -s
cluster:
id: b8b4aa68-d825-43e9-a60a-781c92fec20e
health: HEALTH_WARN
too few PGs per OSD (21 < min 30)
services:
mon: 1 daemons, quorum node1
mgr: node1(active)
osd: 9 osds: 9 up, 9 in
data:
pools: 1 pools, 64 pgs
objects: 0 objects, 0 bytes
usage: 9742 MB used, 82717 MB / 92459 MB avail
pgs: 64 active+clean
从上面可以看到，提示说每个osd上的pg数量小于最小的数目30个。pgs为64，因为是3副本的配置，所以当有9个osd的时候，每个osd上均分了64/9 *3=21个pgs,也就是出现了如上的错误 小于最小配置30个。
集群这种状态如果进行数据的存储和操作，会发现集群卡死，无法响应io，同时会导致大面积的osd down。
解决办法：修改默认pool rbd的pgs
[root@node3 ~]# ceph osd pool set rbd pg_num 128
set pool 1 pg_num to 128
之后查看集群状态
[root@node3 ~]# ceph -s
cluster:
id: b8b4aa68-d825-43e9-a60a-781c92fec20e
health: HEALTH_WARN
Reduced data availability: 5 pgs inactive, 44 pgs peering
Degraded data redundancy: 49 pgs unclean
1 pools have pg_num > pgp_num
services:
mon: 1 daemons, quorum node1
mgr: node1(active)
osd: 9 osds: 9 up, 9 in
data:
pools: 1 pools, 128 pgs
objects: 0 objects, 0 bytes
usage: 9743 MB used, 82716 MB / 92459 MB avail
pgs: 7.031% pgs unknown
38.281% pgs not active
70 active+clean
44 peering
9 unknown
5 activating
可以看到还没ok，提示pg_num 大于 pgp_num，所以还需要修改pgp_num
[root@node3 ~]# ceph osd pool set rbd pgp_num 128
set pool 1 pgp_num to 128
再次查看集群状态：
[root@node3 ~]# ceph -s
cluster:
id: b8b4aa68-d825-43e9-a60a-781c92fec20e
health: HEALTH_OK
services:
mon: 1 daemons, quorum node1
mgr: node1(active)
osd: 9 osds: 9 up, 9 in
data:
pools: 1 pools, 128 pgs
objects: 0 objects, 0 bytes
usage: 9750 MB used, 82709 MB / 92459 MB avail
pgs: 128 active+clean
这里是简单的实验，pool上也没有数据，所以修改pg影响并不大，但是如果是生产环境，这时候再重新修改pg数，会对生产环境产生较大影响。因为pg数变了，就会导致整个集群的数据重新均衡和迁移，数据越大响应io的时间会越长。所以，最好在一开始就设置好pg数。