ceph journal更换位置

只在这里做简单的演示

ceotos7 环境

3个mon节点

3个osd节点

环境搭建我这里不再叙述

我们查看一下分区情况：

[root@ceph_1 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0   99G  0 part
  ├─cl-root 253:0    0   50G  0 lvm  /
  ├─cl-swap 253:1    0  7.8G  0 lvm  [SWAP]
  └─cl-home 253:2    0 41.2G  0 lvm  /home
sdb           8:16   0  100G  0 disk
├─sdb1        8:17   0   95G  0 part /var/lib/ceph/osd/ceph-0
└─sdb2        8:18   0    5G  0 part
sdc           8:32   0  100G  0 disk
sr0          11:0    1  680M  0 rom 

我们要把journal的分区又sdb2上挪到sdc盘上，sdc盘最好是ssd的盘，读写效率高

这里做之前我们先看一下各节点上sdb的分区

ceph_1节点上：

[root@ceph_1 ~]# sgdisk -i 1 /dev/sdb
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: B78B5DF4-FF30-4F43-89A9-CF6FD951F9FE
First sector: 10487808 (at 5.0 GiB)
Last sector: 209715166 (at 100.0 GiB)
Partition size: 199227359 sectors (95.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'

[root@ceph_1 ~]# sgdisk -i 2 /dev/sdb
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 4106357A-CB56-4099-AA4A-028431E91C11
First sector: 2048 (at 1024.0 KiB)
Last sector: 10487807 (at 5.0 GiB)
Partition size: 10485760 sectors (5.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

ceph_2节点上：

[root@ceph_2 ~]# sgdisk -i 1 /dev/sdb
Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: FA672CE1-E5E4-4FD3-8E42-9F453546D63D
First sector: 10487808 (at 5.0 GiB)
Last sector: 209715166 (at 100.0 GiB)
Partition size: 199227359 sectors (95.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'

[root@ceph_2 ~]# sgdisk -i 2 /dev/sdb
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 9C35A75C-E1AA-49E5-8DC3-BFA01304889D
First sector: 2048 (at 1024.0 KiB)
Last sector: 10487807 (at 5.0 GiB)
Partition size: 10485760 sectors (5.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

节点3的就不看了，也是一样的

对比可以得出一个结论：

ceph-osd 分区的都有一个固定的uuid：4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D

ceph-journal分区也有一个固定的uuid：45B0969E-9B03-4F30-B4C6-B4B80CEFF106

接下来就开始换盘的操作

首先先对sdc进行分区

[root@ceph_1 ~]# sgdisk -n 1:0:0 -c 1:"ceph journal" -t 1:45B0969E-9B03-4F30-B4C6-B4B80CEFF106 -g /dev/sdc            ////-c 指定name  -t指定uuid -g 转换为gpt格式
Creating new GPT entries.
The operation has completed successfully.
[root@ceph_1 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0   99G  0 part
  ├─cl-root 253:0    0   50G  0 lvm  /
  ├─cl-swap 253:1    0  7.8G  0 lvm  [SWAP]
  └─cl-home 253:2    0 41.2G  0 lvm  /home
sdb           8:16   0  100G  0 disk
├─sdb1        8:17   0   95G  0 part /var/lib/ceph/osd/ceph-0
└─sdb2        8:18   0    5G  0 part
sdc           8:32   0  100G  0 disk
└─sdc1        8:33   0  100G  0 part
sr0          11:0    1  680M  0 rom 

验证一下sdc是否按我们上面的uuid设置成功

[root@ceph_1 ~]# sgdisk -i 1 /dev/sdc
Partition GUID code: 45B0969E-9B03-4F30-B4C6-B4B80CEFF106 (Unknown)
Partition unique GUID: 2E7E105B-2C0E-4809-9B5B-B030D6B52F8E
First sector: 2048 (at 1024.0 KiB)
Last sector: 209715166 (at 100.0 GiB)
Partition size: 209713119 sectors (100.0 GiB)
Attribute flags: 0000000000000000
Partition name: 'ceph journal'

看一下各个磁盘对应的uuid，后面也需要用到这个

[root@ceph_1 ~]# ll /dev/disk/by-partuuid/
total 0
lrwxrwxrwx 1 root root 10 Oct 27 11:36 2e7e105b-2c0e-4809-9b5b-b030d6b52f8e -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct 27 11:26 4106357a-cb56-4099-aa4a-028431e91c11 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Oct 27 11:26 b78b5df4-ff30-4f43-89a9-cf6fd951f9fe -> ../../sdb1

看一下之前的journal是链接到哪个磁盘上的

[root@ceph_1 ~]# ll /var/lib/ceph/osd/ceph-0/journal
lrwxrwxrwx 1 ceph ceph 58 Oct 27 10:11 /var/lib/ceph/osd/ceph-0/journal -> /dev/disk/by-partuuid/4106357a-cb56-4099-aa4a-028431e91c11

由此可以看出之前的链接是指向sdb2分区上的，

我们要把journal分区sdb2换成sdc1，

第一步：设置策略   ceph osd set noout

[root@ceph_1 ~]# ceph osd set noout
set noout

第二步：停osd服务

systemctl stop ceph-osd@0

查看服务是否停掉：

[root@ceph_1 ~]# systemctl status ceph-osd@0
● ceph-osd@0.service - Ceph object storage daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2017-10-27 15:14:59 CST; 7s ago
 Main PID: 1495 (code=exited, status=0/SUCCESS)

Oct 27 11:21:25 ceph_1 systemd[1]: Starting Ceph object storage daemon...
Oct 27 11:21:28 ceph_1 ceph-osd-prestart.sh[1103]: create-or-move updated item name 'osd.0' weight 0.0488 at location {host=ceph_1,root=default} to crush map
Oct 27 11:21:28 ceph_1 systemd[1]: Started Ceph object storage daemon.
Oct 27 11:21:28 ceph_1 ceph-osd[1495]: starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
Oct 27 11:21:29 ceph_1 ceph-osd[1495]: 2017-10-27 11:21:29.719086 7f0bc5064800 -1 osd.0 32 log_to_monitors {default=true}
Oct 27 15:14:57 ceph_1 systemd[1]: Stopping Ceph object storage daemon...
Oct 27 15:14:57 ceph_1 ceph-osd[1495]: 2017-10-27 15:14:57.666313 7f0b9d2f1700 -1 osd.0 38 *** Got signal Terminated ***
Oct 27 15:14:57 ceph_1 ceph-osd[1495]: 2017-10-27 15:14:57.727750 7f0b9d2f1700 -1 osd.0 38 shutdown
Oct 27 15:14:59 ceph_1 systemd[1]: Stopped Ceph object storage daemon.

第三步：删除之前的软连接，或者移除掉之前的软连接

[root@ceph_1 ~]# rm -rf /var/lib/ceph/osd/ceph-0/journal

第四步：创建软连接并赋予权限：

[root@ceph_1 ~]# ln -fs /dev/disk/by-partuuid/2e7e105b-2c0e-4809-9b5b-b030d6b52f8e /var/lib/ceph/osd/ceph-0/journal     ///////2e7e105b-2c0e-4809-9b5b-b030d6b52f8e为sdc2的partuuid

[root@ceph_1 ~]# chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/journal

第五步：初始化journal

[root@ceph_1 ~]# ceph-osd --mkjournal -i 0
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2017-10-27 15:24:56.752101 7feb86a36800 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected b78b5df4-ff30-4f43-89a9-cf6fd951f9fe,
 invalid (someone else's?) journalSG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2017-10-27 15:24:56.758849 7feb86a36800 -1 created new journal /var/lib/ceph/osd/ceph-0/journal for object store /var/lib/ceph/osd/ceph-0

第六步：启动osd服务并查看状态

[root@ceph_1 ~]# systemctl start ceph-osd@0
[root@ceph_1 ~]# ceph osd tree
ID WEIGHT  TYPE NAME       UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.27809 root default                                     
-2 0.09270     host ceph_1                                  
 0 0.09270         osd.0        up  1.00000          1.00000
-3 0.09270     host ceph_2                                  
 1 0.09270         osd.1        up  1.00000          1.00000
-4 0.09270     host ceph_3                                  
 2 0.09270         osd.2        up  1.00000          1.00000
[root@ceph_1 ~]#

 第七步：取消策略

[root@ceph_1 ~]# ceph osd unset noout
unset noout

[root@ceph_1 ~]# ceph -w
    cluster bc9c1346-a31c-436d-ac52-f5c21cddeb91
     health HEALTH_OK
     monmap e2: 3 mons at {ceph_1=10.10.10.25:6789/0,ceph_2=10.10.10.26:6789/0,ceph_3=10.10.10.27:6789/0}
            election epoch 20, quorum 0,1,2 ceph_1,ceph_2,ceph_3
     osdmap e44: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds
      pgmap v118: 64 pgs, 1 pools, 0 bytes data, 0 objects
            105 MB used, 284 GB / 284 GB avail
                  64 active+clean

2017-10-27 15:27:29.084832 mon.0 [INF] HEALTH_OK