1-15-2-RAID5 企业级RAID磁盘阵列的搭建（RAID1、RAID5、RAID10）

RAID5的搭建

第一步：添加四个磁盘，开机并检查（略过）

第二步：使用fdisk命令分别对四个磁盘进行分区，效果如下图：

第三步：使用mdadm命令创建RAID5磁盘阵列

[root@localhost ~]# mdadm -Cv /dev/md5 -l 5 -n 3 -x 1 /dev/sd[b-e]1

注：-n 指定3个磁盘组成阵列、-x指定有一个热备盘（至少3个磁盘）

[root@localhost ~]# watch cat /proc/mdstat  #查看磁盘阵列信息

[root@localhost ~]# mdadm -Cv /dev/md5 -l 5 -n 3 -x 1 /dev/sd[b-e]1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: size set to 20954112K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdd1[4] sde1[3](S) sdc1[1] sdb1[0]
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      [======>..............]  recovery = 31.9% (6700756/20954112) finish=2.7min speed=85916K/sec
     
unused devices: <none>
[root@localhost ~]# watch cat !$
watch cat /proc/mdstat
[root@localhost ~]# mdadm -Dsv >> /etc/mdadm.conf
[root@localhost ~]# cat !$
cat /etc/mdadm.conf
ARRAY /dev/md5 level=raid5 num-devices=3 metadata=1.2 spares=2 name=localhost.localdomain:5 UUID=5f73c597:481d2f9e:df90d7de:1392d60a
   devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:09:54 2016
          State : clean, degraded, recovering
Active Devices : 2
Working Devices : 4
Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 512K

Rebuild Status : 68% complete

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 11

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       4       8       49        2      spare rebuilding   /dev/sdd1

       3       8       65        -      spare   /dev/sde1
[root@localhost ~]#

第四步：格式化并挂载

[root@localhost ~]# ls /dev/md5
[root@localhost ~]# mkdir /disk
[root@localhost ~]# mount /dev/md5 /disk  #挂载失败，因为没有对分区进行格式化
[root@localhost ~]# mkfs.xfs /dev/md5
[root@localhost ~]# mount /dev/md5 /disk
[root@localhost ~]# df -h | tail -1

第五步：设置开机自动挂载

方法同RAID1

第六步：故障模拟及修复

当损坏一块磁盘时修复：

[root@localhost ~]# echo "hello world! " >> /disk/a.txt  #检查文件
[root@localhost ~]# cat !$
cat /disk/a.txt
hello world!
[root@localhost ~]# mdadm /dev/md5 -f /dev/sdb1  #模拟磁盘损坏
mdadm: set /dev/sdb1 faulty in /dev/md5
[root@localhost ~]# mdadm -D /dev/md5  #检查磁盘状态
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:17:34 2016
          State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

Rebuild Status : 2% complete

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 21

    Number   Major   Minor   RaidDevice State
       3       8       65        0      spare rebuilding   /dev/sde1
       1       8       33        1      active sync   /dev/sdc1
       4       8       49        2      active sync   /dev/sdd1

       0       8       17        -      faulty   /dev/sdb1
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdd1[4] sde1[3] sdc1[1] sdb1[0](F)
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [=>...................]  recovery =  5.9% (1236864/20954112) finish=7.1min speed=45809K/sec
     
unused devices: <none>
[root@localhost ~]# cat /disk/a.txt    #检查磁盘中数据完整性
hello world!
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdd1[4] sde1[3] sdc1[1] sdb1[0](F)
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [=========>...........]  recovery = 48.8% (10229276/20954112) finish=4.4min speed=40192K/sec
     
unused devices: <none>
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:26:30 2016
          State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 48

    Number   Major   Minor   RaidDevice State
       3       8       65        0      active sync   /dev/sde1
       1       8       33        1      active sync   /dev/sdc1
       4       8       49        2      active sync   /dev/sdd1

       0       8       17        -      faulty   /dev/sdb1
[root@localhost ~]# df | grep disk
/dev/md5               41881600   33316  41848284   1% /disk
[root@localhost ~]# umount /disk
[root@localhost ~]# mdadm -S /dev/md5
mdadm: stopped /dev/md5
[root@localhost ~]# mdadm -zero-superblock /dev/sdb1
mdadm: -z does not set the mode, and so cannot be the first option.
[root@localhost ~]# mdadm --zero-superblock /dev/sdb1
[root@localhost ~]# mdadm -As /dev/md5
mdadm: /dev/md5 has been started with 3 drives.
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:30:34 2016
          State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 48

    Number   Major   Minor   RaidDevice State
       3       8       65        0      active sync   /dev/sde1
       1       8       33        1      active sync   /dev/sdc1
       4       8       49        2      active sync   /dev/sdd1
[root@localhost ~]# mdadm /dev/md5 -a /dev/sdb1   #只有当磁盘同步完成后，才可以添加磁盘
mdadm: added /dev/sdb1
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:31:40 2016
          State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 49

    Number   Major   Minor   RaidDevice State
       3       8       65        0      active sync   /dev/sde1
       1       8       33        1      active sync   /dev/sdc1
       4       8       49        2      active sync   /dev/sdd1

       5       8       17        -      spare   /dev/sdb1
模拟两块磁盘损坏

[root@localhost ~]# mdadm /dev/md5 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md5
[root@localhost ~]# mount -a
[root@localhost ~]# cat /disk/a.txt
hello world!
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:32:59 2016
          State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

Rebuild Status : 4% complete

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 53

    Number   Major   Minor   RaidDevice State
       3       8       65        0      active sync   /dev/sde1
       5       8       17        1      spare rebuilding   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1

       1       8       33        -      faulty   /dev/sdc1
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdb1[5] sde1[3] sdd1[4] sdc1[1](F)
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      [=>...................]  recovery =  5.2% (1091216/20954112) finish=10.8min speed=30344K/sec
     
unused devices: <none>
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdb1[5] sde1[3] sdd1[4] sdc1[1](F)
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      [=============>.......]  recovery = 66.8% (13998080/20954112) finish=3.9min speed=29308K/sec
     
unused devices: <none>
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdb1[5] sde1[3] sdd1[4] sdc1[1](F)
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
     
unused devices: <none>
[root@localhost ~]# mdadm /dev/md5 -f /dev/sde1
mdadm: set /dev/sde1 faulty in /dev/md5
[root@localhost ~]# cat /disk/a.txt
hello world!
[root@localhost ~]# clear
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:45:57 2016
          State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 2
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 72

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       5       8       17        1      active sync   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1

       1       8       33        -      faulty   /dev/sdc1
       3       8       65        -      faulty   /dev/sde1
[root@localhost ~]# df
Filesystem            1K-blocks    Used Available Use% Mounted on
/dev/mapper/rhel-root  10475520 3307104   7168416  32% /
devtmpfs                 485144       0    485144   0% /dev
tmpfs                    500680      88    500592   1% /dev/shm
tmpfs                    500680   13528    487152   3% /run
tmpfs                    500680       0    500680   0% /sys/fs/cgroup
/dev/sr0                3947824 3947824         0 100% /mnt
/dev/sda1                201388  127728     73660  64% /boot
tmpfs                    100136      16    100120   1% /run/user/0
/dev/md5               41881600   33316  41848284   1% /disk
[root@localhost ~]# umount /disk
[root@localhost ~]# mdadm -S /dev/md5
mdadm: stopped /dev/md5
[root@localhost ~]# mdadm --zero-superblock /dev/sdc1 /dev/sde1
[root@localhost ~]# mdadm -As /dev/md5
mdadm: /dev/md5 has been started with 2 drives (out of 3).
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:46:45 2016
          State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 74

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       5       8       17        1      active sync   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1
[root@localhost ~]# mdadm /dev/md5 -a /dev/sdc1
mdadm: added /dev/sdc1
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:47:47 2016
          State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

Rebuild Status : 0% complete

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 76

    Number   Major   Minor   RaidDevice State
       3       8       33        0      spare rebuilding   /dev/sdc1
       5       8       17        1      active sync   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1
[root@localhost ~]# mdadm /dev/md5 -a /dev/sde1
mdadm: /dev/md5 has failed so using --add cannot work and might destroy
mdadm: data on /dev/sde1.  You should stop the array and re-assemble it.
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdc1[3] sdb1[5] sdd1[4]
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [>....................]  recovery =  2.7% (571648/20954112) finish=10.8min speed=31424K/sec
     
unused devices: <none>
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdc1[3] sdb1[5] sdd1[4]
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [========>............]  recovery = 43.0% (9018880/20954112) finish=6.1min speed=32130K/sec
     
unused devices: <none>
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdc1[3] sdb1[5] sdd1[4]
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [================>....]  recovery = 80.7% (16911360/20954112) finish=1.8min speed=35542K/sec
     
unused devices: <none>
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md5 : active raid5 sdc1[3] sdb1[5] sdd1[4]
      41908224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
     
unused devices: <none>
[root@localhost ~]# mdadm /dev/md5 -a /dev/sde1
mdadm: added /dev/sde1
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:59:26 2016
          State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 95

    Number   Major   Minor   RaidDevice State
       3       8       33        0      active sync   /dev/sdc1
       5       8       17        1      active sync   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1

       6       8       65        -      spare   /dev/sde1
[root@localhost ~]#

模拟三块磁盘损坏

[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 02:59:26 2016
          State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 95

    Number   Major   Minor   RaidDevice State
       3       8       33        0      active sync   /dev/sdc1
       5       8       17        1      active sync   /dev/sdb1
       4       8       49        2      active sync   /dev/sdd1

       6       8       65        -      spare   /dev/sde1
[root@localhost ~]# mdadm -f /dev/sdc1
mdadm: /dev/sdc1 does not appear to be an md device
[root@localhost ~]# mdadm /dev/md5 -f /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md5
[root@localhost ~]# mdadm /dev/md5 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md5
[root@localhost ~]# mdadm /dev/md5 -f /dev/sde1
mdadm: set /dev/sde1 faulty in /dev/md5
[root@localhost ~]# mdadm -D /dev/md5
/dev/md5:
        Version : 1.2
  Creation Time : Sun Aug 21 02:07:09 2016
     Raid Level : raid5
     Array Size : 41908224 (39.97 GiB 42.91 GB)
  Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug 21 03:05:40 2016
          State : clean, FAILED
Active Devices : 1
Working Devices : 1
Failed Devices : 3
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : localhost.localdomain:5  (local to host localhost.localdomain)
           UUID : 5f73c597:481d2f9e:df90d7de:1392d60a
         Events : 4245

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       0        0        2      removed
       4       8       49        2      active sync   /dev/sdd1

       3       8       33        -      faulty   /dev/sdc1
       5       8       17        -      faulty   /dev/sdb1
       6       8       65        -      faulty   /dev/sde1
[root@localhost ~]# mount /dev/md5 /disk  #当损坏三块磁盘后，数据将不能读取！！！
mount: /dev/md5: can't read superblock
[root@localhost ~]#