zoukankan      html  css  js  c++  java
  • Glusterfs映射成block设备

    Glusterfs分布式存储软件可以创建集群文件设备,但有时我们需要将文件设备转换为block设备。如下示例为Glusterfs + tcmu,将文件设备转换为block设备。

     

    1. 环境说明:
       

      主机名

      IP

      设备

       

       

       

       

      存储节点

      oel7db01

      192.168.56.201

      在gluster上创建一个3副本的volume,然后在其中的一个存储节点上挂载这个volume,然后在这个gluster文件系统上创建5个文件,这5个文件后期用来模拟成block设备

      oel7db02

      192.168.56.202

      oel7db03

      192.168.56.203

      计算节点

      oel7db04

      192.168.56.204

      每个计算节点上最终会看到5个block设备,每个block设备来自于3个链路(因为有3个存储节点)

      oel7db05

      192.168.56.205

       

    2. 存储节点安装软件:

    yum install -y glusterfs-server

    yum install -y tcmu-runner

    yum install -y tcmu-runner-handler-glfs

    安装完成后,存储节点可以看到如下的初始状态:

    [root@oel7db01 network-scripts]# targetcli

    targetcli shell version 2.1.fb46

    Copyright 2011-2013 by Datera, Inc and others.

    For help on commands, type 'help'.

     

    /> ls

    o- / ......................................................................................................................... [...]

    o- backstores .............................................................................................................. [...]

    | o- block .................................................................................................. [Storage Objects: 0]

    | o- fileio ................................................................................................. [Storage Objects: 0]

    | o- pscsi .................................................................................................. [Storage Objects: 0]

    | o- ramdisk ................................................................................................ [Storage Objects: 0]

    | o- user:glfs .............................................................................................. [Storage Objects: 0]

    | o- user:qcow .............................................................................................. [Storage Objects: 0]

    o- iscsi ............................................................................................................ [Targets: 0]

    o- loopback ......................................................................................................... [Targets: 0]

    />

     

    1. 存储节点配置

    fallocate -l 2G file1.img

    fallocate -l 2G file2.img

    fallocate -l 2G file3.img

    fallocate -l 2G file4.img

    fallocate -l 2G file5.img

     

    运行targetcli命令进入targetcli中,继续运行如下命令:

    /backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img

    /backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img

    /backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img

    /backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img

    /backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img

     

    /iscsi/ create iqn.2016-04.org.gluster:oel7db02

     

    从每个客户端获取iscsi的iqn:

    more /etc/iscsi/initiatorname.iscsi

     

    创建本存储节点到该客户端的iqn连接

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c204

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/acls/ create iqn.1988-12.com.oracle:ea8fb3b19c205

     

     

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN1

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN2

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN3

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN4

    /iscsi/iqn.2016-04.org.gluster:oel7db02/tpg1/luns create /backstores/user:glfs/LUN5

     

    saveconfig

     

    需要在所有的存储节点上运行以上步骤。

    但在/backstores/user:glfs下创建LUN时,指定为当前节点的volume,例如:当前节点为oel7db02,则使用的命令如下(同时需要从第1个存储节点上获取刚刚生成的WWN号):

    /backstores/user:glfs create LUN1 2G gv1@oel7db02/file1.img 965b4d0e-fedc-4729-9748-625a397c5f00

    /backstores/user:glfs create LUN2 2G gv1@oel7db02/file2.img 69317210-2ea1-4c8b-87b3-98e52b35a788

    /backstores/user:glfs create LUN3 2G gv1@oel7db02/file3.img 0f7da034-945b-4da6-9fa3-1c54ae6a8d66

    /backstores/user:glfs create LUN4 2G gv1@oel7db02/file4.img d8304b6f-a77e-49cc-8a4e-fa95a34cbd93

    /backstores/user:glfs create LUN5 2G gv1@oel7db02/file5.img f0594c4c-8e43-4a30-abf1-e0beb3129e3c

     

    最终存储节点配置如下所示:

     

    4、计算节点配置

    iscsiadm -m discovery -t sendtargets -p 192.168.56.201

    iscsiadm -m discovery -t sendtargets -p 192.168.56.202

    iscsiadm -m discovery -t sendtargets -p 192.168.56.203

    iscsiadm -m node -l

     

    最终,计算节点能看到所有的盘:

    [root@oel7db04 ~]# fdisk -l |grep dev

    Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors

    /dev/sda1 * 2048 2099199 1048576 83 Linux

    /dev/sda2 2099200 83886079 40893440 8e Linux LVM

    Disk /dev/mapper/ol_oel7db-root: 37.7 GB, 37706792960 bytes, 73646080 sectors

    Disk /dev/mapper/ol_oel7db-swap: 4160 MB, 4160749568 bytes, 8126464 sectors

    Disk /dev/sdb: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdc: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdd: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sde: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdf: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdg: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdh: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdi: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdj: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdk: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdl: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdm: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdn: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdo: 2147 MB, 2147483648 bytes, 4194304 sectors

    Disk /dev/sdp: 2147 MB, 2147483648 bytes, 4194304 sectors

    [root@oel7db04 ~]#

     

     

    安装multipath:

    # yum install -y device-mapper-multipath

     

    cat > /etc/multipath.conf <<-EOF

    #

    # LIO iSCSI

    devices {

    device {

    vendor "LIO-ORG"

    path_grouping_policy "multibus"

    path_selector "queue-length 0"

    path_checker "directio"

    prio "const"

    rr_weight "uniform"

    }

    }

     

    EOF

     

    查看链路:

    [root@oel7db04 ~]# multipath -ll

    36001405f0594c4c8e434a30abf1e0beb dm-6 LIO-ORG ,TCMU device

    size=2.0G features='0' hwhandler='0' wp=rw

    `-+- policy='queue-length 0' prio=1 status=active

    |- 3:0:0:4 sdf 8:80 active ready running

    |- 4:0:0:4 sdk 8:160 active ready running

    `- 5:0:0:4 sdp 8:240 active ready running

    36001405d8304b6fa77e49cc8a4efa95a dm-5 LIO-ORG ,TCMU device

    size=2.0G features='0' hwhandler='0' wp=rw

    `-+- policy='queue-length 0' prio=1 status=active

    |- 3:0:0:3 sde 8:64 active ready running

    |- 4:0:0:3 sdj 8:144 active ready running

    `- 5:0:0:3 sdo 8:224 active ready running

    360014050f7da034945b4da69fa31c54a dm-4 LIO-ORG ,TCMU device

    size=2.0G features='0' hwhandler='0' wp=rw

    `-+- policy='queue-length 0' prio=1 status=active

    |- 3:0:0:2 sdd 8:48 active ready running

    |- 4:0:0:2 sdi 8:128 active ready running

    `- 5:0:0:2 sdn 8:208 active ready running

    36001405965b4d0efedc47299748625a3 dm-2 LIO-ORG ,TCMU device

    size=2.0G features='0' hwhandler='0' wp=rw

    `-+- policy='queue-length 0' prio=1 status=active

    |- 3:0:0:0 sdb 8:16 active ready running

    |- 4:0:0:0 sdg 8:96 active ready running

    `- 5:0:0:0 sdl 8:176 active ready running

    36001405693172102ea14c8b87b398e52 dm-3 LIO-ORG ,TCMU device

    size=2.0G features='0' hwhandler='0' wp=rw

    `-+- policy='queue-length 0' prio=1 status=active

    |- 3:0:0:1 sdc 8:32 active ready running

    |- 4:0:0:1 sdh 8:112 active ready running

    `- 5:0:0:1 sdm 8:192 active ready running

    [root@oel7db04 ~]#

     

    [root@oel7db04 disks]# lsscsi

    [1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0

    [2:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda

    [3:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdb

    [3:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdc

    [3:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdd

    [3:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sde

    [3:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdf

    [4:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdg

    [4:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdh

    [4:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdi

    [4:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sdj

    [4:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdk

    [5:0:0:0] disk LIO-ORG TCMU device 0002 /dev/sdl

    [5:0:0:1] disk LIO-ORG TCMU device 0002 /dev/sdm

    [5:0:0:2] disk LIO-ORG TCMU device 0002 /dev/sdn

    [5:0:0:3] disk LIO-ORG TCMU device 0002 /dev/sdo

    [5:0:0:4] disk LIO-ORG TCMU device 0002 /dev/sdp

    [root@oel7db04 disks]#

     

    至此,计算节点可以识别到在glusterfs上创建的5个文件设备,已经成功转换为block设备。

    后期,只需要将这些block设备通过udev进行绑定即可。

     

     

     

  • 相关阅读:
    Java 第十一届 蓝桥杯 省模拟赛 洁净数
    Java 第十一届 蓝桥杯 省模拟赛 第十层的二叉树
    Java 第十一届 蓝桥杯 省模拟赛 第十层的二叉树
    Java 第十一届 蓝桥杯 省模拟赛 第十层的二叉树
    Java 第十一届 蓝桥杯 省模拟赛 70044与113148的最大公约数
    Java 第十一届 蓝桥杯 省模拟赛 70044与113148的最大公约数
    20. Valid Parentheses
    290. Word Pattern
    205. Isomorphic Strings
    71. Simplify Path
  • 原文地址:https://www.cnblogs.com/missyou-shiyh/p/11546319.html
Copyright © 2011-2022 走看看