GPFS on Redhat:
安装Redhat 6 服务器，一共五台，头节点，子节点两台，io节点两台
- node01
- node02
- ionode01
- ionode02
- mgmt3
磁盘配置
/gpfsdisk/nsdfile (GPFSDISK)这是虚拟机上的文件，用来模拟磁盘，物理机上可以用真实磁盘代替
/dev/sda (OS disk)
[root@mgmt3 gpfs3416_x86_64]# ls -lhtr total 11M -rw-r--r-- 1 root root 92K Sep 18 23:12 gpfs.msg.en_US-3.4.0-16.noarch.rpm -rw-r--r-- 1 root root 478K Sep 18 23:12 gpfs.gpl-3.4.0-16.noarch.rpm -rw-r--r-- 1 root root 200K Sep 18 23:12 gpfs.docs-3.4.0-16.noarch.rpm -rw-r--r-- 1 root root 9.6M Sep 18 23:12 gpfs.base-3.4.0-16.x86_64.rpm
在管理节点上安装编译需要的包
yum -y install compat-libstdc++-33.x86_64 kernel-headers kernel-devel gcc make gcc-c++ rpm-build
安装gpfs的所有包
rpm -ivh /root/gpfs3416_x86_64/gpfs*.rpm
编译gpfs的内核层文件，并打包成rpm格式
cd /usr/lpp/mmfs/src make clean make Autoconfig make World make InstallImages make rpm
安装编译好的rpm包
rpm -i /root/rpmbuild/RPMS/x86_64/gpfs.gplbin-2.6.32-279.el6.x86_64-3.4.0-16.x86_64.rpm
更改环境变量
vi /etc/profile # Path manipulation if [ "$EUID" = "0" ]; then pathmunge /sbin pathmunge /usr/sbin pathmunge /usr/local/sbin pathmunge /usr/lpp/mmfs/bin#加入这行 fi
将gpfs文件拷贝到其他服务器上以备安装
gpfs.base-3.4.0-16.x86_64.rpm (必须) gpfs.gplbin-2.6.32-279.el6.x86_64-3.4.0-16.x86_64.rpm(必须) gpfs.docs-3.4.0-16.noarch.rpm gpfs.msg.en_US-3.4.0-16.noarch.rpm
pscp gpfs.base-3.4.0-16.x86_64.rpm gpfs.docs-3.4.0-16.noarch.rpm gpfs.gplbin-2.6.32-279.el6.x86_64-3.4.0-16.x86_64.rpm gpfs.msg.en_US-3.4.0-16.noarch.rpm all:~/ psh all yum -y install compat-libstdc++-33.x86_64 ksh psh all rpm -ivh gpfs.gplbin-2.6.32-279.el6.x86_64-3.4.0-16.x86_64.rpm gpfs.base-3.4.0-16.x86_64.rpm gpfs.msg.en_US-3.4.0-16.noarch.rpm gpfs.docs-3.4.0-16.noarch.rpm
安装gpfs cluster
[root@ionode01 ~]# mmcrcluster -N ionode01:manager-quorum,ionode02:manager-quorum -p ionode01 -s ionode02 -r /usr/bin/ssh -R /usr/bin/scp -C ibmgpfs Warning: Permanently added 'ionode01.cluster.com,192.168.10.11' (RSA) to the list of known hosts. Wed Apr 10 17:57:45 CST 2013: mmcrcluster: Processing node ionode01.cluster.com Wed Apr 10 17:57:45 CST 2013: mmcrcluster: Processing node ionode02.cluster.com mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmcrcluster: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
检查一下
[root@centos-1 ~]# mmlscluster =============================================================================== | Warning: | | This cluster contains nodes that do not have a proper GPFS license | | designation. This violates the terms of the GPFS licensing agreement. | | Use the mmchlicense command and assign the appropriate GPFS licenses | | to each of the nodes in the cluster. For more information about GPFS | | license designation, see the Concepts, Planning, and Installation Guide. | =============================================================================== GPFS cluster information ======================== GPFS cluster name: ibmgpfs.ionode01 GPFS cluster id: 13882356895093578599 GPFS UID domain: ibmgpfs.ionode01 Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp GPFS cluster configuration servers: ----------------------------------- Primary server: ionode01 Secondary server: ionode02 Node Daemon node name IP address Admin node name Designation ----------------------------------------------------------------------------------------------- 1 ionode01 192.168.10.11 ionode01 quorum-manager 2 ionode02 192.168.10.12 ionode02 quorum-manager
将客户端添加到gpfs cluster中
[root@ionode01 ~]# mmaddnode -N mgmt3,node01,node02 Wed Apr 10 18:02:26 CST 2013: mmaddnode: Processing node mgmt3.cluster.com Wed Apr 10 18:02:36 CST 2013: mmaddnode: Processing node node01.cluster.com Wed Apr 10 18:02:37 CST 2013: mmaddnode: Processing node node02.cluster.com mmaddnode: Command successfully completed mmaddnode: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmaddnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. [root@ionode01 ~]# mmlscluster =============================================================================== | Warning: | | This cluster contains nodes that do not have a proper GPFS license | | designation. This violates the terms of the GPFS licensing agreement. | | Use the mmchlicense command and assign the appropriate GPFS licenses | | to each of the nodes in the cluster. For more information about GPFS | | license designation, see the Concepts, Planning, and Installation Guide. | =============================================================================== GPFS cluster information ======================== GPFS cluster name: ibmgpfs.cluster.com GPFS cluster id: 13882356895096059800 GPFS UID domain: ibmgpfs.cluster.com Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp GPFS cluster configuration servers: ----------------------------------- Primary server: ionode01.cluster.com Secondary server: ionode02.cluster.com Node Daemon node name IP address Admin node name Designation ----------------------------------------------------------------------------------------------- 1 ionode01.cluster.com 192.168.10.11 ionode01.cluster.com quorum-manager 2 ionode02.cluster.com 192.168.10.12 ionode02.cluster.com quorum-manager 3 mgmt3.cluster.com 192.168.10.254 mgmt3.cluster.com 4 node01.cluster.com 192.168.10.1 node01.cluster.com 5 node02.cluster.com 192.168.10.2 node02.cluster.com
安装license，其实就是一张纸。。。。
[root@ionode01 ~]# mmchlicense server --accept -N ionode01 The following nodes will be designated as possessing GPFS server licenses: ionode01.cluster.com mmchlicense: Command successfully completed mmchlicense: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmchlicense: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. mmchlicense server --accept -N ionode02 [root@ionode01 ~]# mmchlicense client --accept -N node01 [root@ionode01 ~]# mmchlicense client --accept -N node02 [root@ionode01 ~]# mmchlicense client --accept -N mgmt3
启动gpfs cluster
[root@ionode01 ~]# mmstartup -a Wed Apr 10 18:05:03 CST 2013: mmstartup: Starting GPFS ... [root@ionode01 ~]# mmgetstate -a Node number Node name GPFS state ------------------------------------------ 1 ionode01 active 2 ionode02 active 3 mgmt3 active 4 node01 active 5 node02 active
配置nsd文件，注意每行末尾不要留空格
vi /home/nsds /gpfsdisk/nsdfile:ionode01::dataAndMetadata::nsd1: /gpfsdisk/nsdfile:ionode02::dataAndMetadata::nsd2:
创建nsd
[root@ionode01 ~]# mmcrnsd -F /root/nsds -v yes mmcrnsd: Processing disk /gpfsdisk/nsdfile mmcrnsd: Processing disk /gpfsdisk/nsdfile mmcrnsd: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
创建文件系统
[root@ionode01 ~]# mmcrfs gpfs1 -F nsds -B 128k -T /gpfs The following disks of gpfs1 will be formatted on node ionode01: nsd1: size 2000000 KB nsd2: size 2000000 KB Formatting file system ... Disks up to size 25 GB can be added to storage pool 'system'. Creating Inode File Creating Allocation Maps Creating Log Files Clearing Inode Allocation Map Clearing Block Allocation Map Formatting Allocation Map for storage pool 'system' Completed creation of file system /dev/gpfs1. mmcrfs: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
挂载文件系统
[root@ionode01 ~]# mmmount /gpfs -a Wed Apr 10 18:17:05 CST 2013: mmmount: Mounting file systems ...
最后赞一下ibm的gpfs，性能好不说，绝对良心产品，装过的人都懂。