greenplum是2(master)+7(segment)的集群规模
系统刚准备上线，是用来做统计数据库的，正在帮忙一个hadoop集群核对其数据的准确性，在这个greenplum库中入了清单数据
后检查分析是部分建表语句存在问题，没有指定字段做分布键，也没有指定其是随机分布，导致默认为第一个字段做为分布键导致数据倾斜。
发现数据库非常慢，几乎是不可用，检查greenplum的状态情况
1、检查greenplum数据库的状态
gpadmin@mdw:~> gpstate 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.3.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 10 2014 14:31:50' 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:15:20:13:007202 gpstate:mdw:gpadmin-[INFO]:-Gathering data from segments... ...... 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:-Greenplum instance status summary 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Master instance = Active 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Master standby = smdw 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Standby master state = Standby host passive 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total segment instance count from metadata = 56 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Primary Segment Status 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total primary segments = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total primary segment valid (at master) = 24 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[WARNING]:-Total primary segment failures (at master) = 4 <<<<<<<< 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 28 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number postmaster processes missing = 4 <<<<<<<< 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 24 20150227:15:20:19:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Mirror Segment Status 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total mirror segments = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total mirror segment valid (at master) = 21 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total mirror segment failures (at master) = 7 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid files found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number of /tmp lock files found = 28 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number postmaster processes missing = 4 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number postmaster processes found = 24 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments = 4 <<<<<<<< 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 24 20150227:15:20:20:007202 gpstate:mdw:gpadmin-[INFO]:-----------------------------------------------------
2、检查数据库服务器的情况
gpadmin@mdw:/> gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 "df -hT" [sdw4] df: "/root/.gvfs": 权限不够 [sdw4] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw4] /dev/sda2 ext3 99G 5.7G 88G 7% / [sdw4] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw4] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw4] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw4] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw4] /dev/sdb xfs 4.6T 3.8T 865G 82% /data [sdw5] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw5] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw5] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw5] tmpfs tmpfs 32G 88K 32G 1% /dev/shm [sdw5] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw5] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw5] /dev/sdb xfs 4.6T 2.4T 2.2T 53% /data [sdw6] df: "/root/.gvfs": 权限不够 [sdw6] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw6] /dev/sda2 ext3 99G 910M 93G 1% / [sdw6] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw6] tmpfs tmpfs 47G 100K 47G 1% /dev/shm [sdw6] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw6] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw6] /dev/sda3 ext3 63G 5.0G 55G 9% /usr [sdw6] /dev/sdb xfs 4.6T 4.5T 93G 99% /data [sdw6] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [sdw7] df: "/root/.gvfs": 权限不够 [sdw7] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw7] /dev/sda2 ext3 60G 440M 56G 1% / [sdw7] devtmpfs devtmpfs 32G 244K 32G 1% /dev [sdw7] tmpfs tmpfs 95G 112K 95G 1% /dev/shm [sdw7] /dev/sda1 ext3 9.9G 180M 9.2G 2% /boot [sdw7] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw7] /dev/sda6 ext3 9.9G 264M 9.1G 3% /opt [sdw7] /dev/sda8 ext3 9.9G 151M 9.2G 2% /srv [sdw7] /dev/sda7 ext3 9.9G 162M 9.2G 2% /tmp [sdw7] /dev/sda9 ext3 40G 4.8G 33G 13% /usr [sdw7] /dev/sda10 ext3 9.9G 358M 9.0G 4% /var [sdw7] /dev/sdb xfs 4.6T 3.7T 943G 80% /data [sdw1] df: "/root/.gvfs": 权限不够 [sdw1] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw1] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw1] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw1] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw1] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw1] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw1] /dev/sdb xfs 4.6T 3.6T 1.1T 78% /data [sdw2] df: "/root/.gvfs": 权限不够 [sdw2] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw2] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw2] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw2] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw2] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw2] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw2] /dev/sdb xfs 4.6T 4.6T 2.0G 100% /data [sdw3] df: "/root/.gvfs": 权限不够 [sdw3] 文件系统 类型 容量 已用 可用 已用% 挂载瀿 [sdw3] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw3] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw3] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [sdw3] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw3] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw3] /dev/sdb xfs 4.6T 3.8T 856G 82% /data 发现segment中的sdw2和sdw6的数据空间/data目录的收益率已经达到100%。
3、数据库上删除清单表
用psql连接数据库，执行drop table 表名，执行后，等待大半天也没反应，使用gpssh -f检查服务器的io使用情况，segment服务器的IO没有读写操作，证明数据库已经没办法分发命令下去。
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
4、分别登陆sdw2和sdw6
#cd /data/primary/gpseg4/gp_log
#ls -ltrh
目录下有gpdb-日期.csv文件，直接执行
#rm -rf *.csv
每个gpseg下的gp_log都执行同样的删除操作，执行完后，空间释放得非常少。
=> df -hT [sdw4] df: "/root/.gvfs": 权限不够 [sdw4] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw4] /dev/sda2 ext3 99G 5.7G 88G 7% / [sdw4] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw4] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw4] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw4] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw4] /dev/sdb xfs 4.6T 3.7T 868G 82% /data [sdw5] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw5] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw5] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw5] tmpfs tmpfs 32G 88K 32G 1% /dev/shm [sdw5] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw5] /dev/sda5 ext3 785G 197M 745G 1% /home [sdw5] /dev/sdb xfs 4.6T 2.4T 2.2T 53% /data [sdw6] df: "/root/.gvfs": 权限不够 [sdw6] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw6] /dev/sda2 ext3 99G 911M 93G 1% / [sdw6] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw6] tmpfs tmpfs 47G 100K 47G 1% /dev/shm [sdw6] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw6] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw6] /dev/sda3 ext3 63G 5.0G 55G 9% /usr [sdw6] /dev/sdb xfs 4.6T 4.5T 96G 98% /data [sdw6] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [sdw7] df: "/root/.gvfs": 权限不够 [sdw7] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw7] /dev/sda2 ext3 60G 440M 56G 1% / [sdw7] devtmpfs devtmpfs 32G 244K 32G 1% /dev [sdw7] tmpfs tmpfs 95G 112K 95G 1% /dev/shm [sdw7] /dev/sda1 ext3 9.9G 180M 9.2G 2% /boot [sdw7] /dev/sda5 ext3 197G 188M 187G 1% /home [sdw7] /dev/sda6 ext3 9.9G 264M 9.1G 3% /opt [sdw7] /dev/sda8 ext3 9.9G 151M 9.2G 2% /srv [sdw7] /dev/sda7 ext3 9.9G 162M 9.2G 2% /tmp [sdw7] /dev/sda9 ext3 40G 4.8G 33G 13% /usr [sdw7] /dev/sda10 ext3 9.9G 359M 9.0G 4% /var [sdw7] /dev/sdb xfs 4.6T 3.7T 945G 80% /data [sdw1] df: "/root/.gvfs": 权限不够 [sdw1] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw1] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw1] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw1] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw1] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw1] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw1] /dev/sdb xfs 4.6T 3.6T 1.1T 78% /data [sdw2] df: "/root/.gvfs": 权限不够 [sdw2] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw2] /dev/sda1 ext3 99G 5.7G 88G 7% / [sdw2] devtmpfs devtmpfs 32G 444K 32G 1% /dev [sdw2] tmpfs tmpfs 95G 100K 95G 1% /dev/shm [sdw2] /dev/sda2 ext3 7.9G 216M 7.3G 3% /boot [sdw2] /dev/sda3 ext3 197G 188M 187G 1% /home [sdw2] /dev/sdb xfs 4.6T 4.6T 21G 100% /data [sdw3] df: "/root/.gvfs": 权限不够 [sdw3] 文件系统 类型 容量 已用 可用 已用% 挂载点 [sdw3] /dev/sda2 ext3 60G 5.5G 51G 10% / [sdw3] devtmpfs devtmpfs 32G 448K 32G 1% /dev [sdw3] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [sdw3] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [sdw3] /dev/sda5 ext3 785G 198M 745G 1% /home [sdw3] /dev/sdb xfs 4.6T 3.8T 859G 82% /data [smdw] df: "/root/.gvfs": 权限不够 [smdw] 文件系统 类型 容量 已用 可用 已用% 挂载点 [smdw] /dev/sda2 ext3 60G 5.8G 51G 11% / [smdw] devtmpfs devtmpfs 32G 448K 32G 1% /dev [smdw] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [smdw] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [smdw] /dev/sda5 ext3 40G 23G 16G 60% /home [smdw] /dev/sda6 xfs 757G 3.3G 754G 1% /data [smdw] /dev/sr0 iso9660 3.1G 3.1G 0 100% /media/SLES-11-SP2-DVD-x86_6407551 [ mdw] df: "/root/.gvfs": 权限不够 [ mdw] 文件系统 类型 容量 已用 可用 已用% 挂载点 [ mdw] /dev/sda2 ext3 60G 6.4G 50G 12% / [ mdw] devtmpfs devtmpfs 32G 456K 32G 1% /dev [ mdw] tmpfs tmpfs 32G 100K 32G 1% /dev/shm [ mdw] /dev/sda1 ext3 9.9G 220M 9.2G 3% /boot [ mdw] /dev/sda5 ext3 40G 281M 38G 1% /home [ mdw] /dev/sda6 xfs 757G 6.4G 751G 1% /data
5、停数据库
执行gpstop
报：-gpstop failed(Reason='FATAL: the database system is shutting down') exiting ...
gpstop停不下来。
6、检查进程
gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 -h mdw -h smdw "ps -ef |grep postgres"
数据库segment上进程没有停。下面是一台segment上的进程：
[sdw4] gpadmin 15168 31843 0 Feb26 ? 00:00:14 postgres: port 40000, cems cems 198.168.11.11(52166) con6959 seg12 idle in transaction [sdw4] gpadmin 15170 31838 0 Feb26 ? 00:00:15 postgres: port 40001, cems cems 198.168.11.11(38065) con6959 seg13 idle in transaction [sdw4] gpadmin 15172 31841 0 Feb26 ? 00:00:16 postgres: port 40002, cems cems 198.168.11.11(3175) con6959 seg14 idle in transaction [sdw4] gpadmin 15174 31837 0 Feb26 ? 00:00:16 postgres: port 40003, cems cems 198.168.11.11(31152) con6959 seg15 idle in transaction [sdw4] gpadmin 15176 31843 0 Feb26 ? 00:00:04 postgres: port 40000, cems cems 198.168.11.11(52194) con6959 seg12 idle [sdw4] gpadmin 15178 31838 0 Feb26 ? 00:00:04 postgres: port 40001, cems cems 198.168.11.11(38093) con6959 seg13 idle [sdw4] gpadmin 15180 31841 0 Feb26 ? 00:00:04 postgres: port 40002, cems cems 198.168.11.11(3203) con6959 seg14 idle [sdw4] gpadmin 15182 31837 0 Feb26 ? 00:00:04 postgres: port 40003, cems cems 198.168.11.11(31180) con6959 seg15 idle [sdw4] gpadmin 15204 31843 0 Feb26 ? 00:00:16 postgres: port 40000, cems cems 198.168.11.11(52298) con6949 seg12 idle in transaction [sdw4] gpadmin 15206 31838 0 Feb26 ? 00:00:15 postgres: port 40001, cems cems 198.168.11.11(38197) con6949 seg13 idle in transaction [sdw4] gpadmin 15208 31841 0 Feb26 ? 00:00:16 postgres: port 40002, cems cems 198.168.11.11(3307) con6949 seg14 idle in transaction [sdw4] gpadmin 15210 31837 0 Feb26 ? 00:00:15 postgres: port 40003, cems cems 198.168.11.11(31284) con6949 seg15 idle in transaction [sdw4] gpadmin 31836 1 0 Feb04 ? 00:00:04 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg27 -p 50003 -b 57 -z 28 --silent-mode=true -i -M quiescent -C 27 [sdw4] gpadmin 31837 1 0 Feb04 ? 00:10:13 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg15 -p 40003 -b 17 -z 28 --silent-mode=true -i -M quiescent -C 15 [sdw4] gpadmin 31838 1 0 Feb04 ? 00:10:11 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg13 -p 40001 -b 15 -z 28 --silent-mode=true -i -M quiescent -C 13 [sdw4] gpadmin 31839 1 0 Feb04 ? 00:00:05 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg5 -p 50001 -b 35 -z 28 --silent-mode=true -i -M quiescent -C 5 [sdw4] gpadmin 31840 1 0 Feb04 ? 00:00:03 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg2 -p 50002 -b 32 -z 28 --silent-mode=true -i -M quiescent -C 2 [sdw4] gpadmin 31841 1 0 Feb04 ? 00:10:20 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg14 -p 40002 -b 16 -z 28 --silent-mode=true -i -M quiescent -C 14 [sdw4] gpadmin 31842 1 0 Feb04 ? 00:00:04 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/mirror/gpseg8 -p 50000 -b 38 -z 28 --silent-mode=true -i -M quiescent -C 8 [sdw4] gpadmin 31843 1 0 Feb04 ? 00:10:26 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/primary/gpseg12 -p 40000 -b 14 -z 28 --silent-mode=true -i -M quiescent -C 12 [sdw4] gpadmin 31844 31841 0 Feb04 ? 00:01:15 postgres: port 40002, logger process [sdw4] gpadmin 31845 31838 0 Feb04 ? 00:01:21 postgres: port 40001, logger process [sdw4] gpadmin 31846 31839 0 Feb04 ? 00:01:07 postgres: port 50001, logger process [sdw4] gpadmin 31847 31836 0 Feb04 ? 00:01:13 postgres: port 50003, logger process [sdw4] gpadmin 31848 31843 0 Feb04 ? 00:01:19 postgres: port 40000, logger process [sdw4] gpadmin 31849 31842 0 Feb04 ? 00:01:07 postgres: port 50000, logger process [sdw4] gpadmin 31850 31837 0 Feb04 ? 00:01:25 postgres: port 40003, logger process [sdw4] gpadmin 31851 31840 0 Feb04 ? 00:01:04 postgres: port 50002, logger process [sdw4] gpadmin 31868 31836 0 Feb04 ? 00:08:55 postgres: port 50003, mirror process [sdw4] gpadmin 31871 31837 0 Feb04 ? 00:09:13 postgres: port 40003, primary process [sdw4] gpadmin 31873 31841 0 Feb04 ? 00:08:56 postgres: port 40002, primary process [sdw4] gpadmin 31874 31868 0 Feb04 ? 01:10:10 postgres: port 50003, mirror receiver process [sdw4] gpadmin 31876 31868 0 Feb04 ? 00:34:17 postgres: port 50003, mirror consumer process [sdw4] gpadmin 31877 31868 0 Feb04 ? 00:12:24 postgres: port 50003, mirror consumer writer process [sdw4] gpadmin 31878 31868 0 Feb04 ? 00:36:29 postgres: port 50003, mirror consumer append only process [sdw4] gpadmin 31879 31838 0 Feb04 ? 00:08:55 postgres: port 40001, primary process [sdw4] gpadmin 31881 31868 0 Feb04 ? 00:07:44 postgres: port 50003, mirror sender ack process [sdw4] gpadmin 31882 31868 0 Feb04 ? 00:00:03 postgres: port 50003, mirror verification process [sdw4] gpadmin 31883 31871 0 Feb04 ? 00:07:12 postgres: port 40003, primary receiver ack process [sdw4] gpadmin 31885 31871 0 Feb04 ? 01:15:22 postgres: port 40003, primary sender process [sdw4] gpadmin 31886 31871 0 Feb04 ? 00:07:00 postgres: port 40003, primary consumer ack process [sdw4] gpadmin 31887 31871 0 Feb04 ? 00:15:41 postgres: port 40003, primary recovery process [sdw4] gpadmin 31888 31842 0 Feb04 ? 00:09:02 postgres: port 50000, mirror process [sdw4] gpadmin 31889 31871 0 Feb04 ? 00:01:08 postgres: port 40003, primary verification process [sdw4] gpadmin 31892 31873 0 Feb04 ? 00:07:19 postgres: port 40002, primary receiver ack process [sdw4] gpadmin 31893 31873 0 Feb04 ? 01:10:48 postgres: port 40002, primary sender process [sdw4] gpadmin 31894 31873 0 Feb04 ? 00:06:47 postgres: port 40002, primary consumer ack process [sdw4] gpadmin 31895 31873 0 Feb04 ? 00:15:34 postgres: port 40002, primary recovery process [sdw4] gpadmin 31896 31873 0 Feb04 ? 00:01:13 postgres: port 40002, primary verification process [sdw4] gpadmin 31898 31839 0 Feb04 ? 00:09:08 postgres: port 50001, mirror process [sdw4] gpadmin 31900 31840 0 Feb04 ? 00:09:03 postgres: port 50002, mirror process [sdw4] gpadmin 31901 31879 0 Feb04 ? 00:07:05 postgres: port 40001, primary receiver ack process [sdw4] gpadmin 31902 31879 0 Feb04 ? 01:07:19 postgres: port 40001, primary sender process [sdw4] gpadmin 31903 31879 0 Feb04 ? 00:09:53 postgres: port 40001, primary consumer ack process [sdw4] gpadmin 31904 31879 0 Feb04 ? 00:15:36 postgres: port 40001, primary recovery process [sdw4] gpadmin 31905 31879 0 Feb04 ? 00:01:13 postgres: port 40001, primary verification process [sdw4] gpadmin 31911 31888 0 Feb04 ? 01:11:45 postgres: port 50000, mirror receiver process [sdw4] gpadmin 31912 31888 0 Feb04 ? 00:34:47 postgres: port 50000, mirror consumer process [sdw4] gpadmin 31913 31898 0 Feb04 ? 01:19:23 postgres: port 50001, mirror receiver process [sdw4] gpadmin 31914 31888 0 Feb04 ? 00:13:06 postgres: port 50000, mirror consumer writer process [sdw4] gpadmin 31915 31898 0 Feb04 ? 00:38:56 postgres: port 50001, mirror consumer process [sdw4] gpadmin 31916 31888 0 Feb04 ? 00:36:17 postgres: port 50000, mirror consumer append only process [sdw4] gpadmin 31917 31898 0 Feb04 ? 00:19:13 postgres: port 50001, mirror consumer writer process [sdw4] gpadmin 31919 31898 0 Feb04 ? 00:36:00 postgres: port 50001, mirror consumer append only process [sdw4] gpadmin 31920 31888 0 Feb04 ? 00:07:49 postgres: port 50000, mirror sender ack process [sdw4] gpadmin 31922 31888 0 Feb04 ? 00:00:03 postgres: port 50000, mirror verification process [sdw4] gpadmin 31923 31898 0 Feb04 ? 00:08:11 postgres: port 50001, mirror sender ack process [sdw4] gpadmin 31924 31898 0 Feb04 ? 00:00:03 postgres: port 50001, mirror verification process [sdw4] gpadmin 31925 31900 0 Feb04 ? 01:05:21 postgres: port 50002, mirror receiver process [sdw4] gpadmin 31926 31900 0 Feb04 ? 00:34:34 postgres: port 50002, mirror consumer process [sdw4] gpadmin 31927 31900 0 Feb04 ? 00:12:25 postgres: port 50002, mirror consumer writer process [sdw4] gpadmin 31928 31900 0 Feb04 ? 00:36:15 postgres: port 50002, mirror consumer append only process [sdw4] gpadmin 31930 31900 0 Feb04 ? 00:07:48 postgres: port 50002, mirror sender ack process [sdw4] gpadmin 31931 31900 0 Feb04 ? 00:00:03 postgres: port 50002, mirror verification process [sdw4] gpadmin 31937 31843 0 Feb04 ? 00:00:40 postgres: port 40000, stats collector process [sdw4] gpadmin 31938 31843 0 Feb04 ? 00:07:18 postgres: port 40000, writer process [sdw4] gpadmin 31939 31843 0 Feb04 ? 00:02:02 postgres: port 40000, checkpoint process [sdw4] gpadmin 31940 31843 0 Feb04 ? 00:01:52 postgres: port 40000, sweeper process [sdw4] gpadmin 31944 31837 0 Feb04 ? 00:00:40 postgres: port 40003, stats collector process [sdw4] gpadmin 31945 31837 0 Feb04 ? 00:07:55 postgres: port 40003, writer process [sdw4] gpadmin 31946 31837 0 Feb04 ? 00:03:28 postgres: port 40003, checkpoint process [sdw4] gpadmin 31947 31837 0 Feb04 ? 00:01:50 postgres: port 40003, sweeper process [sdw4] gpadmin 31948 31841 0 Feb04 ? 00:00:36 postgres: port 40002, stats collector process [sdw4] gpadmin 31949 31841 0 Feb04 ? 00:08:00 postgres: port 40002, writer process [sdw4] gpadmin 31950 31841 0 Feb04 ? 00:02:49 postgres: port 40002, checkpoint process [sdw4] gpadmin 31951 31841 0 Feb04 ? 00:01:52 postgres: port 40002, sweeper process [sdw4] gpadmin 31952 31838 0 Feb04 ? 00:00:36 postgres: port 40001, stats collector process [sdw4] gpadmin 31953 31838 0 Feb04 ? 00:07:39 postgres: port 40001, writer process [sdw4] gpadmin 31954 31838 0 Feb04 ? 00:02:36 postgres: port 40001, checkpoint process [sdw4] gpadmin 31955 31838 0 Feb04 ? 00:01:47 postgres: port 40001, sweeper process [sdw4] gpadmin 37706 37670 0 17:52 pts/0 00:00:00 grep postgres [sdw4] gpadmin 42312 31843 0 Feb12 ? 00:02:50 postgres: port 40000, primary process [sdw4] gpadmin 42313 42312 0 Feb12 ? 00:09:58 postgres: port 40000, primary recovery process
7、在master上执行
#gpstop -af
报：-gpstop failed(Reason='FATAL: the database system is shutting down') exiting ...
同样停不了库
gpadmin@mdw:~> gpstop -M fast 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -M fast 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:42:01:021768 gpstop:mdw:gpadmin-[CRITICAL]:-gpstop failed. (Reason='FATAL: the database system is shutting down ') exiting...
同样不行。
8、直接kill
gpssh -h sdw1 -h sdw2 -h sdw3 -h sdw4 -h sdw5 -h sdw6 -h sdw7 -h mdw -h smdw "for i in `ps -ef |grep postgres |awk {'print $2'}`;do kill $i; done"
使用上面的检查，segment上的进程都已停止，但是master上的部分进程没有停。
9、停止master进程
mdw -h smdw ps -ef |grep postgres [ mdw] gpadmin 15765 15724 0 17:52 pts/11 00:00:00 grep postgres [ mdw] gpadmin 38876 41830 0 Feb26 ? 00:05:11 postgres: port 5432, cems cems 198.168.11.12(51943) con6959 198.168.11.12(51943) cmd24705 BIND [ mdw] gpadmin 41830 1 0 Feb04 ? 00:00:07 /usr/local/greenplum-db-4.3.3.1/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 28 --silent-mode=true -i -M master -C -1 -x 58 -E [ mdw] gpadmin 41838 41830 0 Feb04 ? 00:10:18 postgres: port 5432, master logger process [ mdw] gpadmin 41844 41830 0 Feb04 ? 00:00:18 postgres: port 5432, stats collector process [ mdw] gpadmin 41845 41830 0 Feb04 ? 00:05:02 postgres: port 5432, writer process [ mdw] gpadmin 41846 41830 0 Feb04 ? 00:00:55 postgres: port 5432, checkpoint process [ mdw] gpadmin 41847 41830 0 Feb04 ? 00:00:13 postgres: port 5432, seqserver process [ mdw] gpadmin 41848 41830 0 Feb04 ? 00:04:33 postgres: port 5432, ftsprobe process [ mdw] gpadmin 41851 41830 0 Feb04 ? 00:00:49 postgres: port 5432, sweeper process
执行gkill postgres，没反应。
直接使用kill 41830，没反应，不行。
执行pg_ctl stop -D /data/master/gpseg-1
>pg_ctl stop -D /data/master/gpseg-1
waiting for server to shut down .........................................failed
pg_ctl:server does not shut down
应是新版本中对于进程做了保护，不能直接kill了。
9、使用pg_ctl停库
gpadmin@mdw:~> pg_ctl --help pg_ctl is a utility to start, stop, restart, reload configuration files, report the status of a PostgreSQL server, or signal a PostgreSQL process.
Usage: pg_ctl start [-w] [-t SECS] [-D DATADIR] [-s] [-l FILENAME] [-o "OPTIONS"] pg_ctl stop [-W] [-t SECS] [-D DATADIR] [-s] [-m SHUTDOWN-MODE] pg_ctl restart [-w] [-t SECS] [-D DATADIR] [-s] [-m SHUTDOWN-MODE] [-o "OPTIONS"] pg_ctl reload [-D DATADIR] [-s] pg_ctl status [-D DATADIR] pg_ctl kill SIGNALNAME PID
Common options: -D, --pgdata DATADIR location of the database storage area -s, --silent only print errors, no informational messages -t SECS seconds to wait when using -w option -w wait until operation completes -W do not wait until operation completes --help show this help, then exit --version output version information, then exit --gp-version output Greenplum version information, then exit (The default is to wait for shutdown, but not for start or restart.)
If the -D option is omitted, the environment variable PGDATA is used.
Options for start or restart: -l, --log FILENAME write (or append) server log to FILENAME -o OPTIONS command line options to pass to postgres (PostgreSQL server executable) -p PATH-TO-POSTGRES normally not necessary -c, --core-files allow postgres to produce core files
Options for stop or restart: -m SHUTDOWN-MODE can be "smart", "fast", or "immediate"
Shutdown modes are: smart quit after all clients have disconnected fast quit directly, with proper shutdown immediate quit without complete shutdown; will lead to recovery on restart
Allowed signal names for kill: HUP INT QUIT ABRT TERM USR1 USR2
Report bugs to <pgsql-bugs@postgresql.org>. gpadmin@mdw:~> pg_ctl stop -m immedidate pg_ctl: unrecognized shutdown mode "immedidate" Try "pg_ctl --help" for more information. gpadmin@mdw:~> pg_ctl stop -D /data/master/gpseg-1 -m immedidate pg_ctl: unrecognized shutdown mode "immedidate" Try "pg_ctl --help" for more information. gpadmin@mdw:~> pg_ctl stop -D /data/master/gpseg-1 -m immediate waiting for server to shut down.... done server stopped gpadmin@mdw:~> ps -ef |grep post root 4824 1 0 Feb03 ? 00:00:17 /usr/lib/postfix/master postfix 4857 4824 0 Feb03 ? 00:00:01 qmgr -l -t fifo -u gpadmin 21828 21731 0 18:47 pts/1 00:00:00 grep post
10、启动数据库
gpadmin@mdw:~> gpstart -m 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150' 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss. 20150227:18:47:10:021829 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************
Continue with master-only startup Yy|Nn (default=N): > y 20150227:18:47:13:021829 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20150227:18:47:15:021829 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:47:15:021829 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:47:16:021829 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20150227:18:47:16:021829 gpstart:mdw:gpadmin-[INFO]:-Master Started... gpadmin@mdw:~> gpstop -af 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -af 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:47:28:021878 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='fast' 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Detected 0 connections to database 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Using standard WAIT mode of 600 seconds 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=fast 20150227:18:47:29:021878 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o 'StrictHostKeyChecking no' smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /data/master/gpseg-1 -m fast -w -t 600 stop"' cmd had rc=1 completed=True halted=False stdout='' stderr='pg_ctl: PID file "/data/master/gpseg-1/postmaster.pid" does not exist Is server running? ' 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20150227:18:47:30:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... ........................................................................................................................................................................................................................................................................................................................................... ............................................................................................................................................................................... ....................................................................................................... 20150227:18:57:40:021878 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... .. 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-Failed Segment Stop Information 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:14 FAILED host:'sdw4' datadir:'/data/primary/gpseg12' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:15 FAILED host:'sdw4' datadir:'/data/primary/gpseg13' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:16 FAILED host:'sdw4' datadir:'/data/primary/gpseg14' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:17 FAILED host:'sdw4' datadir:'/data/primary/gpseg15' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:46 FAILED host:'sdw6' datadir:'/data/mirror/gpseg16' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:22 FAILED host:'sdw6' datadir:'/data/primary/gpseg20' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:23 FAILED host:'sdw6' datadir:'/data/primary/gpseg21' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:24 FAILED host:'sdw6' datadir:'/data/primary/gpseg22' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:25 FAILED host:'sdw6' datadir:'/data/primary/gpseg23' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:47 FAILED host:'sdw7' datadir:'/data/mirror/gpseg17' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:26 FAILED host:'sdw7' datadir:'/data/primary/gpseg24' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:27 FAILED host:'sdw7' datadir:'/data/primary/gpseg25' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:28 FAILED host:'sdw7' datadir:'/data/primary/gpseg26' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:29 FAILED host:'sdw7' datadir:'/data/primary/gpseg27' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:10 FAILED host:'sdw3' datadir:'/data/primary/gpseg8' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:11 FAILED host:'sdw3' datadir:'/data/primary/gpseg9' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:12 FAILED host:'sdw3' datadir:'/data/primary/gpseg10' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:13 FAILED host:'sdw3' datadir:'/data/primary/gpseg11' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:2 FAILED host:'sdw1' datadir:'/data/primary/gpseg0' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:3 FAILED host:'sdw1' datadir:'/data/primary/gpseg1' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:4 FAILED host:'sdw1' datadir:'/data/primary/gpseg2' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:5 FAILED host:'sdw1' datadir:'/data/primary/gpseg3' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:48 FAILED host:'sdw1' datadir:'/data/mirror/gpseg18' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:6 FAILED host:'sdw2' datadir:'/data/primary/gpseg4' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down
STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?
' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:7 FAILED host:'sdw2' datadir:'/data/primary/gpseg5' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down
STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?
' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:8 FAILED host:'sdw2' datadir:'/data/primary/gpseg6' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down
STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?
' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:9 FAILED host:'sdw2' datadir:'/data/primary/gpseg7' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down
STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?
' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:49 FAILED host:'sdw2' datadir:'/data/mirror/gpseg19' with reason:'cmd had rc=255 completed=True halted=False stdout='20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Starting gpsegstop.py with args: -D /data/primary/gpseg4:40000 -D /data/primary/gpseg5:40001 -D /data/primary/gpseg6:40002 -D /data/primary/gpseg7:40003 -D /data/mirror/gpseg19:50003 -m fast -t 600 -V postgres (Greenplum Database) 4.3.3.1 build 1 20150227:18:47:32:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Issuing shutdown commands to local segments... 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGSCONT to 37112 20150227:18:57:33:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGTERM to 37112 (smart shutdown) 20150227:18:57:36:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGINT to 37112 (fast shutdown) 20150227:18:57:39:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:-Sending SIGQUIT to 37112 (immediate shutdown) 20150227:18:57:40:028986 gpsegstop.py_sdw2:gpadmin:sdw2:gpadmin-[INFO]:- COMMAND RESULTS STATUS--DIR:/data/primary/gpseg4--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg4/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg5--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg5/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/primary/gpseg6--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: waiting for server to shut down........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... failed stderr: pg_ctl: server does not shut down
STATUS--DIR:/data/primary/gpseg7--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/primary/gpseg7/postmaster.pid" does not exist Is server running?
STATUS--DIR:/data/mirror/gpseg19--STOPPED:False--REASON:Shutdown failed: rc: 1 stdout: stderr: pg_ctl: PID file "/data/mirror/gpseg19/postmaster.pid" does not exist Is server running?
' stderr=''' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:45 FAILED host:'sdw1' datadir:'/data/mirror/gpseg15' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:51 FAILED host:'sdw1' datadir:'/data/mirror/gpseg21' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:54 FAILED host:'sdw1' datadir:'/data/mirror/gpseg24' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:41 FAILED host:'sdw7' datadir:'/data/mirror/gpseg11' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:44 FAILED host:'sdw7' datadir:'/data/mirror/gpseg14' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:50 FAILED host:'sdw7' datadir:'/data/mirror/gpseg20' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:37 FAILED host:'sdw6' datadir:'/data/mirror/gpseg7' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:40 FAILED host:'sdw6' datadir:'/data/mirror/gpseg10' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:43 FAILED host:'sdw6' datadir:'/data/mirror/gpseg13' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:30 FAILED host:'sdw2' datadir:'/data/mirror/gpseg0' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:52 FAILED host:'sdw2' datadir:'/data/mirror/gpseg22' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:55 FAILED host:'sdw2' datadir:'/data/mirror/gpseg25' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:32 FAILED host:'sdw4' datadir:'/data/mirror/gpseg2' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:35 FAILED host:'sdw4' datadir:'/data/mirror/gpseg5' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:38 FAILED host:'sdw4' datadir:'/data/mirror/gpseg8' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:57 FAILED host:'sdw4' datadir:'/data/mirror/gpseg27' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:31 FAILED host:'sdw3' datadir:'/data/mirror/gpseg1' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:34 FAILED host:'sdw3' datadir:'/data/mirror/gpseg4' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:53 FAILED host:'sdw3' datadir:'/data/mirror/gpseg23' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:56 FAILED host:'sdw3' datadir:'/data/mirror/gpseg26' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:33 FAILED host:'sdw5' datadir:'/data/mirror/gpseg3' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:36 FAILED host:'sdw5' datadir:'/data/mirror/gpseg6' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:39 FAILED host:'sdw5' datadir:'/data/mirror/gpseg9' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:42 FAILED host:'sdw5' datadir:'/data/mirror/gpseg12' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:18 FAILED host:'sdw5' datadir:'/data/primary/gpseg16' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:19 FAILED host:'sdw5' datadir:'/data/primary/gpseg17' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:20 FAILED host:'sdw5' datadir:'/data/primary/gpseg18' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-DBID:21 FAILED host:'sdw5' datadir:'/data/primary/gpseg19' with reason:'Shutdown failed' 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- Segments stopped successfully = 0 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segments with errors during stop = 45 <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segments that are currently marked down in configuration = 11 <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:- (stop was still attempted on these segments) 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 0 of 56 segment instances <<<<<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:------------------------------------------------- 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Segment instance shutdown failures reported 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Failed to shutdown 45 of 56 segment instances <<<<< 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-A total of 45 errors were encountered 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-Review logfile /home/gpadmin/gpAdminLogs/gpstop_20150227.log 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:-For more details on segment shutdown failure(s) 20150227:18:57:42:021878 gpstop:mdw:gpadmin-[WARNING]:------------------------------------------------- gpadmin@mdw:~> gpadmin@mdw:~> gpadmin@mdw:~> gpstart -a 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150' 20150227:18:58:42:022377 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20150227:18:58:43:022377 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20150227:18:58:43:022377 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Master Started... 20150227:18:58:44:022377 gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg0 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg3 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg6 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg9 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/mirror/gpseg12 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg16 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg17 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg18 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw5 directory /data/primary/gpseg19 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg22 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw2 directory /data/mirror/gpseg25 <<<<< 20150227:18:58:46:022377 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Process results... 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:9 FAILED host:'sdw2' datadir:'/data/primary/gpseg7' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:7 FAILED host:'sdw2' datadir:'/data/primary/gpseg5' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:6 FAILED host:'sdw2' datadir:'/data/primary/gpseg4' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:54 FAILED host:'sdw1' datadir:'/data/mirror/gpseg24' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:51 FAILED host:'sdw1' datadir:'/data/mirror/gpseg21' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:45 FAILED host:'sdw1' datadir:'/data/mirror/gpseg15' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:4 FAILED host:'sdw1' datadir:'/data/primary/gpseg2' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:3 FAILED host:'sdw1' datadir:'/data/primary/gpseg1' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:23 FAILED host:'sdw6' datadir:'/data/primary/gpseg21' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:26 FAILED host:'sdw7' datadir:'/data/primary/gpseg24' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-DBID:17 FAILED host:'sdw4' datadir:'/data/primary/gpseg15' with reason:'Failure in segment mirroring; check segment logfile' 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 34 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Failed segment starts, from mirroring connection between primary and mirror = 11 <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- Other failed segment starts = 0 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 11 <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Successfully started 34 of 45 segment instances, skipped 11 other segments <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Segment instance startup failures reported 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Failed start 11 of 45 segment instances <<<<<<<< 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Review /home/gpadmin/gpAdminLogs/gpstart_20150227.log 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-For more details on segment startup failure(s) 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-Run gpstate -s to review current segment instance status 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-There are 11 segment(s) marked down in the database 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20150227:19:08:50:022377 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20150227:19:08:51:022377 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20150227:19:09:05:022377 gpstart:mdw:gpadmin-[INFO]:-Starting standby master 20150227:19:09:05:022377 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /data/master/gpseg-1 20150227:19:09:08:022377 gpstart:mdw:gpadmin-[WARNING]:-Number of segments which failed to start: 11
12、连接数据库，删除不需要的清单表
gpadmin@mdw:~> psql -d template1 psql (8.2.15) Type "help" for help.
template1=# dn List of schemas Name | Owner --------------------+--------- gp_toolkit | gpadmin information_schema | gpadmin pg_aoseg | gpadmin pg_bitmapindex | gpadmin pg_catalog | gpadmin pg_toast | gpadmin public | gpadmin (7 rows)
template1=# l List of databases Name | Owner | Encoding | Access privileges -----------+---------+----------+--------------------- cems | gpadmin | UTF8 | dw | gpadmin | UTF8 | postgres | gpadmin | UTF8 | template0 | gpadmin | UTF8 | =c/gpadmin : gpadmin=CTc/gpadmin template1 | gpadmin | UTF8 | =c/gpadmin : gpadmin=CTc/gpadmin (5 rows)
13、gprecoverseg恢复
gprecoverseg Continue with segment recovery procedure Yy|Nn (default=N): > Y 20150227:22:43:00:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15 segment(s) to recover 20150227:22:43:00:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring 15 failed segment(s) are stopped 20150227:22:43:01:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38594: /data/mirror/gpseg0 20150227:22:43:03:035973 gprecoverseg:mdw:gpadmin-[INFO]:-41988: /data/mirror/gpseg1 20150227:22:43:04:035973 gprecoverseg:mdw:gpadmin-[INFO]:-1525: /data/mirror/gpseg2 20150227:22:43:05:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15788: /data/mirror/gpseg3 20150227:22:43:06:035973 gprecoverseg:mdw:gpadmin-[INFO]:-41987: /data/mirror/gpseg4 20150227:22:43:08:035973 gprecoverseg:mdw:gpadmin-[INFO]:-1524: /data/mirror/gpseg5 20150227:22:43:09:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15787: /data/mirror/gpseg6 20150227:22:43:10:035973 gprecoverseg:mdw:gpadmin-[INFO]:-30986: /data/mirror/gpseg7 20150227:22:43:11:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2730: /data/mirror/gpseg15 20150227:22:43:12:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15786: /data/primary/gpseg18 20150227:22:43:14:035973 gprecoverseg:mdw:gpadmin-[INFO]:-15785: /data/primary/gpseg19 20150227:22:43:15:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2729: /data/mirror/gpseg21 20150227:22:43:16:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38595: /data/mirror/gpseg22 20150227:22:43:17:035973 gprecoverseg:mdw:gpadmin-[INFO]:-2728: /data/mirror/gpseg24 20150227:22:43:19:035973 gprecoverseg:mdw:gpadmin-[INFO]:-38596: /data/mirror/gpseg25 20150227:22:43:23:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files from 15 segment(s) ........................................................................................................................................................................................................................................................................................................................................................................................................................................... 20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[INFO]:-Building template directory 20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1 Command was: 'ssh -o 'StrictHostKeyChecking no' sdw1 ". /usr/local/greenplum-db/./greenplum_path.sh; /usr/bin/scp -o 'StrictHostKeyChecking no' -r /data/primary/gpseg0/postgresql.conf mdw:/data/master/gpbuildingsegment_02272015_35973/schema"' rc=1, stdout='', stderr='ssh: Could not resolve hostname mdw: Name or service not known lost connection
报错，20150227:22:50:31:035973 gprecoverseg:mdw:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 1 Command was: 'ssh -o 'StrictHostKeyChecking no' sdw1 ". /usr/local/greenplum-db/./greenplum_path.sh; /usr/bin/scp -o 'StrictHostKeyChecking no' -r /data/primary/gpseg0/postgresql.conf mdw:/data/master/gpbuildingsegment_02272015_35973/schema"' rc=1, stdout='', stderr='ssh: Could not resolve hostname mdw: Name or service not known lost connection
经分析，应是hosts有问题了，直接ssh hosts的域名不行，直接ssh ip地址没有问题。
经检查，segment上的hosts有问题，和hadoop集群的hosts一样了。应该hadoop集群是后安装的，当时我要同事帮忙scp修改的hosts，估计把gp的集群也覆盖了。
修改hosts完后，
再执行gprecoverseg命令
gpadmin@mdw:~> gpstate -m 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -m 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.3.1 build 1' 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.3.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct 10 2014 14:31:50' 20150228:11:43:19:019954 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:--Current GPDB mirror list and status 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:--Type = Spread 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg0 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg1 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg2 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg3 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg4 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg5 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg6 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg7 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg8 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg9 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg10 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg11 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw5 /data/mirror/gpseg12 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg13 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg14 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg15 50003 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw6 /data/mirror/gpseg16 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg17 50001 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg18 50002 Acting as Primary Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg19 50003 Acting as Primary Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw7 /data/mirror/gpseg20 50000 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg21 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg22 50002 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg23 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data/mirror/gpseg24 50000 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data/mirror/gpseg25 50001 Passive Resynchronizing 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data/mirror/gpseg26 50002 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:- sdw4 /data/mirror/gpseg27 50003 Passive Synchronized 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20150228:11:43:20:019954 gpstate:mdw:gpadmin-[WARNING]:-2 segment(s) configured as mirror(s) are acting as primaries
恢复正常