Linux LVM commands result in
-
Couldn't find device with uuid
-
Couldn't find all physical volumes for volume group
Help! Commands like 'lvs', 'lvdisplay', 'vgdisplay', and 'pvscan' result in an error like the following:
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Volume group "metabackupvg" not found
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
PV /dev/emcpoweraa1 VG srcdata1vg lvm2 [50.00 GB / 0 free]
PV /dev/emcpowerz1 VG srcdata2vg lvm2 [50.00 GB / 0 free]
PV /dev/emcpowerj1 VG metalogvg lvm2 [3.00 GB / 0 free]
PV /dev/emcpowerl1 VG metalogvg lvm2 [3.00 GB / 0 free]
PV /dev/emcpowern1 VG metalogvg lvm2 [4.99 GB / 0 free]
PV /dev/emcpowerad1 VG metalogvg lvm2 [14.00 GB / 0 free]
PV /dev/emcpowery1 VG auditbackupvg lvm2 [229.99 GB / 0 free]
PV /dev/emcpoweru1 VG metabackupvg lvm2 [45.00 GB / 0 free]
PV unknown device VG metabackupvg lvm2 [69.99 GB / 0 free]
PV /dev/emcpoweraf1 VG metabackupvg lvm2 [124.99 GB / 0 free]
PV /dev/emcpowerv1 VG auditlogvg lvm2 [3.00 GB / 0 free]
PV /dev/emcpowerw1 VG auditlogvg lvm2 [3.00 GB / 0 free]
PV /dev/emcpowerx1 VG auditlogvg lvm2 [3.00 GB / 0 free]
PV /dev/emcpowert1 VG auditlogvg lvm2 [89.99 GB / 0 free]
PV /dev/emcpowerb1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerc1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerd1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowere1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerk1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerm1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowero1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerq1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerp1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerr1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerab1 VG auditdatavg lvm2 [25.00 GB / 0 free]
PV /dev/emcpowerf1 VG metadatavg lvm2 [11.00 GB / 0 free]
PV /dev/emcpowerg1 VG metadatavg lvm2 [11.00 GB / 0 free]
PV /dev/emcpowerh1 VG metadatavg lvm2 [11.00 GB / 0 free]
PV /dev/emcpoweri1 VG metadatavg lvm2 [11.00 GB / 0 free]
PV /dev/emcpowers1 VG metadatavg lvm2 [20.00 GB / 0 free]
PV /dev/emcpowera1 VG metadatavg lvm2 [20.00 GB / 0 free]
PV /dev/emcpowerae1 VG metadatavg lvm2 [14.00 GB / 0 free]
Total: 32 [1.04 TB] / in use: 32 [1.04 TB] / in no VG: 0 [0 ]
Solution
This scenario indicates that a disk is missing from the volume group. This is clear since pvscan scans all disks for physical volumes, and it's telling us that it cannot find a device. In my case, I deleted a LUN from the EMC that I knew was no longer being used, but I forgot to update the LVM configuration. To correct the issue, I removed the lost physical volume from the volume group using the --removemissing argument of the vgreduce command.
Tip: Run vgreduce with the --test and --verbose options to validate what will be executed and confirm this is what you want to do.
Finding volume group "metabackupvg"
Wiping cache of LVM-capable devices
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find all physical volumes for volume group metabackupvg.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Couldn't find device with uuid 'R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP'.
Archiving volume group "metabackupvg" metadata (seqno 73).
metabackupvg/backup2lv has missing extents: removing (including dependencies)
Removing LV backup2lv from VG metabackupvg
Removing PV with UUID R5zG2s-g4yi-ytEO-8Xvk-2B0s-S36G-9nWhXP from VG metabackupvg
Creating volume group backup "/etc/lvm/backup/metabackupvg" (seqno 74).
Wrote out consistent volume group metabackupvg
[root@dwetlprod2 ~]#
If a device needs to be added back, a solution for recovery is to cut another 70G LUN on the EMC, present it to the host and add it back into the volume group with the same UUID. Read my post on adding a new lun.
Physical volume "/dev/emcpowerah" successfully created
[root@dwetlprod2 ~]# vgcfgrestore metabackupvg
Restored volume group metabackupvg
[root@dwetlprod2 ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "srcdata1vg" using metadata type lvm2
Found volume group "srcdata2vg" using metadata type lvm2
Found volume group "metalogvg" using metadata type lvm2
Found volume group "auditbackupvg" using metadata type lvm2
Found volume group "auditlogvg" using metadata type lvm2
Found volume group "metabackupvg" using metadata type lvm2
Found volume group "auditdatavg" using metadata type lvm2
Found volume group "metadatavg" using metadata type lvm2
Run a file system check on the device then mount it. If your device (LUN) was mistakenly deleted, or in the case of physical media the disk has gone kaput, then you'll need to restore data from backups.
You may have noticed that I didn't even fdisk the replacement device. Don't forget to do this step if you will be restoring data and you're restoring to a partition.