How to rebuild RPM database on a Red Hat Enterprise Linux system?

本文是笔者最近遇到的一个故障的处理过程，解决方案是Rebuild RPM 的DB，后面内容其实是REDHAT官方的solutions,不过我遇到的现象和解决方案都与官方有点出入，故一直帖出来：

我遇到的故障现象：

[root@migrationLin6 ~]# rpm -qa
error: rpmdbNextIterator: skipping h#     514 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     259 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#       4 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD

没想到，按照官方步骤做了 rpm --rebuilddb以后，YUM又现现如下错误

[root@migrationLin6 rpm]# yum clean all
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 294, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 115, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 228, in getOptionsConfig
    self.conf
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 910, in <lambda>
    conf = property(fget=lambda self: self._getConfig(),
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 299, in _getConf                                                                                                                     ig
    startupconf = config.readStartupConfig(fn, root)
  File "/usr/lib/python2.6/site-packages/yum/config.py", line 892, in readStartu                                                                                                                     pConfig
    startupconf.releasever = _getsysver(startupconf.installroot, startupconf.dis                                                                                                                     troverpkg)
  File "/usr/lib/python2.6/site-packages/yum/config.py", line 1043, in _getsysve                                                                                                                     r
    hdr = idx.next()
StopIteration

中间走过各种坑，归根结底还是由于rpmdb损坏引起的。网上碰到这种现象的比较少，最后我的解决办法是：

rm -f /var/lib/rpm/*
rpm -vv -rebuilddb
yum update -y

附上官方的方案：　　

https://access.redhat.com/solutions/6903

如果没有redhat订阅，可能看不了，我粘上来了：

                             How to rebuild RPM database on a Red Hat Enterprise Linux system?

Solution 已验证 - 已更新2015年三月23日12:35

环境

• Red Hat Enterprise Linux (RHEL) - All Versions - All architectures

问题

• RPM is not working correctly, updates cannot be applied.
• How to rebuild the RPM database?
• Why does rpm command segfault when executed?
• yum update fails with rpmdb errors.
• Damaged header instance error
• rpm command terminates with "cannot open Packages database in /var/lib/rpm"
• Why does rpm or yum terminate with rpmdb: Lock table is out of available locker entries?
• Attempted to do an rpm -q on a couple of packages and got errors like this:

rpmdb: /var/lib/rpm/Packages: unexpected file type or format  
error: cannot open Packages index using db3 - Invalid argument (22)

• On doing rpm query or install, it is failing with following error:-

error: rpmdbNextIterator: skipping h#    1601 Header V4 RSA/SHA1 signature: BAD, key ID 2874ffbe

• Why "rpm -qa" command is not showing installed packages?
• rpm command hangs/hung
• rpm database got corrupted

决议

• Depending on the particular scenario, there are several different paths that correct this situation

Important: Before attempting to rebuild the rpm database, first refer to the following known issue for Red Hat Enterprise Linux 6 systems:

Update to nss-softokn 3.14.3-19 breaks RPM

1) Stale lock file cleanup - files needed for updating are being held open.

Note: There are two methods (1A & 1B) to correct this:

1A) Reboot system (preferred)

1. The best course of action is to simply reboot the machine, as the system cleans up stale locks during the sysinit portion of boot. The cleanup occurs at this stage as the system is assured there is no process that would have a lock on the RPM database.

   # grep rpm /etc/rc.d/rc.sysinit
   rm -f /var/lib/rpm/__db* &> /dev/null
   

1B) Reboot not feasible (more risky than reboot)

1. Delete all files in /var/spool/up2date

   # cd /var/spool/up2date
   # rm *
   # rm .*
   

2. Ensure there are no processes with the RPM database files open. One way to do this is using lsof & ps as the root user:

   # ps -aux | grep -e rpm -e yum -e up2date
   # lsof | grep /var/lib/rpm
   

3. If, and only if, there is no RPM database activity going on it is safe to delete the lock files.
   WARNING: If the system is in multi-user mode it is possible that a process may open the RPM database at any time. Therefore, the recommended method for clearing stale locks is the reboot which cleans up the locks prior to entering multi-user mode.

   # rm -f /var/lib/rpm/__db*
   

---

---

2) DB corruption recovery process - The RPM database has become corrupt and recovery is desired.

1. As always it best to start by taking a backup before attempting to repair.

   # cd /var/lib
   # tar -zcvf /var/preserve/rpmdb-$(date +%Y-%m-%d_%H-%M-%S).tar.gz rpm
   

   Note: This tar backup can be used if the attempt to recover the RPM database has issues.

2. Verify integrity of the Packages file:

   # cd /var/lib/rpm
   # rm -f __db*      # to avoid stale locks
   # /usr/lib/rpm/rpmdb_verify Packages
   

   If the last command shown in method 2 returns without any error, then we can be sure about the integrity of the rpm database. You can also issue the following command to be sure of the success of the rpmdb_verify command.

   $ /usr/lib/rpm/rpmdb_verify Packages
   $ echo $?
   0
   

   A return value of 0 from echo $? means that the last command has been executed successfully.

3. If the above verify reports success, go to step 6.

4. If the above verify reports any errors, a dump and load of the database is required.

   # mv Packages Packages.orig
   # /usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages
   

5. Verify the newly loaded Packages file, as well as rpm output.

   # /usr/lib/rpm/rpmdb_verify Packages
   # rpm -qa >/dev/null    # Validating there are no errors
   # rpm -qa               # Validating list of installed RPMs
   

6. If no errors and everything looks good, rebuild the RPM indexes

   # rpm -vv --rebuilddb
   

7. Once again verify RPM database:

   # cd /var/lib/rpm
   # /usr/lib/rpm/rpmdb_verify Packages
   

Note: If the above recovery steps fail or throw errors, restoring /var/lib/rpm directory from a recent system backup may be your best option. Remember to not overwrite the tar backup as it may still be valuable.

Note: If you have another machine having the exactly same installed package set, you can rebuild the rpm db with Packages file on the machine.

Note: If you have a rpm database in other directory except for /var/lib/rpm and it's corrupted, you will need to do the same things to the rpm database, not one in /var/lib/rpm. The rpm command has --root option to specify the root directory for the rpm database. Especially, please take care of the rpm -vv --rebuilddb command to rebuild the rpm database. For example, if you are using /opt/var/lib/rpm for a directory including rpm database, run this command rpm -vv --rebuilddb --root /opt.

---

3.) Other resources:

• To restore rpm database file from /var/log/rpmpkgs, refer to: How to recover rpm database file /var/lib/rpm/Packages using /var/log/rpmpkgs ?
• Comprehensive notes on RPM database recovery can be found at: http://people.redhat.com/berrange/notes/rpmrecovery.html.

根源

• rpm database corruption

诊断步骤

• yum update fails with the following:

[root@localhost ~]# yum update
rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(-30974) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@localhost ~]# yum clean all
rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(-30974) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

• rpm shows damaged header instances:

$ rpm -Va
error: rpmdb: damaged header instance #<header number> retrived, skipping
error: rpmdb: damaged header instance #<header number> retrived, skipping
(above message repeated ad infinitum)

• rpm commands terminate with the error below:

rpmdb: unable to join the environment  
error: db3 error(11) from dbenv->open: Resource temporarily unavailble  
error: cannot open Packages index using db3 - Resource temporarily unavailable (11)  
error: cannot open Packages database in /var/lib/rpm

• rpm and yum commands return the following:

$ rpm -qa  
...  
rpmdb: Lock table is out of available locker entries  
error: db4 error(22) from db->close: Invalid argument