归档日志满处理（OS Audit file could not be created; failing after 6 retries）

早上同事说测试环境的数据库连不上了。检查alert日志，发现报错如下：

************************************************************************
ARC3: Error 19809 Creating archive log file to '/u01/app/oracle/fast_recovery_area/SGDSDB/archivelog/2017_05_31/o1_mf_1_4695_%u_.arc'
ARC1: Error 19809 Creating archive log file to '/u01/app/oracle/fast_recovery_area/SGDSDB/archivelog/2017_05_31/o1_mf_1_4694_%u_.arc'
Errors in file /u01/app/oracle/diag/rdbms/sgdsdb/sgdsdb/trace/sgdsdb_arc0_5237.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 104857600000 bytes is 100.00% used, and has 0 remaining bytes available.
Errors in file /u01/app/oracle/diag/rdbms/sgdsdb/sgdsdb/trace/sgdsdb_arc1_5239.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 104857600000 bytes is 100.00% used, and has 0 remaining bytes available.
************************************************************************
************************************************************************
You have following choices to free up space from recovery area:
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
2. Back up files to tertiary device such as tape using RMAN
   BACKUP RECOVERY AREA command.
   BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
3. Add disk space and increase db_recovery_file_dest_size parameter to
   reflect the new space.
   reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
4. Delete unnecessary files using RMAN DELETE command. If an operating
   system command was used to delete files, then use RMAN CROSSCHECK and
   system command was used to delete files, then use RMAN CROSSCHECK and   DELETE EXPIRED commands.

   DELETE EXPIRED commands.
************************************************************************

本以为是很简单的归档日志满造成的报错，按照以往的操作过程，先把db_recovery_file_dest对应的很久之前的归档删除，再等数据库自己把没有归档的日志进行恢复。

结果删除完归档后，发现仍然报错。sqlplus和rman也无法进入。

因为是测试环境，所以就直接将pmon进程干掉，然后将数据库启动到nomount状态，结果发现仍然报错：

OS Audit file could not be created; failing after 6 retries
OS Audit file could not be created; failing after 6 retries
Wed May 31 10:57:32 2017
Performing implicit shutdown abort due to dead PMON
Shutting down instance (abort)

这个应该是相应的audit的目录没有健全，或者oracle的执行文件的权限不对。

首先检查权限：

[oracle@dsdev trace]$ ll $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 oracle oinstall 239626731 Jul  8  2015 /u01/app/oracle/product/11.2/db_1/bin/oracle
[oracle@dsdev trace]$ 

权限是对的，接着检查对应的目录

[oracle@dsdev dbs]$ ls -l /u01/app/oracle/admin/sgdsdb/adump
ls: cannot access /u01/app/oracle/admin/sgdsdb/adump: No such file or directory
[oracle@dsdev dbs]$ 

发现audit_file_dest的目录不存在，创建好之后，数据库恢复正常。

随后在rman中crosscheck archivelog all和delete expired copy 删除过期和无效的归档日志。

另外，在随后的监听启动中发现

启动监听，报错

[oracle@dsdev ~]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 31-MAY-2017 14:02:02

Copyright (c) 1991, 2009, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dsdev)(PORT=1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused

直接用lsnrctl status仍然报错，去查看相应的log无具体提示。

经检查发现是hosts里面的127.0.0.1对应的localhost被修改掉，重新添加127.0.0.1 localhost后正常(这个和安装时没有设置localhost有异曲同工之妙)。