Hit an strange error just now.
[oracle@racnode1 ~]$ srvctl # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGBUS (0x7) at pc=0x42604f0a, pid=20424, tid=3077707456 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_30-b03 mixed mode, sharing) # Problematic frame: # /u01/app/11.2.0/grid/bin/srvctl: line 240: 20424 Segmentation fault (core dumped) $JRE $JRE_OPTIONS -DORACLE_HOME=/u01/app/11.2.0/grid -classpath $CLASSPATH $SRVM_PROPERTY_DEFS oracle.ops.opsctl.OPSCTLDriver "$@" [oracle@racnode1 ~]$ srvctl # # An unexpected error has been detected by HotSpot Virtual Machine: # # SIGBUS (0x7) at pc=0x42604f0a, pid=20515, tid=3077949120 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_30-b03 mixed mode, sharing) # Problematic frame: # /u01/app/11.2.0/grid/bin/srvctl: line 240: 20515 Segmentation fault (core dumped) $JRE $JRE_OPTIONS -DORACLE_HOME=/u01/app/11.2.0/grid -classpath $CLASSPATH $SRVM_PROPERTY_DEFS oracle.ops.opsctl.OPSCTLDriver "$@"
The srvctl hit error.. The error looks like java issue. So i check the java
[oracle@racnode1 ~]$ java -version OpenJDK Client VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_oracle/20523 Try using the -Djava.io.tmpdir= option to select an alternate temp location. java version "1.7.0_09-icedtea" OpenJDK Runtime Environment (rhel-2.3.4.1.0.1.el6_3-i386) OpenJDK Client VM (build 23.2-b09, mixed mode)
There is a line like this
OpenJDK Client VM warning: Insufficient space for shared memory file:
Seems some disk is full. So i check disk usage with df -h as below. Yes the root directory is full. after housekeep problem fixed.
[oracle@racnode1 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_racnode1-lv_root 16G 15G 0 100% / tmpfs 1012M 150M 862M 15% /dev/shm