[原创]记一次java执行段错误及解决过程

最近一周左右，网管监控系统经常监控到tomcat异常退出，由于有检测tomcat pid的脚本，所以会自动重启服务器，查询tomcat日志没有报错信息，查询系统message日志，可以看到如下信息。

Dec 15 22:08:07 net-wap3 kernel: java[8299]: segfault at 00000000453caa98 rip 00002aaac4d055bd rsp 00000000453caa80 error 6

然后用nohup strace -p pid &> strace.log &将程序与内核之间传递的信号记录下来，记录下如下信息：

Process 9754 attached - interrupt to quit
futex(0x40b589d0, FUTEX_WAIT, 9762, NULL <unfinished ...>
+++ killed by SIGSEGV +++

可以看到是被11号信号kill的，然后我们需要得到程序退出时的corefile。

编辑/etc/profile将其中的ulimit -S -c 0 > /dev/null 2>&1改成ulimit -S -c unlimited > /dev/null 2>&1，然后source /etc/profile，用ulimit -c查看查看，将程序重启。

然后查看一下重启后的程序的pid，然后cat /proc/pid/limits，其中

Limit                      Soft Limit       Hard Limit         Units 

Max core file size     unlimited         unlimited          bytes

确定红色地方为unlimited，确保可以输出corefile，然后运行echo "/corefile/core_%p_%s_%t_%e" > /proc/sys/kernel/core_pattern，将corefile输出到/corefile下，%p代表pid，%s代表引起程序退出的信号，%t代表自1970年1月1日0时到程序退出时的秒数，%e代表可执行文件名，如我得到的文件core_29151_11_1387033918_java。

然后用gdb java core_29151_11_1387033918_java和jstack java core_29151_11_1387033918_java分析。

在jstack输出的结果中发现如下信息

- org.apache.jsp._480.mfb.zt.qc._30999750.index_jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=1680, line=296 (Interpreted frame)

- org.apache.jasper.runtime.HttpJspBase.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=70 (Interpreted frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

- org.apache.jasper.servlet.JspServletWrapper.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=320, line=377 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.serviceJspFile(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.String, java.lang.Throwable, boolean) @bci=126, line=313 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=347, line=260 (Compiled frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=376, line=290 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)

- com.wondertek.mobiletv.portal.webapp.filter.XSSServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=17, line=59 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse, org.apache.catalina.core.ApplicationDispatcher$State) @bci=393, line=646 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.doInclude(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=260, line=551 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=488 (Compiled frame)

- sun.reflect.GeneratedMethodAccessor307.invoke(java.lang.Object, java.lang.Object[]) @bci=48 (Compiled frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Compiled frame)

- net.bull.javamelody.JspWrapper.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=85, line=151 (Compiled frame)

- net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=7, line=282 (Compiled frame)

- $Proxy47.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=20 (Compiled frame)

- org.apache.jasper.runtime.JspRuntimeLibrary.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse, java.lang.String, javax.servlet.jsp.JspWriter, boolean) @bci=45, line=968 (Interpreted frame)

- org.apache.jsp._480.mfb.zt.qc._30999750.index_jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=1695, line=299 (Interpreted frame)

- org.apache.jasper.runtime.HttpJspBase.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=70 (Interpreted frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

- org.apache.jasper.servlet.JspServletWrapper.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=320, line=377 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.serviceJspFile(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.String, java.lang.Throwable, boolean) @bci=126, line=313 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=347, line=260 (Compiled frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=376, line=290 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)

- com.wondertek.mobiletv.portal.webapp.filter.XSSServletFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=17, line=59 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame)

- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse, org.apache.catalina.core.ApplicationDispatcher$State) @bci=393, line=646 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.doInclude(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=260, line=551 (Compiled frame)

- org.apache.catalina.core.ApplicationDispatcher.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=488 (Compiled frame)

- sun.reflect.GeneratedMethodAccessor307.invoke(java.lang.Object, java.lang.Object[]) @bci=48 (Compiled frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Compiled frame)

- net.bull.javamelody.JspWrapper.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=85, line=151 (Compiled frame)

- net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=7, line=282 (Compiled frame)

- $Proxy47.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=20 (Compiled frame)

- org.apache.jasper.runtime.JspRuntimeLibrary.include(javax.servlet.ServletRequest, javax.servlet.ServletResponse, java.lang.String, javax.servlet.jsp.JspWriter, boolean) @bci=45, line=968 (Interpreted frame)

- org.apache.jsp._480.mfb.zt.qc._30999750.index_jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=1695, line=299 (Interpreted frame)

- org.apache.jasper.runtime.HttpJspBase.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=70 (Interpreted frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

- org.apache.jasper.servlet.JspServletWrapper.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=320, line=377 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.serviceJspFile(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.String, java.lang.Throwable, boolean) @bci=126, line=313 (Compiled frame)

- org.apache.jasper.servlet.JspServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=347, line=260 (Compiled frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=717 (Compiled frame)

 

发现程序异常是由于上面的页面中存在循环导致，至此故障恢复。