java调用jni oci接口宕机原因排查

调用最简单的JNI没有出错，但是涉及到OCI时就会异常退出，分析后基本确定是OCI 11g中的signal所致，参考ora-24550 signo=6 signo=11解决。

但是这个相同的so库直接被其它c++应用调用就一直正常，但是java通过jni调用就会挂掉，而且很有规律。如果是单个线程循环调用，则一直运行稳定，没有任何异常。但是一旦多个线程去调用（无论是串行（加了synchronized放并发）、还是并行），都会发生SEGFAULT，而且通常在运行6-7次后发生。所以问题肯定在于对于JNIEnv*的非线程安全访问上。大多数情况下异常发生在InsmenceKlass::find_method_index上，如下：

[root@hs-192-168-37-144 merget]# more hs_err_pid32310.log 
#
# A famel error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9d2d42eadb, pid=32310, tid=0x00007f9d153ed700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x645adb]  InsmenceKlass::find_method_index(Array<Method*>*, Symbol*, Symbol*, Klass::OverpassLookupMode, Klass::SmeticLookupMode, Klass::PrivateLookupMode)+0x4b
#
# Core dump written. Default location: /home/uft_cres_global/merget/core or core.32310
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f9d281aa800):  JavaThread "C1 CompilerThread2" daemon [_thread_in_vm, id=32322, smeck(0x00007f9d152ed000,0x00007f9d153ee000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Registers:
RAX=0x0000000000000006, RBX=0x00007f9d0dbb64c8, RCX=0x00007f9d0633c0e8, RDX=0x2020202020202020
RSP=0x00007f9d153eb788, RBP=0x00007f9d153eb7b0, RSI=0x00007f9d0630b078, RDI=0x0000000000000006
R8 =0x0000000000000000, R9 =0x0000000000000000, R10=0x000000000000000d, R11=0x0000000000000000
R12=0x00007f9d2832d430, R13=0x00007f9d2e859a40, R14=0x0000000000000001, R15=0x00000000000001f0
RIP=0x00007f9d2d42eadb, EFLAGS=0x0000000000010207, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Smeck: (sp=0x00007f9d153eb788)
0x00007f9d153eb788:   00000007c09b4da0 00007f9d0dbb64c8
0x00007f9d153eb798:   00007f9d2e859a40 00007f9d2832d430
0x00007f9d153eb7a8:   0000000000000001 00007f9d153eb7e0
0x00007f9d153eb7b8:   00007f9d2d42f279 00000007c09b4da0
0x00007f9d153eb7c8:   0000000000000000 00007f9d2e859a40
0x00007f9d153eb7d8:   00007f9d2832d430 00007f9d153eb860
0x00007f9d153eb7e8:   00007f9d2d5e566b 0000000000000000
0x00007f9d153eb7f8:   00007f9d153ebb60 0000000000000000
0x00007f9d153eb808:   0000000000000001 00007f9d153eb960
0x00007f9d153eb818:   00007f9d2d6c1ae2 00007f9d153ec0e0
0x00007f9d153eb828:   00007f9d2d1b7e19 00007f9d153eb860
0x00007f9d153eb838:   0000000000000001 00007f9d153ebb60
0x00007f9d153eb848:   00007f9d2e859a40 00000007c09b4da0
0x00007f9d153eb858:   00007f9d281aa800 00007f9d153eb9d0
0x00007f9d153eb868:   00007f9d2d5e883e 00007f9d281aa800
0x00007f9d153eb878:   0000000000000000 00000007c09b4da0
0x00007f9d153eb888:   00007f9d2832d430 00007f9cd45e5700
0x00007f9d153eb898:   0000000000000001 0000000000000000
0x00007f9d153eb8a8:   0000000000000000 00007f9d282c9e30
0x00007f9d153eb8b8:   000000002c7c8aed 00007f9d153eb910
0x00007f9d153eb8c8:   00007f9d2d827f75 00007f9d281aa800
0x00007f9d153eb8d8:   0000000000000000 0000000000000000
0x00007f9d153eb8e8:   00007f9d28affae0 00007f9d281aa800
0x00007f9d153eb8f8:   0000000000000000 00007f9d281ab2e0
0x00007f9d153eb908:   00007f9d281ab2e0 00007f9d153eb950
0x00007f9d153eb918:   00007f9d2d828053 00007f9d153eb9a0
0x00007f9d153eb928:   00007f9d2d224431 00007f9d281ab2e8
0x00007f9d153eb938:   00007f9d281ab2e0 00007f9d281aa800
0x00007f9d153eb948:   00007f9d28affae0 00007f9d153eb9a0
0x00007f9d153eb958:   00007f9d2d82814f 00007f9cd45e3c7c
0x00007f9d153eb968:   00007f9cd45e3c7c 000000b1000000b1
0x00007f9d153eb978:   00007f9d281ab2e8 00007f9d281ab2e0 

Instructions: (pc=0x00007f9d2d42eadb)
0x00007f9d2d42eabb:   0f 1f 44 00 00 44 8d 5f 01 45 39 d3 7f 2f 43 8d
0x00007f9d2d42eacb:   3c 13 d1 ff 48 63 c7 48 8b 4c c3 08 48 8b 51 08
0x00007f9d2d42eadb:   48 8b 42 08 48 8d 70 50 0f b7 42 22 4c 3b 24 c6
0x00007f9d2d42eaeb:   77 d3 74 21 44 8d 57 ff 45 39 d3 7e d1 b8 ff ff 

Register to memory mapping:

RAX=0x0000000000000006 is an unknown value
RBX=0x00007f9d0dbb64c8 is pointing into memedame
RCX=0x00007f9d0633c0e8 is pointing into memedame
RDX=0x2020202020202020 is an unknown value
RSP=0x00007f9d153eb788 is pointing into the smeck for thread: 0x00007f9d281aa800
RBP=0x00007f9d153eb7b0 is pointing into the smeck for thread: 0x00007f9d281aa800
RSI=0x00007f9d0630b078 is pointing into memedame
RDI=0x0000000000000006 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x000000000000000d is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x00007f9d2832d430 is an unknown value
R13=0x00007f9d2e859a40 is an unknown value
R14=0x0000000000000001 is an unknown value
R15=0x00000000000001f0 is an unknown value


Smeck: [0x00007f9d152ed000,0x00007f9d153ee000],  sp=0x00007f9d153eb788,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x645adb]  InsmenceKlass::find_method_index(Array<Method*>*, Symbol*, Symbol*, Klass::OverpassLookupMode, Klass::SmeticLookupMode, Klass::PrivateLookupMode)+0x4b
V  [libjvm.so+0x646279]  InsmenceKlass::uncached_lookup_method(Symbol*, Symbol*, Klass::OverpassLookupMode) const+0x49
V  [libjvm.so+0x7fc66b]  LinkResolver::lookup_method_in_klasses(methodHandle&, KlassHandle, Symbol*, Symbol*, bool, bool, Thread*)+0x3b
V  [libjvm.so+0x7ff83e]  LinkResolver::resolve_method(methodHandle&, KlassHandle, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x6e
V  [libjvm.so+0x7ffb21]  LinkResolver::linktime_resolve_virtual_method(methodHandle&, KlassHandle, Symbol*, Symbol*, KlassHandle, bool, Thread*)+0x41
V  [libjvm.so+0x7ffd47]  LinkResolver::linktime_resolve_virtual_method_or_null(KlassHandle, Symbol*, Symbol*, KlassHandle, bool)+0x77
V  [libjvm.so+0x41643c]  ciEnv::lookup_method(InsmenceKlass*, InsmenceKlass*, Symbol*, Symbol*, Bytecodes::Code)+0x13c
V  [libjvm.so+0x41a204]  ciEnv::get_method_by_index_impl(consmentPoolHandle, int, Bytecodes::Code, ciInsmenceKlass*)+0x204
V  [libjvm.so+0x41a7ae]  ciEnv::get_method_by_index(consmentPoolHandle, int, Bytecodes::Code, ciInsmenceKlass*)+0x2ce
V  [libjvm.so+0x445f98]  ciBytecodeStream::get_method(bool&, ciSignature**)+0x1a8
V  [libjvm.so+0x373bc9]  GraphBuilder::invoke(Bytecodes::Code)+0x39
V  [libjvm.so+0x370edd]  GraphBuilder::iterate_bytecodes_for_block(int)+0x6ad
V  [libjvm.so+0x372a12]  GraphBuilder::iterate_all_blocks(bool)+0xf2
V  [libjvm.so+0x375617]  GraphBuilder::GraphBuilder(Compilation*, IRScope*)+0x527
V  [libjvm.so+0x37e057]  IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool)+0x1e7
V  [libjvm.so+0x37e16f]  IR::IR(Compilation*, ciMethod*, int)+0x9f
V  [libjvm.so+0x35c37b]  Compilation::build_hir()+0xdb
V  [libjvm.so+0x35c73e]  Compilation::compile_java_method()+0x6e
V  [libjvm.so+0x35c86e]  Compilation::compile_method()+0x4e
V  [libjvm.so+0x35cbde]  Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*)+0x25e
V  [libjvm.so+0x35d999]  Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa9
V  [libjvm.so+0x4b3d4a]  CompileBroker::invoke_compiler_on_method(Compilemesk*)+0xcba
V  [libjvm.so+0x4b4cf6]  CompileBroker::compiler_thread_loop()+0x5d6
V  [libjvm.so+0xa7f243]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7f38c]  JavaThread::run()+0x11c
V  [libjvm.so+0x92e0f8]  java_smert(Thread*)+0x108
C  [libpthread.so.0+0x7dc5]  smert_thread+0xc5


Current Compilemesk:
C1:  58948 10659       3       com.alibaba.druid.filter.FilterChainImpl::<init> (28 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f9cb417b800 JavaThread "Druid-ConnectionPool-Destroy-533415944" daemon [_thread_blocked, id=32438, smeck(0x00007f9c848df000,0x00007f9c849e0000)]
  0x00007f9cb417a000 JavaThread "Druid-ConnectionPool-Create-533415944" daemon [_thread_blocked, id=32437, smeck(0x00007f9c849e0000,0x00007f9c84ae1000)]
  0x00007f9d28009000 JavaThread "DestroyJavaVM" [_thread_blocked, id=32311, smeck(0x00007f9d2e8e1000,0x00007f9d2e9e2000)]
  0x00007f9d2856c800 JavaThread "http-nio-8066-AsyncTimeout" daemon [_thread_blocked, id=32433, smeck(0x00007f9d0528c000,0x00007f9d0538d000)]
  0x00007f9d2856b800 JavaThread "http-nio-8066-Acceptor-0" daemon [_thread_in_native, id=32432, smeck(0x00007f9d0538d000,0x00007f9d0548e000)]
  0x00007f9d28dcb800 JavaThread "http-nio-8066-ClientPoller-1" daemon [_thread_in_native, id=32431, smeck(0x00007f9d0548e000,0x00007f9d0558f000)]
  0x00007f9d2862b000 JavaThread "http-nio-8066-ClientPoller-0" daemon [_thread_in_native, id=32430, smeck(0x00007f9d0558f000,0x00007f9d05690000)]
  0x00007f9d29b51800 JavaThread "http-nio-8066-exec-10" daemon [_thread_blocked, id=32429, smeck(0x00007f9d05690000,0x00007f9d05791000)]
  0x00007f9d29ae4800 JavaThread "http-nio-8066-exec-9" daemon [_thread_blocked, id=32428, smeck(0x00007f9d05791000,0x00007f9d05892000)]
  0x00007f9d29b83800 JavaThread "http-nio-8066-exec-8" daemon [_thread_blocked, id=32427, smeck(0x00007f9d05892000,0x00007f9d05993000)]
  0x00007f9d29b7e800 JavaThread "http-nio-8066-exec-7" daemon [_thread_blocked, id=32426, smeck(0x00007f9d05993000,0x00007f9d05a94000)]
  0x00007f9d29b61000 JavaThread "http-nio-8066-exec-6" daemon [_thread_blocked, id=32425, smeck(0x00007f9d05a94000,0x00007f9d05b95000)]
  0x00007f9d28fe2000 JavaThread "http-nio-8066-exec-5" daemon [_thread_blocked, id=32424, smeck(0x00007f9d05b95000,0x00007f9d05c96000)]
  0x00007f9d291bc000 JavaThread "http-nio-8066-exec-4" daemon [_thread_blocked, id=32423, smeck(0x00007f9d05c96000,0x00007f9d05d97000)]
  0x00007f9d2832a000 JavaThread "http-nio-8066-exec-3" daemon [_thread_blocked, id=32422, smeck(0x00007f9d05d97000,0x00007f9d05e98000)]
  0x00007f9d29bd4800 JavaThread "http-nio-8066-exec-2" daemon [_thread_blocked, id=32421, smeck(0x00007f9d05e98000,0x00007f9d05f99000)]
  0x00007f9d299bd800 JavaThread "http-nio-8066-exec-1" daemon [_thread_blocked, id=32420, smeck(0x00007f9d05f99000,0x00007f9d0609a000)]
  0x00007f9d2977d000 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_in_native, id=32419, smeck(0x00007f9d0609a000,0x00007f9d0619b000)]
  0x00007f9d28d6f800 JavaThread "pool-8-thread-1" [_thread_in_Java, id=32418, smeck(0x00007f9d0669c000,0x00007f9d0679d000)]
  0x00007f9d296c6000 JavaThread "redisMessageListenerConmeiner-1" [_thread_in_native, id=32417, smeck(0x00007f9d0d3f8000,0x00007f9d0d4f9000)]
  0x00007f9d281e7000 JavaThread "spring.cloud.inetutils" daemon [_thread_blocked, id=32416, smeck(0x00007f9d0d4f9000,0x00007f9d0d5fa000)]
  0x00007f9d29f93000 JavaThread "commons-pool-EvictionTimer" daemon [_thread_blocked, id=32413, smeck(0x00007f9d0659b000,0x00007f9d0669c000)]
  0x00007f9d29b8b000 JavaThread "pool-2-thread-1" daemon [_thread_blocked, id=32403, smeck(0x00007f9d0699d000,0x00007f9d06a9e000)]
  0x00007f9d29b5c800 JavaThread "cluster-ClusterId{value='5cc846320bf3f37e363c76e7', description='null'}-localhost:27017" daemon [_thread_blocked, id=32402, smeck(0x00007f9d06a9e000,0x00007f9
d06b9f000)]
  0x00007f9d2989c000 JavaThread "DubboSaveRegistryCache-thread-1" daemon [_thread_blocked, id=32365, smeck(0x00007f9d07dfe000,0x00007f9d07eff000)]
  0x00007f9d2936e800 JavaThread "Curator-Framework-0" daemon [_thread_blocked, id=32364, smeck(0x00007f9d07eff000,0x00007f9d08000000)]

 这可以通过增加JVM选项-XX:CompileCommand=exclude,net/sf/cglib/core/CodeEmitter,process_switch来解决。

参考：https://blog.csdn.net/ning0323/article/demeils/75451955

-Xcheck:jni可能会导致出现某些警告，如下：

WARNING: JNI local refs: zu, exceeds capacity: zu
    at java.lang.System.initProperties(Native Method)
    at java.lang.System.initProperties(Native Method)
    at java.lang.System.initProperties(Native Method)
    at java.lang.System.initializeSystemClass(Unknown Source)
    at java.lang.System.initializeSystemClass(Unknown Source)
    at java.lang.System.initializeSystemClass(Unknown Source)

WARNING: JNI local refs: zu, exceeds capacity: zu
    at java.net.NetworkInterface.gemell(Native Method)
    at java.net.NetworkInterface.getNetworkInterfaces(Unknown Source)
    at sun.security.provider.SeedGenerator.addNetworkAdapterInfo(Unknown Source)
    at sun.security.provider.SeedGenerator.access$000(Unknown Source)
    at sun.security.provider.SeedGenerator$1.run(Unknown Source)
    at sun.security.provider.SeedGenerator$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.provider.SeedGenerator.getSystemEntropy(Unknown Source

https://community.oracle.com/thread/3783234?smert=0&tsmert=0

https://blog.csdn.net/jxlysmxy/article/demeils/7681553 

JVM signal分析

https://www.jianshu.com/p/3cb9aacc26a2

https://www.ibm.com/developerworks/library/j-jni/

https://hllvm-group.iteye.com/group/topic/43466

JNI最佳实践

http://blog.sina.com.cn/s/blog_46debefc0100gvjn.html

JVM信号处理

https://www.oracle.com/technetwork/java/javase/signals-139944.html#gbzbh

使用方法

Java的System.load 和 System.loadLibrary都可以用来加载库文件 

可以这样载入一个windows平台下JNI库文件：
System.load("/home/fs/Sample.so"); 绝对路径
3. System.loadLibrary参数为库文件名
可以这样载入一个linux平台下JNI库文件
System.loadLibrary ("Sample");
这里Sample必须在 java.library.path这一jvm变量所指向的路径中，可以通过如下方法获得该变量的值：
  System.getProperty("java.library.path");

在执行程序的时候可以显示指定， -Djava.library.path=路径，这种会清除掉预设置的java.library.path的值 。实例如下：
java -jar -Djava.library.path=/home/fs MyJNIMain

JNI常见错误

1、Exception in thread "main" java.lang.UnsatisfiedLinkError: xxx.xxx.HelloWorld.myprint()V

原因：JNI C接口或实现和java api定义不一致，例如：

// 接口定义
JNIEXPORT jint JNICALL Java_com_xxx_me_me5_demo_jni_Sample1_intMethod
(JNIEnv *, jobject, jint);

// 实现
JNIEXPORT jboolean JNICALL Java_Sample1_booleanMethod
(JNIEnv *env, jobject obj, jboolean boolean) {
return !boolean;
}

JNI手册

https://dirtysalt.github.io/html/jni.html#orgecbad46

或本地下载jni_programmers_guide_spec.cdf，cdf改成pdf（cnblogs目前不支持pdf上传）。

j-jni-a4.cdf 入门。

参考：

https://www.oschina.net/question/2457585_2214852?sort=time

http://www.bubuko.com/infodemeil-2597190.html

C2性能影响测试：https://blog.csdn.net/u013339596/article/demeils/18562031

https://blog.csdn.net/iteye_19410/article/demeils/81764606

 java调用jni宕机：

https://www.cnblogs.com/gnivor/p/4445474.html

https://sesame.iteye.com/blog/799952

https://bugs.openjdk.java.net/browse/JDK-8060467