WinDbg: 执行 SOS 扩展命令 !clrstack时报错 Access violation exception (0xC0000005)

今天在调试分析一个dump文件时，当我执行了".loadby sos clrjit"指令后，准备分析托管代码的问题，于是我又输入了"!clrstack"扩展指令想看下托管栈，结果Windbg给我如下输出：

0:000> .loadby sos clrjit
0:000> !clrstack
c0000005 Exception in C:WINDOWSMicrosoft.NETFrameworkv4.0.30319sos.clrstack debugger extension.
      PC: 0c53d0f3  VA: 00000000  R/W: 0  Parameter: 00000000
这还是第一次遇到，当我在执行"!pe"指令时，又有正确输出了

0:000> !pe
Exception object: 1eda1120
Exception type:   System.ExecutionEngineException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
然后在执行"!clrstack"，结果也是能正确输出：

0:000> !clrstack
OS Thread Id: 0x6f90 (0)
Child SP       IP Call Site
00efc708 50e51a49 [PrestubMethodFrame: 00efc708] XXXXXX..cctor()
00efc8ec 50e51a49 [GCFrame: 00efc8ec]
00efce54 50e51a49 [PrestubMethodFrame: 00efce54] XXXXXX..ctor()
00efcec4 1d8908f4 XXXXXX(Int64)
00efd208 50e1f066 [DebuggerU2MCatchHandlerFrame: 00efd208]
00efcfc8 50e1f066 [HelperMethodFrame_PROTECTOBJ: 00efcfc8] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
00efd2a4 78dfc799 System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
00efd2c8 78dfc2aa System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
00efd2fc 1d7b2cf0 *** WARNING: Unable to verify checksum for Adapter.dll
.?A0xc8a1231f.CallFunctionByReflectionInternal(DiyHomePlugin.Plugin, System.String, System.Object ByRef, System.Object[])
00efd364 1d8907eb .XXXX.AppStart(CAdapter*, Int64)
00efd380 1d890731 DomainBoundILStubClass.IL_STUB_ReversePInvoke(Int32, Int64)
后来又多试了几次，发现当加载了sos.dll扩展后，第一个扩展指令(任何指令)都会报错，然后第二条指令不会。看来要研究研究一下，开启Windbg调试自己：

(9ec.4b0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
sos!GetCMDOption+0x63:
00007ff9`a35ac7e3 488b01          mov     rax,qword ptr [rcx] ds:00000000`00000000=????????????????
0:004> ub
sos!GetCMDOption+0x3f:
00007ff9`a35ac7bf 4889542420      mov     qword ptr [rsp+20h],rdx
00007ff9`a35ac7c4 488bf1          mov     rsi,rcx
00007ff9`a35ac7c7 4d85c0          test    r8,r8
00007ff9`a35ac7ca 743f            je      sos!GetCMDOption+0x8b (00007ff9`a35ac80b)
00007ff9`a35ac7cc 488d5a18        lea     rbx,[rdx+18h]
00007ff9`a35ac7d0 392dd2e20500    cmp     dword ptr [sos!ControlC (00007ff9`a360aaa8)],ebp
00007ff9`a35ac7d6 0f85fe000000    jne     sos!GetCMDOption+0x15a (00007ff9`a35ac8da)
00007ff9`a35ac7dc 488b0dfd6d0500  mov     rcx,qword ptr [sos!g_ExtControl (00007ff9`a36035e0)]
0:004> k
 # Child-SP          RetAddr           Call Site
00 0000000a`05aad2c0 00007ff9`a35973ac sos!GetCMDOption+0x63
01 0000000a`05aad3b0 00007ff9`a86f353b sos!ClrStack+0x21c
02 0000000a`05aad570 00007ff9`a86f3718 dbgeng!ExtensionInfo::CallA+0x233
03 0000000a`05aad630 00007ff9`a86f37f8 dbgeng!ExtensionInfo::Call+0x16c
04 0000000a`05aad830 00007ff9`a86f2689 dbgeng!ExtensionInfo::CallAny+0x78
05 0000000a`05aad870 00007ff9`a872a89b dbgeng!ParseBangCmd+0x4a9
06 0000000a`05aadd30 00007ff9`a872b6ab dbgeng!ProcessCommands+0xa8f
07 0000000a`05aade00 00007ff9`a8685fe8 dbgeng!ProcessCommandsAndCatch+0x8f
08 0000000a`05aade70 00007ff9`a868628f dbgeng!Execute+0x24c
09 0000000a`05aae340 00007ff7`0d0c5c72 dbgeng!DebugClient::ExecuteWide+0x83
0a 0000000a`05aae3a0 00007ff7`0d0c60d5 windbg!ProcessCommand+0x2b2
0b 0000000a`05aae7c0 00007ff7`0d0c7c17 windbg!ProcessEngineCommands+0x185
0c 0000000a`05aaf800 00007ff9`d6a316ad windbg!EngineLoop+0x3e3
0d 0000000a`05aaf840 00007ff9`d7244629 KERNEL32!BaseThreadInitThunk+0xd
0e 0000000a`05aaf870 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

让我们为sos！g_ExtControl变量上的写访问设置一个断点，以查找其归零位置。

0:001> ba w8 sos!g_ExtControl
0:001> g
0:004> k
 # Child-SP          RetAddr           Call Site
00 0000008d`40d2d3f0 00007ff9`a7b1dc6e dbgeng!DebugClient::QueryInterface+0xb
01 0000008d`40d2d420 00007ff9`a7b471d0 sos!ExtQuery+0x2e
02 0000008d`40d2d450 00007ff9`a86f353b sos!ClrStack+0x40
...
0:004> k
 # Child-SP          RetAddr           Call Site
00 0000008d`40d2c080 00007ff9`a7b1dc6e dbgeng!DebugClient::QueryInterface+0xb
01 0000008d`40d2c0b0 00007ff9`a7b1d973 sos!ExtQuery+0x2e
02 0000008d`40d2c0e0 00007ff9`a86f2d25 sos!DebugExtensionInitialize+0x83
03 0000008d`40d2c120 00007ff9`a86f4182 dbgeng!ExtensionInfo::Load+0x48d
04 0000008d`40d2c3f0 00007ff9`a87312f6 dbgeng!ExtensionInfo::CheckAdd+0x6e
05 0000008d`40d2c430 00007ff9`a8731ca8 dbgeng!LoadSOSAndCheckVer+0x36
06 0000008d`40d2c690 00007ff9`a86f00e1 dbgeng!ProcessInfo::LoadClrDebugDllForExt+0x828
07 0000008d`40d2cec0 00007ff9`a7b61004 dbgeng!ExtIoctl+0xc6d
08 0000008d`40d2d410 00007ff9`a7b4722c sos!LoadClrDebugDll+0x24
09 0000008d`40d2d450 00007ff9`a86f353b sos!ClrStack+0x9c
...
0:004> k
 # Child-SP          RetAddr           Call Site
00 0000008d`40d2c0b0 00007ff9`a7b1d98d sos!ExtRelease+0x28
01 0000008d`40d2c0e0 00007ff9`a86f2d25 sos!DebugExtensionInitialize+0x9d
02 0000008d`40d2c120 00007ff9`a86f4182 dbgeng!ExtensionInfo::Load+0x48d
03 0000008d`40d2c3f0 00007ff9`a87312f6 dbgeng!ExtensionInfo::CheckAdd+0x6e
04 0000008d`40d2c430 00007ff9`a8731ca8 dbgeng!LoadSOSAndCheckVer+0x36
05 0000008d`40d2c690 00007ff9`a86f00e1 dbgeng!ProcessInfo::LoadClrDebugDllForExt+0x828
06 0000008d`40d2cec0 00007ff9`a7b61004 dbgeng!ExtIoctl+0xc6d
07 0000008d`40d2d410 00007ff9`a7b4722c sos!LoadClrDebugDll+0x24
08 0000008d`40d2d450 00007ff9`a86f353b sos!ClrStack+0x9c
...
0:004> g
(fe4.f70): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
sos!GetCMDOption+0x63:
00007ff9`a7b5c7e3 488b01          mov     rax,qword ptr [rcx] ds:00000000`00000000=????????????????
0:004> k
 # Child-SP          RetAddr           Call Site
00 0000008d`40d2d360 00007ff9`a7b473ac sos!GetCMDOption+0x63
01 0000008d`40d2d450 00007ff9`a86f353b sos!ClrStack+0x21c
...
sos!ExtQuery:
00007ff9`6a16dc40 48895c2408      mov     qword ptr [rsp+8],rbx
00007ff9`6a16dc45 4889742410      mov     qword ptr [rsp+10h],rsi
00007ff9`6a16dc4a 57              push    rdi
00007ff9`6a16dc4b 4883ec20        sub     rsp,20h
00007ff9`6a16dc4f 33f6            xor     esi,esi
00007ff9`6a16dc51 4c8d0588590900  lea     r8,[sos!g_ExtControl (00007ff9`6a2035e0)]
00007ff9`6a16dc58 488d1571760600  lea     rdx,[sos!GUID_d4366723_44df_4bed_8c7e_4c05424f4588 (00007ff9`6a1d52d0)]
00007ff9`6a16dc5f 4889357a590900  mov     qword ptr [sos!g_ExtControl (00007ff9`6a2035e0)],rsi
00007ff9`6a16dc66 488b01          mov     rax,qword ptr [rcx]
00007ff9`6a16dc69 488bf9          mov     rdi,rcx
00007ff9`6a16dc6c ff10            call    qword ptr [rax] ds:00007ff9`60e018d0={dbgeng!DebugClient::QueryInterface (00007ff9`60fd0240)}
sos!ExtRelease:
00007ff9`6a16ddc0 4053            push    rbx
00007ff9`6a16ddc2 4883ec20        sub     rsp,20h
00007ff9`6a16ddc6 488b0d13580900  mov     rcx,qword ptr [sos!g_ExtControl (00007ff9`6a2035e0)]
00007ff9`6a16ddcd 33db            xor     ebx,ebx
00007ff9`6a16ddcf 48891d02580900  mov     qword ptr [sos!g_ExtClient (00007ff9`6a2035d8)],rbx
00007ff9`6a16ddd6 4885c9          test    rcx,rcx
00007ff9`6a16ddd9 740d            je      sos!ExtRelease+0x28 (00007ff9`6a16dde8)
00007ff9`6a16dddb 488b01          mov     rax,qword ptr [rcx]
00007ff9`6a16ddde ff5010          call    qword ptr [rax+10h] ds:00007ff9`60e01358={dbgeng!DebugClient::Release (00007ff9`60fd0200)}

我们可以看到，IDebugControl2接口获得了两次调用，然后仍在使用被释放的sos!ClrStack函数。另外，我们可以看到sos.dll是第二次加载的。为了找出原因，让我们调试dbgeng!LoadSOSAndCheckVerr函数。在dbgeng!ExtensionInfo::Add方法我们可以看到它正在调用dbgeng!ExtensionInfo::FindByName方法中按名称查找扩展名dbgeng!ExtensionInfo::s_Chain扩展信息链表

0:003> k
 # Child-SP          RetAddr           Call Site
00 0000003a`a963c970 00007ff9`61044168 dbgeng!ExtensionInfo::Add
01 0000003a`a963c9d0 00007ff9`610812f6 dbgeng!ExtensionInfo::CheckAdd+0x54
02 0000003a`a963ca10 00007ff9`61081ca8 dbgeng!LoadSOSAndCheckVer+0x36
03 0000003a`a963cc70 00007ff9`610400e1 dbgeng!ProcessInfo::LoadClrDebugDllForExt+0x828
04 0000003a`a963d4a0 00007ff9`6a601004 dbgeng!ExtIoctl+0xc6d
05 0000003a`a963d9f0 00007ff9`6a5e722c sos!LoadClrDebugDll+0x24
06 0000003a`a963da30 00007ff9`6104353b sos!ClrStack+0x9c
...
0:003> r
rax=0000000000000036 rbx=0000003aa9505490 rcx=0000003aa963ca40
rdx=0000000000000000 rsi=0000000000000037 rdi=0000003aa963ca40
rip=00007ff961043dc5 rsp=0000003aa963c970 rbp=0000000000000000
 r8=0000000000000000  r9=0000003aa963c9f0 r10=0000000000000000
r11=0000003aa963c9e0 r12=0000000000000000 r13=0000000000000000
r14=0000003aa963c9f0 r15=0000003aa963cd01
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
dbgeng!ExtensionInfo::Add+0x95:
00007ff9`61043dc5 e876feffff      call    dbgeng!ExtensionInfo::FindByName (00007ff9`61043c40)
0:003> du @rcx
0000003a`a963ca40  "C:WindowsMicrosoft.NETFramewo"
0000003a`a963ca80  "rk64v4.0.30319SOS.dll"
0:003> u dbgeng!ExtensionInfo::FindByName
dbgeng!ExtensionInfo::FindByName:
00007ff9`61043c40 488bc4          mov     rax,rsp
00007ff9`61043c43 48895808        mov     qword ptr [rax+8],rbx
00007ff9`61043c47 48896810        mov     qword ptr [rax+10h],rbp
00007ff9`61043c4b 48897018        mov     qword ptr [rax+18h],rsi
00007ff9`61043c4f 48897820        mov     qword ptr [rax+20h],rdi
00007ff9`61043c53 4156            push    r14
00007ff9`61043c55 4883ec20        sub     rsp,20h
00007ff9`61043c59 4883cfff        or      rdi,0FFFFFFFFFFFFFFFFh
00007ff9`61043c5d 488bf2          mov     rsi,rdx
00007ff9`61043c60 488be9          mov     rbp,rcx
00007ff9`61043c63 4533f6          xor     r14d,r14d
00007ff9`61043c66 48ffc7          inc     rdi
00007ff9`61043c69 6644393479      cmp     word ptr [rcx+rdi*2],r14w
00007ff9`61043c6e 75f6            jne     dbgeng!ExtensionInfo::FindByName+0x26 (00007ff9`61043c66)
00007ff9`61043c70 488b1d91041c00  mov     rbx,qword ptr [C:WindowsMicrosoft.NETFramewo"
0000003a`a94fb360  "rk64v4.0.30319sos"
0000003a`a94f8830  "dbghelp"
0000003a`a94f8560  "ext"
0000003a`a94f8290  "exts"
0000003a`a94f7fc0  "uext"
0000003a`a94f7cf0  "ntsdexts"

如果没有，它会叫dbgeng!ExtensionInfo::Link方法将新的扩展dll添加到链接列表中。

0:003> r
rax=0000000000000000 rbx=0000003aa5f1e6b0 rcx=0000003aa5f1e6b0
rdx=0000003aa5f1e6d0 rsi=0000000000000037 rdi=0000003aa963ca40
rip=00007ff961043ed0 rsp=0000003aa963c970 rbp=0000000000000000
 r8=0000000000000000  r9=0000000000000000 r10=0000003aa95abd30
r11=0000003aa5f1e6d0 r12=0000000000000000 r13=0000000000000000
r14=0000003aa963c9f0 r15=0000003aa963cd01
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
dbgeng!ExtensionInfo::Add+0x1a0:
00007ff9`61043ed0 e8e7110000      call    dbgeng!ExtensionInfo::Link (00007ff9`610450bc)
0:003> du @rdx
0000003a`a5f1e6d0  "C:WindowsMicrosoft.NETFramewo"
0000003a`a5f1e710  "rk64v4.0.30319SOS.dll"
0:003> !list -x "du poi(@$extret+8)" poi(dbgeng!ExtensionInfo::s_Chain)
0000003a`a5f1e6d0  "C:WindowsMicrosoft.NETFramewo"
0000003a`a5f1e710  "rk64v4.0.30319SOS.dll"
0000003a`a94fb320  "C:WindowsMicrosoft.NETFramewo"
0000003a`a94fb360  "rk64v4.0.30319sos"
0000003a`a94f8830  "dbghelp"
0000003a`a94f8560  "ext"
0000003a`a94f8290  "exts"
0000003a`a94f7fc0  "uext"
0000003a`a94f7cf0  "ntsdexts"

最后是dbgeng!ExtensionInfo::CheckAdd方法调用dbgeng!ExtensionInfo::Load加载sos.dll：

0:003> u
dbgeng!ExtensionInfo::CheckAdd+0x69:
00007ff9`6104417d e816e7ffff      call    dbgeng!ExtensionInfo::Load (00007ff9`61042898)

这是DbgEng.dll中的一个错误。为coreclr添加了一个解决方案。不确定它是否已经进入了完整的框架。可以通过下面的命令之一来解决：

.loadby sos.dll clr

.loadby sos.dll clrjit

.cordll -ve -u -l