WinDbg basics HOWTO for QA engineers

Index

• Abstract
• Setup
  • Common settings, Target machine
  • Target machine settings for remote debugging (dual machine)
    NT3.51/NT4/2000/XP/2003
  • Target Vista machine settings for remote debugging (dual machine)
    Vista
  • Target machine settings for remote debugging (dual machine)
    on system setup stage or in
    Windows PE environment
  • Host machine
• Debugging -how to get logs and (if necessary) crashdump
  • Single machine (local) debugging
  • Remote debugging (two machines)

Abstract

There is such debugger - WinDbg. It is developed by Microsoft and can be downloaded from this location:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx.

In general there are 2 possible ways of debugging: with single machine and with 2 machines. When we have single machine we can store debug messages (logs) to file and if crash happens analyze automatically created memory dump (crashdump). After reboot, of course. When we use 2 machines, one of them (host) monitors another (target). The plus of two-machines way is that memory dump and latest logs can always be obtained against single machine method.

Setup

Common settings, Target machine

1. install latest version of DbgPrintLog with following command line:

   update.bat
   DbgPrintLog.exe --drv:inst B --drv:opt DoNotPassMessagesDown 1 --drv:opt BufferSize 16384
   

   This will allow to extract debug messages preceding system crash from memory dump and significantlyreduce traffic over communication channel if dual machine debugging is used. If you are going to debugboot-driver (e.g. for disk controller), use the following command line switch:
   --drv:inst 1
2. Setup crashdump:
   • Make sure that pagefile (virtual memory) is set to be greater than physical memory and is located on the same volume where running copy Windows is installed.
   • Make sure if there is enough disk space on the disk on the volume where running copy of Windows is installed. Memory dump includes snapshot of entire physical memory and some additional information.
   • My Computer -> Properties -> Advanced -> Startup and recovery
   • Enable memory dump
   • set dump type to Kernel Dump or Full Memory Dump.
     Attention: do not set Minidump/Small dump option, since it is absolutely useless for saving logs.
3. Setup verifier. It is Windows built-in driver validation tool, helps to detect problems early.
   • run verifier.exe. Since WinXP it is built-in, under 2000 it comes with DDK.
   • chose "Custom settings" - second option. Next
   • chose "Select from list" - bottom radio-button. Next
   • check everything, probably except "Low Resources Simulation". Next
     Attention: "Low Resources Simulation" - is intended to check driver's error handling in awfull stress conditions. It just shall not to crash. Never use this option for testing in normal operation mode.
   • chose "Select Driver from list" - last option. Next
   • check required driver(s). If target driver is not loaded at this moment, use "Add not loaded drivers to list" button under list. Drivers are usually located in WINDOWS\System32\drivers\ directory.
   • OK, do not reboot now
4. reboot machine to make sure all settings are saved to disk before installingapplication and/or driver to be debugged.

When single machine debugging is used, no more preparation is required. You can continue reading from Debugging section.

Target machine settings for remote debugging (dual machine)
NT3.51/NT4/2000/XP/2003

1. connect machines with IEEE1394 (FireWire) orNULL-modem cable.It is recommended to use cable withfull orpartial handshaking.
   Note: IEEE1394 is supported since Windows XP only. Windows 95/98/ME, Windows NT4 and Windows 2000cannot be used for remote debugging over IEEE1394.
2. do not forget to make common settings (see above)
3. open with some editor BOOT.INI file in the root directory of your boot volume.
4. find there record corresponding to your OS, smth. like this:

      multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional"
   

5. make a copy of this record in the next line
6. add to the end of copied string the following:
   • if IEEE-1394 cable is used:

            /debug /debugport=1394 /CHANNEL=3
         

   • if NULL-modem cable connected to COM2 is used:

            /debug /debugport=COM2 /baudrate=115200
         

   final string shall look like this (without line break):

   multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Pro" /noexecute=optin /fastdetect /debug
       /debugport=1394 /CHANNEL=3
   

7. make sure that TIMEOUT value in BOOT.INI has non-zero value
8. now BOOT.INI shall looks like this:

   [Boot Loader]
   Timeout=5
   Default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
   [Operating Systems]
   multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Pro" 
   multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Pro" /noexecute=optin /fastdetect /debug
       /debugport=1394 /CHANNEL=3
   

9. disable FireWire (IEEE1394) card in Device Manager:
   My Computer -> Manage -> Device Manager

Target Vista machine settings for remote debugging (dual machine)
Vista

1. connect machines with IEEE1394 (FireWire) orNULL-modem cable.
   Note: IEEE1394 is supported since Windows XP only. Windows 95/98/ME, Windows NT4 and Windows 2000cannot be used for remote debugging over IEEE1394.
2. do not forget to make common settings (see above)
3. run cmd.exe with administratice rights.
   Note: It is not the same sa run it under administrative account.This is special action available via context menu.
4. execute the following comamnds:
5. if IEEE-1394 cable is used:

      bcdedit /dbgsettings 1394 CHANNEL:3
   

6. if NULL-modem cable connected to COM2 is used:

      bcdedit /dbgsettings serial debugport:2 baudrate:115200
   

7. if USB-link is used with session name 'debugging':

      bcdedit /dbgsettings USB targetname:debugging
   

8. enable debugging:

      bcdedit /debug on
   

Target machine settings for remote debugging (dual machine)
on system setup stage or in
Windows PE environment

1. connect machines with IEEE1394 (FireWire) orNULL-modem cable.
   Note: IEEE1394 is supported since Windows XP only.
2. find and open with some editor txtsetup.sif file
3. find there:

   OsLoadOptions ="/fastdetect /minint"
   

4. and replace it with

   OsLoadOptions ="/fastdetect /minint /debug /debugport=COM2 /baudrate=115200"
   

   or

   OsLoadOptions ="/fastdetect /minint /debug  /debugport=1394 /CHANNEL=3"
   

Host machine

1. do not forget to connect machines with cable!
2. install WinDbg
3. put dbgprn.dll from latest version of DbgPrintLogto 'dbgext' subdirectory of WinDbg install directory.(e.g. C:\Program Files\WinDbg\dbgext)
4. Run WinDbg
5. follow menu items: File -> Kernel Debug ->
   • 1394, chose channel 3, OK, Answer 'Save' if prompted for workspace saving. If you run WinDbg for the 1st time, error message "Unable to open file" may appear. This means, that system didn't detect remote FireWire port on target machine yet. In this case do the following:
     • keep WinDbg open
     • reboot target machine and chose OS item marked with [Debug] or [Debugger Enabled] (or smth. like this) in boot menu
     • host machine shall display New Hardware Found message and install drivers for Debug Port. You may be prompted for Windows installation disk.
     • in some cases you may need to reboot host machine and repeat procedure once again.
     • if properly connected and configured as described above machines still cannot establish debug connection, I don't know what else can help. May be try to replace your FireWire PCI cards with different ones, I saw some cards, those seemed to be good, but didn't work properly on data transfers.
   • COM, just tell to which COM-port of host machine NULL-modem cable is connected and set speed to 115200.

Debugging

Single machine (local) debugging

1. run DbgPrintLog for capturing logs:

   DbgPrintLog.exe --full --cpu -l 10
   

   This will start console, where help message and current log file nameare displayed. Do not close it. Logs will be stored in files DbgPrintXXX.log insame directory. XXX in filename is a sequential number of file.The very first file has name DbgPrint.log. Previously created files are notoverwritten and numbering is continued. -l 10 instructs logger to keep only10 latest log files.
2. save debug info files .PDB from the build you are about to test to someseparate location.If problem happens only with Release build, please configure your project togenerate debug info for your binaries. If debug info is generated, .PDB filesshall appear near compiled .EXE, .SYS, .DLL. When tuning Debug Info options, donot chose 'Separate Types' option.
3. mark saved .PDBs somehow. If system crashes, .PDBs generated for the binaries were installedwould be required. Ideally, after each build binaries and their .PDBs should be stored together.
4. run dsync.exe from freboot packageto synchronize system cache with on-disk data. This improvesFile System consistency if crash happens. You can download it from the following address:
   http://alter.org.ua/soft/win/fastreboot
5. now install your drivers, services and application.
6. make your tests until something goes wrong:
   • Unexpected behavior
     1. switch to logger console and press 'Esc' to stop logging
     2. pack *.LOG files from the folder where DbgPrintLog.exe was run to archive.
     3. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
   • System crash, BSOD
     1. memory dump shall be created in the directory, specified during preparation. Usually it is created in WINDOWS directory and is named MEMORY.DMP.
     2. after reboot put copy of MEMORY.DMP together with binaries and .PDBs for possible future use.
        Note: verify creation time of this file. It may happen so, that you see dump of previous crash. In this case this file and further steps are useless.
     3. run WinDbg
     4. open MEMORY.DMP:
        File -> Open Crash Dump
     5. in the command prompt of WinDbg execute the following commands:

        !analyze -v
        !dbgprn.save -a -e --full --cpu -T R --file <full path to file>
        

        if !dbgprn.save says "Cannot find DbgPrnHk!DbgPrnHk_State export", try the following command:

        !dbgprn.lsig
        !dbgprn.save -a -e --full --cpu -T R --file <full path to file>
        

     6. Log will be saved to file <full path to file>. Pack obtained log-file to archive.
     7. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
     8. select entire text from log window of WinDbg and copy-paste it to some file, place this file together with copy of memory dump, archived logs and snapshot of binaries and .PDBs
   • System gets frozen
     1. In most cases this is actually BSOD. But system cannot generate memory dump for some reason. Usually, if you switch to dusl-machine debugging, you will see this BSOD in WinDbg and can make memory dump. Set it up :)
7. send archive(s) with logs to developer and tell what did you do before problem appeared.

Remote debugging (two machines)

1. do not forget to connect machines with cable!
2. Run WinDbg
3. follow menu items: File -> Kernel Debug ->
   • 1394 chose channel 3, OK, Answer 'Save' if prompted for workspace saving.
   • COM just tell to which COM-port of host machine NULL-modem cable is connected and set speed to 115200.
4. reboot target machine and chose OS item marked with [Debug] or [DebuggerEnabled] (or smth. like this) in boot menu.
5. WinDbg on host machine shall display information about establishing debug connection andprint some information about target system.
6. when target system is up, run there DbgPrintLog.exe for capturing logs:

   start DbgPrintLog.exe --full --cpu -l 10
   

   This will start console, where help message and current log file nameare displayed. Do not close it. Logs will be stored in files DbgPrintXXX.log insame directory. XXX in filename is a sequential number of file.The very first file has name DbgPrint.log. Previously created files are notoverwritten and numbering is continued. -l 10 instructs logger to keep only10 latest log files.
7. save debug info files .PDB from the build you are about to test to someseparate location.If problem happens only with Release build, please configure your project togenerate debug info for your binaries. If debug info is generated, .PDB filesshall appear near compiled .EXE, .SYS, .DLL. When tuning Debug Info options, donot chose 'Separate Types' option.
8. mark saved .PDBs somehow. If system crashes, .PDBs generated for the binaries were installedwould be required. Ideally, after each build binaries and their .PDBs should be stored together.
9. run dsync.exe from freboot packageto synchronize system cache with on-disk data. This improvesFile System consistency if crash happens. You can download it from the following address:
   http://alter.org.ua/soft/win/fastreboot
10. now install your drivers, services and application.
11. make your tests until something goes wrong:
    • Unexpected behavior
      1. switch to logger console and press 'Esc' to stop logging
      2. pack *.LOG files from the folder where DbgPrintLog.exe was run to archive.
      3. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
    • System crash, BSOD
      1. message about system stop shall appear in WinDbg window on host machine. Target machine will look frozen at this moment.
      2. in the command prompt on WinDbg execute the following commands:

         .dump /f <DumpFileName>
         !analyze -v
         !dbgprn.save -a --full --cpu -T R --file <full path to file>
         

         if !dbgprn.save says "Cannot find DbgPrnHk!DbgPrnHk_State export", try the following command:

         !dbgprn.lsig
         !dbgprn.save -a --full --cpu -T R --file <full path to file>
         

      3. Log will be saved to file <full path to file>. Memory dump shall be stored in <DumpFileName> file. Pack obtained files to archive.
      4. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
      5. select entire text from log window of WinDbg and copy-paste it to some file, place this file together with copy of memory dump, archived logs and snapshot of binaries and .PDBs
      6. if you see magic word "KeBugCheckEx" in WinDbg output - reboot target machine with power or reset button, since it is unusable in curent state. But if you see "first chance exception" - you can try continue execution with the following WinDbg command:

         g
         

12. send archive(s) with logs to developer and tell what did you do before problem appeared.