Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Analyst’s Perspective: 10 WinDBG Commands You Might Not Know (But Should)

Can you count the number of WinDBG commands you know on one hand? Been meaning to learn some commands other than !analyze –v but been too busy to crack the docs open? Well then, this article is for you! I’m going to break down ten WinDBG commands that I couldn’t live without.

System Information Commands

Sometimes as part of your analysis, you’d like a bit more detailed information about the target system that generated the crash dump. The commands in this section are going to let you find out critical details about your system that just might be the clues you need to perform your analysis.

!vm

Don’t be fooled by the name, the !vm command gives you a great quick view into the virtual and physical memory usage on a system. When I run !vm I like to use a flags value of 0x21, which will omit some process specific memory usage information and add in some extra info about the kernel address space on platforms that support it:

NOTE: The !vm output currently has a bug where the non-paged pool usage will always be listed as zero. The actual non-paged pool usage is listed as, “NonPagedPoolNx Usage” in the output.

Note here that we see the amount of physical memory in the system as well as how much memory is currently free. We then get to note the current usage of the system PTEs as well as the pools. If we suspect some sort of resource exhaustion going on in the system, we can use this command to quickly pinpoint which resource is being consumed.

!sysinfo

Do you have a customer that can repeatedly reproduce a problem but you just can’t reproduce it with the exact same procedure? Maybe you’re not using a fast enough processor or the right BIOS version, but in any event, how can you tell what system configuration the customer is using from just a dump file? Enter !sysinfo, a command that can tell you just about anything you’d want to know about your system using information cached on the target. For example, let’s see what kind of processor is in this system:

How about the BIOS version and other platform information?

There’s more here as well if you go exploring the documentation for the command. For example, you can even query information about which RAM slots are populated using the smbios switch (e.g. !sysinfo smbios –memory).

Suspected Race Condition Commands

Race conditions are the worst. They’re difficult to track, difficult to reproduce, and when you get a crash it may be too late. The race has already happened and when the system crashes you’re dealing with the secondary failure, so there’s nothing that can be done, right? Wrong! WinDBG has a couple of commands that can make you feel like you’ve won the lottery and pinpoint the racing thread with ease.

!running

If you’re lucky, the thread that is racing with your crashing thread is still running on another processor. This is where !running comes in, which will show you information about each thread that is currently running on a processor in the system. Whenever I run this command I like to specify the –ti switch, to include thread stacks in the output as well as idle threads:

!ready

If the thread isn’t actively running, you might think that you would have to go the long way and try finding a racing thread with !process 0 7. However, WinDBG also provides us a way to look at threads that are ready to run, with the !ready command. Maybe the current thread pre-empted another thread and that’s the reason for the race, in which case the other thread will be in the ready state. Whenever using !ready, I like to use the 0xF flags value so that I can see the call stacks of the threads, though I won’t do that here just to keep the output short:

Memory Analysis

Have an address and want to know what it is? Is it a pool allocation? Is it paged out? Here are a couple of commands that will get you the information that you need.

!pool

!pool is a standard command for any toolbox, so I suspect that most of you know it and love it already. However, for those that might not be aware, !pool will take an arbitrary virtual address and let you know if it is a pool allocation or not. If it is indeed a pool allocation, you’ll be told some details about it, such as whether it’s allocated or freed, the length of the allocation, the tag, etc. When I use !pool, I like to specify a flags value of 2 to suppress information about other allocations surrounding the address:

Before moving on, I’d like to note something in the output here that often confuses people. The previous size value mentioned here is not the, “previous size of this allocation.” Instead, what it is telling you is the size of the allocation preceding this entry in the pool page. This is used as part of a consistency check by the Memory Manager to validate that the page of memory has not been corrupted by buffer overruns or underruns.

!pte

Sometimes you’d like to view the virtual memory structures for a given virtual address, such as the PDE and PTE. In that case, you can use the !pte command, which will provide decoded information about a virtual address. Here’s some example output for a valid virtual address:

We can also see what happens if we specify a virtual address that isn’t valid to the hardware, such as one with its backing page currently in transition:

Now we have some further details as to why the address is invalid, which may be invaluable to our investigation.

Viewing O/S Trace Information

The O/S has some built in trace facilities that you can turn on to collect data that might be useful during analysis. Unfortunately these facilities need to be turned on before the problem happens, but knowing that this information is available can be useful in some situations.

!verifier

We’re all using Driver Verifier, right? Well, what you might not realize is that starting in Windows Vista Verifier has been enhanced to keep a log of interesting events that happen in your driver. Assuming that you’ve enabled Driver Verifier on your driver, you can now extract valuable information with the following !verifier commands:

  • !verifier 0x80 Address – This command dumps the allocate and free log, which logs each pool allocate and free made by your driver. Included in the output is the call stack of the operation, which can be invaluable when you’re trying to track down use after free or double free bugs. Optionally, the command takes an address value that will limit the output to only include allocation ranges including that address.
  • !verifier 0x100 Address – This command dumps the IRP log, which logs each call to IoAllocateIrp, IoCancelIrp, and IoCompleteRequest made by your driver.
  • !verifier 0x200This command dumps the critical region log, which logs each call to KeEnterCriticalRegion and KeLeaveCriticalRegion made by your driver.

!htrace and !obtrace

Handle leaks and object reference leaks can be very tricky to track down, especially when working with a large code base. Luckily, the O/S has built in facilities for logging handle and reference count activities. All you need to do is enable them and be aware of the commands available for extracting the logs, which in this case are !htrace and !obtrace.

Handle tracing needs to be enabled on a per-process basis, which can be done by using Application Verifier. As driver writers, however, we’re typically only interested in kernel handles. By implementation, kernel handles are actually just handles from the handle table of the System process. And, as luck would have it, if you enable Driver Verifier handle tracing is automatically turned on for the System process. Thus, as long as Driver Verifier is enabled on the target you can dump the handle tracing log for all kernel handles with !htrace 0 PEPROCESS:

Object reference tracing, on the other hand, needs to be enabled on a system wide basis with GFlags. Due to the volume of tracing generated, when you enable tracing you must specify the pool tag of the object you want to trace (e.g. ‘File’) and you can also limit the tracing to only apply to a single process’ objects. Once you have enabled tracing via GFlags, you can view the trace for a given object with !obtrace:

Plug and Play and Power Issues

Nothing is more annoying than when the system hangs during a plug and play or power operation. Luckily, the debugger provides a quick way to identify the threads participating in the operation so that you can get right to resolving the issue.

!pnptriage

!pnptriage is a nifty command that combines the output of several PnP related debugging commands. It will identify any of your devnodes with problems as well as dump out any PnP worker threads that are currently executing, which will give you the ability to quickly identify the threads in the system that might be of interest to you:

!poaction

!poaction is the essential command for debugging any of your power related issues. Most importantly, !poaction will show any outstanding query or set power operations and the driver to which they were sent, which can be used to quickly identify which devices are preventing the power operations from occurring. Great for getting insight into what’s going on when the system will mysteriously refuse to enter or resume from a lower power state:

Did I Miss Any?

Got your own favorite command that wasn’t represented here? Send me an email at ap@osr.com and let me know!

 

Analyst’s Perspective is a column by OSR Consulting Associate, Scott Noone. When he’s not root-causing complex kernel issues, he’s leading the development and instruction of OSR’s Kernel Debugging seminar. Comments or suggestions for this or future Analyst’s Perspective columns can be addressed to ap@osr.com.

Summary
Article Name
Analyst's Perspective: 10 WinDBG Commands You Might Not Know (But Should)
Description
Chief analyst Scott Noone reviews WinDBG commands that we find useful in our own kernel debugging work (and you will too!)
Author