OSR

Windows System Software -- Unique Expertise, Guaranteed Results

Fix Your (Offline) Symbols

Community Contributor
David Boyce

Scott Noone’s blog post entitled Setting the WinDbg Symbol Search Path (you can read it here) contains some great advice. But his advice doesn’t really apply if you’re working in an isolated environment where you don’t have direct access to Microsoft’s Public Symbol Server.  In these environments, there’s typically some mechanism to logically or physically isolate your testing environment from the Internet.  This isolation typically makes use of what’s called an ‘air gap’ between your debugging environment and the Internet.  If your organisation has this type of network isolation in place, presumably there are good reasons why it’s there.  If you still have to debug your driver… what do you do?

The question becomes: Is it possible to make a debug target’s (T’s) kernel symbols available to your (isolated) debugging host (H)?  Yes, if you can meet the following conditions:

  1. You have access to the internet somewhere (machine ‘C’ above).
  2. You have a means of transferring data in both directions across the air gap;
  3. The means described in (2) is acceptable to your organisation.

Conditions (1) and (2) are basically technical issues, and the reader is assumed to be able to address them.

It’s condition (3) that can get you into trouble if you don’t pay careful attention.  Some organisations will be concerned about bringing malware in from the outside, while others will be concerned about letting internal data out.  These are both legitimate concerns, and any organisation which has set up an air gap will know what matters to them.  Your job (or more) may depend on correctly following the rules laid down in this area.

Figure 1 -- An "Air Gap"
Figure 1 — An “Air Gap”

SymChk with Manifest Files

The basic approach used here is described in the Microsoft Dev Center page Using a Manifest File with SymChk.  But here I’ve also included the extra details needed to address the particular requirements of WinDbg kernel symbols.

In brief, the procedure is:

  1. Create a symbol manifest file on the target (T, in Figure 1);
  2. Transfer that manifest across the air gap to a machine (C, in Figure 1) with internet access;
  3. Use the manifest to obtain the symbol files from the Microsoft Symbol Server;
  4. Transfer the symbol files back across the air gap to the debug host (H, in Figure 1).

A few things specific to kernel-mode debugging are also important:

  • Use the architecturally-appropriate version of SymChk on the target, i.e. 32-bit on x86 targets, and 64-bit on x64 targets;
  • Use the name of the NT kernel for /ie argument;
  • Avoid name resolution delays arising from attempts to access the public server (this is what often makes running SymChk offline painfully slow).

SymChk for the Target Architecture

You will need the following executables gathered together for the target platform:

  • symchk.exe
  • SymbolCheck.dll
  • symsrv.dll
  • symsrv.yes
  • symstore.exe
  • dbghelp.dll
  • dbgeng.dll

You can find these in the Debuggers folder of your WDK installation on your debugging host machine (H, in Figure 1).   You may need to install the WDK on a machine of the “other” architecture (perhaps a VM) to get that architecture’s version.

You cannot run the x64 versions on a 32-bit OS for obvious reasons, and if you try to use the x86 versions on a 64-bit OS, you will see messages like

[SYMCHK] Can't load file "C:Windows\system32\hal.dll"

for each 64-bit executable.

I have two sets of these (one set for 32-bit systems, one for 64-bit) on a USB stick in separate folders so I can run the version appropriate to the target.

On the Debug Target

Open a command prompt and enter a command similar to:

symchk /om manifest.txt /ie ntoskrnl.exe
/s c:\Users\Foo\DeskTop\Empty

Here, manifest.txt is name of the manifest file to be created (‘/om’) which will be transferred across the air gap outwards.   The other arguments are explained below.

/ie Needs the Running Kernel’s Name

It might seem obvious, but if you want the kernel symbols, you’re going to need the name of the kernel executable, which is always in %SYSTEMROOT%\System32\ .   In current versions of Windows, this name is (nearly) always ntoskrnl.exe, although I have also seen ntkrnlpa.exe on XP.   In the following example, it’s clearly ntoskrnl.exe, as the kernel is unlikely to be named ntprint.exe:

C:> dir /b %SYSTEMROOT%\System32\nt*.exe

ntoskrnl.exe

ntprint.exe

symchk’s /ie option requires the name of a currently executing program.  In the isolated environment scenario, there’s a straightforward, if somewhat counter-intuitive, way to determine whether you have an incorrect executable name: if it reports

SYMCHK: FAILED files = 0

you’ve got the wrong name!  A non-zero value indicates that it found the executable.

Of course, this isn’t completely fool proof, since using the name of any currently executing program (not just the kernel) will produce similar reports.  It does act as a useful sanity check, though.

When you’re done, you’ll know whether you’ve got the right symbols by the way WinDbg responds when you have completed the transfer.

Avoiding Name Resolution Delays on the Target

This is easy:  create an empty folder and tell SymChk that’s where to get its symbols from (/s):

mkdir c:\Users\Foo\DeskTop\Empty

symchk /om manifest.txt /ie ntoskrnl.exe
/s c:\Users\Foo\DeskTop\Empty

By specifying an empty local folder, symchk doesn’t attempt to contact any symbol server.

Data Going Out

At this point, you should have a manifest file which needs to be transferred across the air gap to the internet-connected machine (H, in Figure 1).  The content of the file is a normal text file which can be opened by any editor.  Each line represents an executable of some kind together with information indicating its version.  Typically there are between 200-300 such lines in manifests wanting kernel symbols.

The vast majority of these lines represent executables which would be present on any Windows installation, and fetching their symbols does not represent any kind of security breach.  However, it is possible that in some secure environments, there are executables listed for which the presence on the target system is confidential, and the attempt to fetch their symbols from the server would expose this.  The simplest remedy (assuming that you don’t actually need their symbols) is simply to remove lines referencing them from the manifest before transferring it.

Fetching the Symbols

On the host with internet access (C, in Figure 1), prepare an empty folder (e.g. C:\MySymbols) to receive the symbols and run the following command:

symchk /im manifest.txt
/s SRV*C:\MySymbols*http://msdl.microsoft.com/download/symbols

Here, manifest.txt is the transferred manifest file which determines which symbols are to be obtained from the Microsoft public symbol server http://msdl.microsoft.com/download/symbols and placed below C:\MySymbols.

On success, C:\MySymbols contains the kernel symbols (and a number of others) that you need for debugging.  The content of this folder needs to be transferred back across the air gap to the debug host (H, in Figure 1).

When using the /im option on a 64-bit machine, you can run either version of symchk.exe.

Fetch Errors

Don’t be surprised to see some messages of the form

SYMCHK: <filename> ERROR = Unable to download file. Error reported was 2

SYMCHK: FAILED files = <non-zero number>

even when <filename> is ntoskrnl.exe, provided the reported PASSED + IGNORED files is also non-zero.

For these purposes, such messages are (usually) benign; what’s important is that C:\MySymbols is no longer empty and contains the kernel’s PDB folder, e.g.  ntkrnlmp.pdb\.

Data Coming Back

The symbols files are binaries – a mixture of .dll, .sys and .pdb files.  You may wish to run a malware scanner over these before transferring them.

Finally…

When you’ve got your symbols back across the air gap, you have a local symbol cache at which you can point your WinDbg in the usual way.   Good bug hunting!

 

 
David cut his programming teeth at Honeywell working on a middleware product written in assembler, then migrated to C and C++ product development on various platforms.  He’s come relatively recently to Windows Drivers, but hey – he always enjoys a technical challenge!  He can be contacted at dboyce@becrypt.com.