Updates to Driver Testing — Not Just for Certification

Long ago, Microsoft created the Windows Hardware Quality Labs (WHQL, pronounced “wickle”) program. The goal of this program was to certify that a particular device and its associated driver met a set of WHQL defined requirements. The requirements could include anything from how the driver was developed (e.g. which driver model it used) to whether or not the device and driver could survive the system being transitioned into and out of Standby multiple times. If your device and driver passed these requirements, your driver would be signed by Microsoft and you would be allowed to affix the “Designed for Windows” logo to your packaging. For this reason, we often referred to going through WHQL as “achieving logo” or “logo-ing”.

WHQL had two major downsides:

1. It wasn’t sufficiently transparent. When you submitted your driver for logo, you either passed or you didn’t. Also, it wasn’t clear at which point in your development process you were eligible for logo. As submissions weren’t free, it wasn’t in your interest to simply attempt to get a logo and see where you stood in the process.
2. The WHQL process happened at Microsoft’s campus. You had to physically ship them your hardware so they could validate it against their requirements. They were also under no obligation to send it back, which was, shall we say, inconvenient.

These downsides weren’t a mystery and Microsoft did work to make this process easier. Thus, WHQL was effectively scrapped for the Windows Logo Program. The big change with this new program was the ability to logo your device and driver using the Windows Hardware Compatibility Test (HCT). No more shipping stuff off to Microsoft, you could achieve logo in the comfort of your own office!

The HCTs were a nightmare to set up and use, so they were replaced with the Windows Logo Kit (WLK) for Windows Vista. The WLK was a nightmare to set up and use, so it was replaced with the Windows Hardware Certification Kit (WHCK) for Windows 7. Given that it’s now a Certification Kit, we also now certify devices and drivers instead of logo-ing them.

Needless to say, all of this turmoil turned off the driver development community from using any of these kits unless they absolutely had to. In this case, “absolutely had to” meaning that their customers actually cared about the Designed for Windows logo. This is unfortunate because the certification process is designed to find bugs and Windows incompatibilities in devices and drivers, why wouldn’t we want to go through that process in the hopes of increasing quality in our products?

To that end, with Windows 10 Microsoft is taking steps to make it easier to run the certification tests early and often. The first step is to kill off the WHCK and introduce the Hardware Lab Kit (HLK). This is said to be an “evolution” of the WHCK, so we can hope for continued ease of installation and use. Oh, and the testing isn’t called “certification testing” anymore… it’s called “compatibility testing.”

We’ll update on that when we’ve had a chance to play with the HLK more. One neat addition though is the ability to control the HLK process through C# and PowerShell, which should make automation possible. Unfortunately, the HLK will only be used to test drivers for Windows 10 and later, which means that you’ll still need a WHCK environment for older platforms.

The second step is more interesting still: It is now not only possible, but practical, to run a subset of the compatibility tests that are bundled with the HLK directly from your development system. At least that’s what the presentation at WinHEC in Shenzhen promised.

Microsoft recognized the desirability of getting driver devs to run a few tests right from their desks since Windows 8, when the WDK and Visual Studio were first combined. It’s always been a goal to “just press F5” to rebuild your driver, lob it onto a target system (a real system in your office or a VM, depending on the type of driver you’re writing), have it installed automatically, and start a series of basic tests. It MIGHT be possible to do this today with the Windows 8.1 Driver Kit, if you set things up right and everything works. Peter swears he’s seen it work once. I don’t know whether I should believe him, because I’ve never seen it work. But according to a talk at WinHEC Shenzhen, the WDK for Windows 10 will make this process a reality.

The Device Fundamentals, or DevFund, tests have long been a part of the Windows Certification testing matrix. DevFund contains a great set of generic tests for things such as correct PnP, Power, and I/O handling. The Windows 10 WDK reportedly will make it easy to launch the DevFund tests directly from within Visual Studio or from a USB drive, which means that passing DevFund can easily be incorporated into a developer’s test procedure. While passing DevFund doesn’t exactly mean that your code is bug free, it’s a great start to ensuring a baseline of quality in your code.

The Windows 10 WDK brings with it a significant number of improvements, ranging from support for Visual Studio 2015, to support for building Universal Drivers. The ability to easily launch a basic set of tests, with one keystroke, would be the realization of a long-awaited goal. We’re looking forward to spending some time testing this feature, and we’ll be sure to report back to you what we find.