By Nik Twerdochlib
While progressing through the WHQL battle for one of our drivers, we hit a proverbial wall; full speed. The “wall” popped up out of nowhere under Windows 8.1 while running the DevFund tests. Almost all these tests failed, yet the same test continued to pass on other platforms: Windows 7, Windows 10, Windows Server 2019… So, my first step down the path of enlightenment begins.
After gaining some better insight into the issue, we decided to make a change to how IOCTLs would be sent from the user mode service. Prior to this, a user mode service would open a handle to the device and pass IOCTLs directly. This driver emulates a physical device (Smart Card Reader), and as such we had been skeptical about handling the interaction with the device this way. A Smart Card Reader can be accessed a few different ways. So, it must support potentially having multiple connections. A symbolic link is even created to support access from legacy software. This meant care had to be taken with the handle the user mode service established. The user mode service also controls when one of these devices is created, what its configuration is, and when it is removed through a virtual bus driver.
The change we decided to implement was to remove the complexity of having the user mode service manage the additional handle to the Smart Card Reader device and instead have the user mode service communicate solely through the bus driver This allowed us to simplify the logic in the service, reduce the overhead of managing the open handle to the device, and allow our custom IOCTLs to be processed as internal IOCTLs in the device. This change would also address the issue of contending with all the Smart Card IOCTLs being sent to the device by the Smart Card Manager (SCardMgr). It also gave us peace of mind of not having our custom IOCTLs accessible through publicly available device.
To provide the ability for the bus driver to accept IOCTLs for a device and then forward them to that device, an IoTarget is created for each new device created on the bus. A unique identification number is stored in a context for the IoTarget, allowing for a method of determining which IoTarget represents the desired target device. IOCTLs received by the bus driver are first formatted as internal IOCTLs using WdfIoTargetFormatRequestForInternalIoctl. A completion routine is added to allow for the bus driver to complete the Request, and then sent to the target device using WdfRequestSend asynchronously.
The Problem
Initial testing after all these changes were put in place was a little surprising. The Requests failed the format call, always! Our trusty companion WinDbg showed that the error was STATUS_REQUEST_NOT_ACCEPTED.
Alright, why? Based on the error code’s “name” I immediately set out to understand why the function driver would not accept the request. At first, I suspected a discrepancy in the actual IOCTL value, but that would have been too easy. I dumped the WDF log in the debugger expecting to see some relevant message(s) about the request arriving and then being rejected. After all the error was STATUS_REQUEST_NOT_ACCEPTED. You know that level of disappointment you get when you don’t see something you fully expected to see? Well, this was one of those times. There was nothing in the log containing any reference to the Request that had been attempted to be formatted and sent. Hopefully I am not the only developer that did not know exactly what WdfIoTargetFormatRequestForInternalIoctl is really doing under the hood…
Turning focus back to the bus driver, I dumped the WDF log and found the relevant entries:
347: FxIoTargetFormatIoctl – enter: WDFIOTARGET 0x0000307E69F15228, WDFREQUEST 0x0000307E6A7ED198, IOCTL 0x55ff9c0e, internal 0, input WDFMEMORY 0x0000307E6A7ED0C1, output WDFMEMORY 0x0000307E6A7ED0B1
348: FxRequestBase::ValidateTarget – Cannot reallocate PIRP for WDFREQUEST 0000307E6A7ED198 using WDFIOTARGET 0000307E69F15228, 0xc00000d0(STATUS_REQUEST_NOT_ACCEPTED)
349: FxIoTargetFormatIoctl – Exit WDFIOTARGET 0x0000307E69F15228, WDFREQUEST 0x0000307E6A7ED198, 0xc00000d0(STATUS_REQUEST_NOT_ACCEPTED)
Confident that the problem was confined to the bus driver, I set about to understand why. The task I was trying to perform was a simple one, an implemented in hundreds of drivers. Why does it fail here? Time to leverage the recently open sourced WDF code by Microsoft.
In Search of the Truth
When I first learned Microsoft published the WDF source on github.com, I remember thinking “How cool!”. I went straight to it and gave it a quick run through. Now it was time for me to dig deep and understand. Searching for the phrase “Cannot reallocate PIPR for WDFREQUEST” led me to FxRequestBase::ValidateTarget. I worked my way backwards from this point to gain a better understanding of what was going on. From the WinDbg session, I knew that the WDFREQUEST was valid, and that its current IRP was also valid. So, I could ascertain from the code in this method that the call to FxIoTarget::HasValidStackSize was not failing, and thus it must be the call to FxIoTarget::HasEnoughStackLocations that was the culprit. Now the question as to why?
At this point the thought crossed my mind that perhaps I had pushed this particular Windows 8.1 test system just a smidge too far, and it would be worthwhile to switch to a fresh system just to confirm I wasn’t in fact going down a rabbit hole! On a fresh test system, the same issue occurred. It was also reproducible on a few different Windows 10 test systems.
To gain a better understanding of what was going on, I moved the driver over to a Windows 10 test system where I could step through the WDF code within the WinDbg session. This was a big help and proved extremely insightful. Stepping through the code landed me in FxIoTarget::HasEnoughStackLocations. This exposed that the WDFREQUEST had a smaller stack size than the target.
Some deeper insight into this function driver: As stated earlier, it emulates a Smart Card Reader. For those whom have not developed a Smart Card Reader driver under windows, Microsoft provides a WDM library (smclib.lib) to make the implementation process much easier, which it does. This library assumes the responsibility of handling all Smart Card related IOCTLs and provides callbacks to allow for the hosting driver to customize certain functionality. As Smart Card IOCTLs arrive, the driver simply needs to pass them down to smclib with a call to SmartCardDeviceControl.
During device creation, we must increase the stack size by a value of 1 so that we can add a completion routine, allowing the driver to complete the request rather than through smclib. This little detail turned out to be one of the keys to understanding the problem.
The Ah-Ha (No Sh*t) Moment
Something I can honestly state as not having given much thought to it previously, is that even when a driver owns a Request, that does not necessarily mean the driver originated the underlying IRP that Framework Request represents. The Request in this scenario originates from a user mode service calling DeviceIoControl. The Framework did not create the underlying IRP associated with the Request, instead it was dispatched to this driver. As a result of this, the Framework is not free to deallocate that original IRP and allocate a new one with the necessary number of I/O Stack Locations. Thus, the Framework cannot allocate the required additional stack locations and the call to FxBaseRequest::HasEnoughStackLocations fails.
This can be further verified by following the flow through the Framework source. Starting from where IRPs arrive in the Framework, FxPkgIo::Dispatch, an FxIrp object is created to represent the incoming IRP. FxPkgIo::DispatchStep1 is then called which passes the request to any registered dynamic dispatch callbacks if one exists, otherwise it calls FxPkgIo::DispatchStep2 which handles placing the request on the appropriate queue. In FxPkgIo::DispatchStep2 a call to FxRequest::_CreateForPackage is made to create an FxRequest object associating the IRP to the device. The FxRequest object is created by a call to FxRequestFromLookaside, which allocates the request from the driver tracking pool. The FxRequestFromLookaside class is derived from FxRequest and passes in a key value as the third argument to the constructor of its base class: FxRequestDoesNot OwnIrp.
We now have confirmation that the driver does not own the underlying IRP and the answer to why the Framework isn’t correcting the stack size issue for us behind the scene.
The Fix
Having now identified the root cause of the issue as the stack size of the bus driver device being less than the stack size of the function driver device, what do we need to do to fix the issue? The stack size of the bus driver device must be increased to include enough stack locations to allow a Request to be forwarded to the function driver. In my case, the function driver gets initialized with a stack size of 2 which is then increased by 1 to accommodate for the use of a WDM IO completion routine. This leaves it with a stack size of 3. The IoTarget that is created to connect to the function device will increase the stack size by 1, so the stack size of the bus driver will need to be incremented by a value of 5 (the stack size presented in the IoTarget plus 1) to allow for the forward operation. This allows the check for having enough stack locations to pass when formatting the Request to forward it to an IoTarget. With this change in place, the bus driver was in fact able to successfully format the Request and forward it on to the child device.
The takeaway from this endeavor has caused me to reflect upon the intricacies of the relationship between WDF and WDM. These include the importance of not losing sight of WDM functionality underneath WDF, how beneficial Microsoft’s release of the WDF source really is, oh, and the importance of understanding the relationship between your drivers.
Nik Twerdochlib got his introduction to programming at a young age shortly after his father brought home a Texas Instruments TI99. This soon gave way to a lifelong fascination with the interaction of software and hardware and embedded computing. Nik can be reached at nt@nikom.net.
OSR Open Systems Resources, Inc.