Last reviewed and updated: 10 August 2020
Most of the WDF topics I write about suggest themselves to me as a result of a set of exchanges on our NTDEV list, or as a result of trends I see in driver code that I review or update. This time, I was inspired by both of these.
There was a recent, and reasonably interesting, thread on NTDEV about the use of Cleanup and Cancel Event Processing Callbacks. Also recently, I had the opportunity to update two very different device drivers. One was a WDM driver that was written in the NT V4 timeframe. The other was a KMDF driver written by a new Windows driver developer shortly after the release of KMDF. While these two drivers were for very different types of devices, they had one thing in common: The developers of both misunderstood how to terminate in-progress Requests and used (or, rather, misused) Cleanup events in place of Cancel.
That’s how I came to be motivated to write at length about Cleanup, Close, and Cancel handling in WDF drivers. An additional, private, motivation is that while we cover this topic in our Writing WDF Drivers seminar, we almost never have the time to talk about it in the level of depth that I’d prefer. So, I figured an article would be useful for many purposes.
Some Preliminaries
The first thing to understand is that, at least in the driver world (as opposed to the file systems world), you need to consider processing for Cleanup, Close, and Cancel together. This is because they’re often confused and processing for these functions often interacts.
Also, as we begin our discussion, please keep in mind that what we say here about Cleanup, Close, and Cancel is entirely specific to WDF. As in most things WDF, it is never a good idea to attempt to “map” from WDM to WDF (or vice-versa) in your head and expect to be able to figure things out. That way madness lies. The WDF Framework is its own I/O processing ecosystem. Sure, it interacts with WDM, but the rules for how WDF drivers work are established solely and entirely by WDF. Just because something works a certain way in WDM is no reason to believe the same is true for WDF.
The Problem
Let’s start with a couple of simple examples of the problems that Cleanup, Close, and Cancel are designed to solve. I want to emphasize that these are just two random examples in which these operations can be useful. There are lots of other important uses for Cleanup, Close, and Cancel events that these two examples don’t address.
Assume your driver has a single Queue that uses Sequential Dispatching. A user-mode application successfully calls the Win32 function CreateFile specifying FILE_FLAG_OVERLAPPED to allow it to do asynchronous I/O. The app then successfully calls ReadFile 100 times from within a loop, specifying an OVERLAPPED structure each time. The result is that when the application’s loop is complete, there are 100 Read Requests pending in your driver.
Now let’s say that application calls CloseHandle on the handle it used to send those 100 reads to your device. At least some of those 100 reads are still in progress. What happens to the in-progress I/O Requests? How does your driver handle this situation?
As a second example, consider what happens when instead of closing the handle to your device when those 100 reads are in progress, the application exits or is terminated. As Windows begins termination processing for the app, your driver potentially has a pointer to one or more data buffers into which to return data into the application’s address space. You can see this in Figure 1.
In Figure 1, you can see a driver with a single WDFQUEUE that has two Read Requests queued (I didn’t want to put all 100 I/O Requests in the diagram). Each Read Request has (stored internally) a pointer to the data buffer into which the Read (output) data is to be returned.
If Windows were to allow the Process to terminate completely with I/O requests in progress, and if it were to free the physical memory the process was using including the application’s data buffers, the driver would have the potential of writing on random memory and corrupting the system. That would be a very bad thing. As a result, Windows will not allow the process to exit completely until its I/O operations are completed. This prevents a driver from unintentionally corrupting memory by processing an ordinary Read Request. Thus, when an application exists with I/O Requests in progress, Windows won’t allow the application to exit, and may even appear to hang, until the pending Requests are completed.
So, when an application exits or is terminated, the question once again becomes: What happens to the in-progress I/O Requests? How does a driver handle this situation?
Cleanup, Close, and Cancel Defined
The two problems we just described, closing a handle with I/O in progress and an application attempting to exit with I/O in progress, are two of the primary issues that Cleanup, Close, and Cancel are designed to manage. Let’s start with some basic definitions of each of these operations. In the next section, we’ll expand on these definitions and describe more about the specifics of how your driver can handle the operations.
Event Processing Callbacks for Cleanup, Close, and Cancel are all technically optional in WDF. That means, your driver will need to specifically opt-in to handle them. Cleanup and Close are File Object specific, so Event Processing Callbacks for these operations are related to WDF File Object handling. If, as you read this section, you feel like you need a quick refresher about Windows File Objects, see the sidebar About File Objects.
[infopane color=”6″ icon=”0182.png”]About File Objects
A File Object in Windows – the WDM data structure named FILE_OBJECT and the WDF Object WDFFILEOBJECT – represents a single, specific, open instance of a file or device. For example, when an application successfully calls CreateFile, a File Object is created that represents that open instance. If the call to CreateFile is successful, it returns a handle to this File Object, that the application uses to perform subsequent I/O operations on the file or device. The returned handle is a process-specific reference to a particular File Object.
The File Object is the general I/O Subsystem object that is used to keep track of and manage I/O operations on a given handle. For example, when you issue sequential read operations to a file on disk without specifying a specific file offset, the file system uses the File Object to keep track of the next offset in the file to read. Each File Object is almost always associated with only a single handle. There are, however, some relatively rare cases (such as when the Win32 function DuplicateHandle has been used) when there can be more than one handle associated with a single File Object.
You can read more about WDF File Objects, their properties, and the Event Processing Callbacks associated with them, in The NT Insider article WDF File Object Callbacks and Properties Demystified.
Each time the application issues an I/O request using the File Object’s handle, the reference count on the File Object is incremented. Each time a driver or file system completes an I/O operation on a given File Object, the File Object’s reference count is decremented.
[/infopane]
Let’s start by discussing Cleanup. When the last handle to a File Object is closed, Windows issues a Cleanup request and WDF calls the associated driver’s EvtFileCleanup Event Processing Callback function. Because there’s almost always only one handle associated with a given File Object, as soon as an application calls CloseHandle (and synchronously from within that function call), WDF will call the driver’s EvtFileCleanup Event Processing Callback. EvtFileCleanup is always called in the context of the thread/process calling CloseHandle, and at IRQL PASSIVE_LEVEL. For a description of the unusual case when an apps call to CloseHandle may NOT result in an immediate Cleanup operation, see the sidebar entitled About File Objects.
In addition to calling EvtFileCleanup, WDF goes one step further in its Cleanup processing. After calling the driver’s EvtFileCleanup Event Processing Callback (if any), WDF scans your driver’s WDF Queues for Requests originally submitted via the File Object that’s the target of the Cleanup. For each such Request found, WDF issues a Cancel operation. We’ll discuss the details of Cancel later, but for now let’s just say that issuing a Cancel operation for these Requests almost certainly results in their being removed from the Queue and completed by WDF with an error status.
It is important to note, however, that as part of handling Cleanup, WDF does not perform Cancel processing for pending Requests that are not stored on a WDF Queue. Thus, if you have an in-progress Request, and you’re keeping track of that Request by storing its WDFREQUEST handle in your Device Context, that Request will not automatically be Canceled by WDF as part of Cleanup processing and therefore will remain in progress.
There are some other interesting special cases where Windows instantly follows a Cleanup operation with a Cancel for Requests sent via the associated File Object. For a discussion of these cases, see the sidebar When Cleanup Becomes Cancel.
[infopane color=”6″ icon=”0182.png”]When Cleanup Becomes Cancel
From reading the article, you already know that when WDF receives a Cleanup operation for a given File Object, it issues a Cancel operation for any Requests issued via the File Object that are on WDF Queues. There are two other times when the Windows I/O subsystem extends Cleanup processing by issuing a Cancel. These are when the I/O operations that are issued are associated with an I/O Completion Port, and when the I/O operations that are issued come from a thread in a Thread Pool.
Completion Ports and Thread Pools are two optimized mechanisms designed to efficiently handle asynchronous callbacks to an application. The case we’re particularly interested in is handling of I/O completion callbacks.
When the last handle to a File Object is closed, Windows always issues a Cleanup operation. What makes the processing of CloseHandle operations associated with Completion Ports and Thread Pools special is that after the Cleanup operation has completed, the I/O Subsystem issues a Cancel operation for any I/O operations that remain in progress from the File Object. This results (or, rather, “should result” if the relevant drivers are properly doing their jobs) in any outstanding I/O operations that were initiated via the File Object being quickly terminated. Thus, the Completion Port or Thread Pool is promptly notified of the termination of any pending I/Os.
[/infopane]
To summarize: When an application calls CloseHandle, your driver will be called at its EvtFileCleanup (NOT EvtFileClose, as is commonly thought) Event Processing Callback. WDF will then issue a Cancel for any Requests that are resident on a WDF Queue.
When the last reference to a File Object goes away (that is, the File Object’s reference count is decremented to zero) Windows issues a Close request and WDF calls the associated driver’s EvtFileClose Event Processing Callback. The reference count on the File Object being zero means (for the purposes of this article) that there are no I/O operations in progress or pending for the File Object.
If there are no I/O requests outstanding when CloseHandle is called, EvtFileClose will be called immediately after EvtFileCleanup. Your driver’s EvtFileClose Event Processing Callback will be called at IRQL PASSIVE_LEVEL, and in an arbitrary process and thread context. Thus, when you driver’s EvtFileClose is called you do not know which thread is “current thread” or which process is “current process”.
To summarize: When the File Object reference count goes to zero, signaling that there are no further I/O requests pending for the File Object, your driver will be called at its EvtFileClose Event Processing Callback.
Our final definition is for Cancel. When a thread exits with pending I/O, or an application explicitly calls the Win32 function CancelIo or CancelIoEx, the pending I/O requests are Canceled. This results in WDF calling one of the driver’s Cancel-related Event Processing Callbacks, if one has been specified, for each Request to be canceled.
Unlike Cleanup and Close, in which the operation takes place based on a specific File Object, Cancel operations are managed in Windows on an individual, specific, Request basis. A great feature of WDF is that it is designed to automatically assist your driver with Cancel processing. The chief way it does this is that when Windows Cancels a given Request, WDF will search all your driver’s WDF Queues for that Request. If it finds the Request that’s being Canceled on a WDF Queue, WDF will remove the Request from the Queue and, by default, complete the Request with STATUS_CANCELLED and an information field of zero. This even applies to Requests that are on a Queue waiting to be presented for the first time to your driver (that is, Requests that your driver has not seen yet).
Consider how this helps driver processing in our second example, in which an application exits with multiple Requests in progress. In this case, as the application exits, Windows will issue a Cancel operation for each I/O request that is still in progress when the application attempts to exit. When WDF receives this Cancel operation, it will search your driver’s WDF Queues for the Request being Canceled, and if it finds the Request, it will remove it from the Queue and complete it (with STATUS_CANCELLED). If the Canceled Request was waiting on your driver’s Queue, and had not yet been presented to your driver for processing, the Request is removed from the Queue and completed without your driver ever having to deal with it. Once again, WDF makes driver writing easier, through its principle of “reasonable defaults.”
Of course, you might not want Requests that you’re holding on a Queue to be automatically removed from the Queue and completed by WDF during Cancel processing. For example, this might be the case when your driver holds Requests on a Queue with Manual Dispatching while it’s waiting for a device to finish its work. In this case, your driver can elect to handle Cancel processing itself for that particular Queue by registering an EvtIoCanceledOnQueue Event Processing Callback for the Queue. If your driver has provided an EvtIoCanceledOnQueue Event Processing Callback for a Queue, whenever a Request on that Queue is Canceled, WDF will remove the Request from the Queue and call your driver’s EvtIoCanceledOnQueue Event Processing Callback with the handle to the Request being Canceled.
If your driver holds Requests in-progress, but not on a Queue, it can register an EvtRequestCancel Event Processing Callback for a specific Request. For example, if your driver starts a Request on your device and you’re waiting for the device to complete its processing, your driver might simply save the handle to the pending Request in a location in your Device Context. This is, in fact, very common. If you want to be able to handle Cancel events for that Request while it is pending, waiting for your device, you can optionally register an EvtRequestCancel Event Processing Callback for that Request.
To Summarize: Whenever an application begins its exit processing with I/O in progress, or an application calls CancelIo or CancelIoEx, your driver may be called at its EvtRequestCancel or EvtIoCancledOnQueue Event Processing Callback.
A handy summary of Cleanup, Close, and Cancel processing is shown in table form in Figure 2.
Given the above definitions, let’s examine how your driver can use the Cleanup, Close and Cancel events to its best advantage.
When Should You Handle Cleanup?
When WDF gets a Cleanup notification for a particular File Object, it calls your driver’s EvtFileCleanup Event Processing Callback if one has been specified. EvtFileCleanup has the following prototype:
VOID EvtFileCleanup( _In_ WDFFILEOBJECT FileObject)
Should your WDF driver implement Cleanup? If so, what should you do as part of Cleanup processing?
The answer is simple for most drivers: In almost all cases you do not need to handle Cleanup – You therefore should not provide an EvtFileCleanup Event Processing Callback in your driver.
Cleanup operations in Windows are designed for basic two purposes:
1) As a quick notification to the driver that the calling application does not intend to issue any further I/O requests on the associated handle (remember, Cleanup happens on a per File Object basis);
2) To provide the driver a “last chance” to perform any operations that need to be done in the context of the requesting application (as the application is indicating its intention to stop interacting with the driver via this specific open instance).
As part of item 1, above, Cleanup can be used as a wholesale method of discarding a bunch of Requests that probably no longer matter to the application that issued them. WDF facilitates this by performing a Cancel operation on any Requests that were issued on the File Handle being closed and that are on a WDF Queue when the Cleanup is processed. The primary result of this is that any Requests from the Cleaned-up File Object that were on a Queue awaiting presentation to your driver will never reach your driver. WDF will remove them from the Queue and Cancel them, without your driver having to do anything.
Item number 2, above, is slightly more interesting. Because EvtFileCleanup is called in the context of the process/thread that called CloseHandle, this Callback gives your driver the opportunity to tear down any context that it was maintaining on a per-process basis. For example, if your driver mapped some memory into the application’s virtual address space, the EvtFileCleanup Event Processing Callback is the perfect place to undo that mapping. This type of operation isn’t common. But it is one of the key uses of Cleanup in WDF. And, yes, before the expert readers raise an uproar, let me add that there’s more to mapping and unmapping memory into an application’s virtual address space than simply unmapping in EvtFileCleanup. I’m not attempting to address that here.
It’s important to note that there’s also a nasty little detail inherent in Cleanup handling: Since handles are per-process entities, one thread in a process can be sending I/O requests on a handle (repeatedly calling ReadFile, for example), while another thread simultaneously calls CloseHandle on that same handle. If you think about this, you’ll realize that this means that it’s possible for your driver to receive one or more new Requests on a given handle, even after your driver has been called at its EvtFileCleanup Event Processing Callback for that handle. This makes for a nasty race condition, and contributes to making Cleanup the wrong place to definitively purge Requests from your driver to avoid, for example, an application hanging on exit.
Thus, in general, WDF drivers do not need to – and in fact should not – directly handle Cleanup. As I always say “If you don’t write any code, you don’t have any bugs.” So, when you don’t need to implement a function, that’s usually a good thing.
When Should You Implement Close Handling?
When WDF gets a Close notification for a particular File Object, it calls your driver’s EvtFileClose Event Processing Callback if one has been specified. EvtFileClose has the following prototype:
VOID EvtFileClose( _In_ WDFFILEOBJECT FileObject)
Should your WDF driver implement Close? If so, what should you do as part of Close processing?
As with Cleanup, for most drivers the answer is simple: For most drivers, there’s no need to handle Close, and you should therefore not provide an EvtFileClose Event Processing Callback in your driver.
Close is your last notification that a File Object is going away. When EvtFileClose is called, there are no more Request pending on the File Object. As soon as you return from your EvtFileClose routine (if you provide one) the associated File Object will be destroyed.
So, what’s the purpose of Close? The primary purpose of Close is to allow your driver a “last chance” to tear down any handle-related context it might have been maintaining. For example, suppose that during Open processing (called at CreateFile time) your driver allocates some data structures that it uses for tracking various state and information. Or maybe your driver tracks the number of open instances of its devices. In both these cases, EvtFileClose provides you the perfect opportunity to reverse those operations. To return the handle-related context, for example. Or to decrement the count of open instances.
So, in general, WDF drivers do not need to – and in fact should not – directly handle Close. The only time you need to do so is when you have per-handle context that you need to tear-down. In that case, EvtFileClose is the right place to handle this tear-down.
How Should You Deal with Cancel?
Remember that, unlike Cleanup and Close, Cancel notification is implemented on a per-Request basis. As we said earlier, when a thread exits with I/O in progress, or an application explicitly requests one or more I/O operations to be terminated before they are completed, Windows issues a Cancel for specific Requests, one at a time.
The purpose of Cancel is to tell a driver that it should stop processing a given I/O operation as soon as it is convenient for the driver to do so, because the results of that I/O operation are no longer relevant. As we described earlier, the most common use of Cancel is as part of thread and process exit. When a thread exits, Windows will Cancel any uncompleted I/O requests from that thread. This is referred to as I/O rundown processing. I/O rundown is important because, as we noted earlier, Windows will not allow the process to exit completely until its I/O operations are completed. This prevents a driver from unintentionally corrupting memory merely by processing an otherwise valid Request.
Should your WDF driver directly handle Cancel? And, if so, how should it handle it?
In general, if your driver does any asynchronous Request processing – that is, it does not immediately complete every Request it receives within its associated EvtIo Event Processing Callback – it should implement Cancel. If your driver leaves one or more Requests in-progress, such as while waiting for a device to perform a particular function, your driver must implement Cancel.
The reason for these rules should be clear: Because Windows will not allow a process to exit with in-progress I/O, the process will be made to wait for any I/O requests to complete before it can exit. If you don’t implement Cancel, the process will hang until all its I/O operations have completed in their ordinary course way. If you’re absolutely certain that every Request you start in your driver will complete quickly, you’re probably OK not handling Cancel (we’ll talk more about this in the next section). On the other hand, if a Request can remain in progress for a long, or even an unknown, amount of time, you must implement Cancel to terminate that Request and allow the issuing thread/process to exit.
WDF provides some help by automatically handling Cancel for Requests that happen to be on a WDF Queue when they are Canceled. WDF removes the Request from the Queue and calls your driver’s EvtIoCanceledOnQueue Event Processing Callback associated with the Queue if one has been supplied. The prototype for this function is:
VOID EvtIoCanceledOnQueue( _In_ WDFQUEUE Queue, _In_ WDFREQUEST Request)
When this Callback is called, the indicated Request has already been removed from the indicated Queue. How your driver deals with this Request is up to you. However, you cannot return the Request to a Queue – you must complete it, either within your EvtIoCanceledOnQueue Event Processing Callback or later. The general intention of EvtIoCanceledOnQueue is to allow your driver to stop whatever hardware operation that may be in progress as a result of the Request, and then have your driver complete the Request with STATUS_CANCELLED and an information field of zero.
If a EvtIoCanceledOnQueue Event Processing Callback has not been supplied, WDF simply removes the request from the Queue and completes it with STATUS_CANCELLED and an information field of zero. As we also described earlier, one of the reasons this is so helpful is that it automatically dispenses with Requests that are Queued but not yet presented to your driver.
If your driver holds one or more Requests in progress, but not on a Queue, then your driver will need to specify a EvtRequestCancel Event Processing Callback for each of these Requests. Your driver does this by calling WdfRequestMarkCancelable, specifying a Request handle and pointer to an EvtRequestCancel Event Processing Callback implemented by your driver. The EvtRequestCancel Callback has the following prototype:
VOID EvtRequestCancel( _In_ WDFREQUEST Request)
Again, the goal for your driver in this Callback is to stop any device operations associated with the Request that happen to be in progress, and complete the Request with STATUS_CANCELLED and an information field of zero.
Comments on Cancel: The Real World
While I’ve tried to make the general goals and processes for handling Cancel in your driver clear, I hope it’s also evident that handling Cancel can sometimes be difficult. And, sometimes, it can be very, very, difficult. For instance, it’s often not exactly easy to stop a device that’s in the midst of processing an I/O operation. Or, worse, it’s often not easy to stop a device from processing one single, specific, I/O operation when it’s in the midst of processing several (hundred or thousand) I/O operations simultaneously.
Thus, as alluded to earlier, in some cases it’s probably best to not implement any Cancel handling at all for in-progress I/O requests. Be aware that this is only a reasonable strategy when you’re certain that all your in-progress I/O requests will complete in a timely manner. Yes, when you’re notified of a Cancel operation there’s a thread and/or process that’s waiting to exit. But if it takes another 10ms or 20ms to complete a DMA transfer, and then complete the pending Request normally, that amount of time delay isn’t going to matter to the thread or process. And that’s a whole lot easier than trying to cancel the in-progress operation. Trust me. I have the scars to prove this.
There’s also another detail you need to know: Unless you’re doing so from within your EvtRequestCancel Event Processing Callback, you must never call WdfRequestComplete (or any variant therefore) with a Request that’s been marked as Cancelable. This means that if you associate a Cancel routine with a Request, and you complete that Request as part of your regular processing (that is, you don’t complete it as part of Cancel processing) you must first call WdfRequestUnmarkCancelable.
Unfortunately, Cancel handling remains one of the most complex parts of I/O operations. While WDF provides you with some help by canceling Request that reside on a Queue and have yet to be presented to your driver, it doesn’t do much to help you with Requests that have been presented to your driver and are in progress when they’re canceled.
Back to Those Examples
To summarize what we’ve discussed, let’s return again to the examples we used at the start of this article. In our first example, an application sends 100 asynchronous Read Requests to your driver, and then calls CloseHandle, while some of those Request are still in progress. Your driver uses a single Queue with Sequential Dispatching to service these Requests.
Now, you know what happens.
As soon as the application calls CloseHandle, Windows will immediately issue a Cleanup. If your driver needed to handle any state specific to the process that was doing the CloseHandle operation, it would provide an EvtFileCleanup Event Processing Callback. From within this Callback, it would tear down the process specific state. Whether or not your driver provided an EvtFileCleanup Event Processing Callback, WDF will Cancel any Requests that are pending on your driver’s Queue.
Assuming your driver does not provide a EvtIoCanceledOnQueue Event Processing Callback for your top-edge Queue (a safe assumption), WDF will remove from the Queue and terminate any pending Requests that were issued on the handle. Thus, the vast majority of Requests that were in progress will be terminated automatically by WDF.
If you driver has a Request (and, it can only be one… the example uses Sequential Dispatching) in progress when the application calls CloseHandle, unless that Request is on a Queue when the Cleanup is received, that Request will remain in progress in your driver. Either your driver will complete it as part of its ordinary processing, or the Request will be canceled when the issuing thread exits. Either way, there’s no problem. And, whenever the last I/O operation is completed on the File Object, your driver’s EvtFileClose Event Processing Callback will be called, if you specified one.
So, that’s what happens in our first example.
In our second example, the application issued the same 100 reads, but instead of closing the handle it exited or was terminated.
In this case, Windows first undertakes I/O rundown processing for the indicated thread. Windows will issue a Cancel operation for each of the in-progress Requests. Thus, as described previously, assuming you have not specified an EvtIoCanceledOnQueue Event Processing Callback for your top-edge, WDF will remove from the Queue and terminate each Canceled Request.
If your driver is holding a Request in progress, and the Request is not guaranteed to complete quickly, you will have specified a per-request EvtRequestCancel Event Processing Callback by calling WdfRequestMarkCancelable. Within that callback, your driver will complete the Request with STATUS_CANCELLED.
Of course, if your driver is holding a Request in progress, and the Request is absolutely guaranteed to complete quickly, your driver need do nothing more.
Thus, at the end of Cancel handling or very shortly thereafter, there will be no more I/O Requests in progress from the application to your driver.
When all the pending Requests have been completed, Windows will issue a Cleanup (and WDF will call your driver’s EvtFileCleanup Event Processing Callback if one has been specified) and then a Close (and WDF will call your driver’s EvtFIleClose Event Processing Callback if one has been specified). Unless your driver needs to tear-down some per-process specific state, it won’t have specified a Cleanup Callback. And unless your driver needs to tear-down per-handle specific state, it won’t have provided a Close Callback.
Now You Know
Now you know the meaning of Cleanup, Close, and Cancel in Windows. You know how WDF helps you handle these operations. And you know, at least in general terms, what your driver needs to do to handle each of them.
Hopefully you’ll never again be confused by these three similar sounding operations!
OSR Open Systems Resources, Inc.