We’ve finally started running our new and improved Developing File System Minifilters for Windows seminar! Of course, this means that I’ve spent a considerable amount of time musing about Windows file systems, file system filters, and the Filter Manager framework. While this isn’t exactly unusual, recently my thoughts have revolved around answering a deceivingly difficult question: how do we teach a developer to be a successful file system filter driver writer in 2019?
To answer this question, I decided that the beginning was the best place to start. Architecturally, what is a Windows file system filter anyway? Also, Filter Manager wasn’t released until the Windows Vista/XP SP2 timeframe. Why? Clearly folks were writing file system filters before its release, so why introduce a new model? What problems does it solve? And, more importantly, what problems does it not solve?
In Windows, we use a layered, packet based I/O model. Each I/O request is represented by a unique I/O Request Packet (IRP), which is sufficient to fully describe any I/O request in the system. I/O requests are initially presented to the top of a Device Stack, which is a set of attached Device Objects. The I/O requests then flow down the Device Stack, being passed from driver to driver until the I/O request is completed.
If the I/O request reaches the bottom of a Device Stack, the driver at the bottom may choose to pass the request on to the top of another Device Stack. For example, if an application attempts to read data from a file, the I/O request is initially presented to the top of the file system stack. If the request reaches the bottom of the file system stack, the I/O request may then be passed on to the top of the volume stack. At the bottom of the volume stack, the I/O request may then be passed on to the top of the disk stack. At the bottom of the disk stack, the I/O request may then be passed on to the top of the storage adapter stack.
Figure 1 is a representation of this processing if all these Device Stacks had a single Device Object.
An important thing to note is that the I/O requests are passed using a call through model. This means that, using Figure 1 as our example, NTFS passes the request along by calling directly into the Volume driver. The Volume driver then directly calls into the Disk driver. The Disk driver then passes the request by calling directly into the Storage Adapter driver. Depending on the number of Device Objects and number of Device Stacks involved, this can create significantly long call chains (as we’ll see in a moment).
An awesome feature of the I/O Manager in Windows is that each individual Device Stack may contain one or more filter Device Objects. By attaching a filter Device Object to a Device Stack, a filter driver writer may intercept I/O requests as they pass through the Device Stack. For example, a filter in the file system stack intercepts file level operations before (“pre-“) the file system driver has a chance to process them. Likewise, a filter in the volume stack sees volume level operations before the volume driver has a chance to process them.
Filter drivers are also given the opportunity to intercept operations after (“post-“) the lower drivers process the request. For example, a file system filter driver could intercept the data read from a file by opting to process the I/O request after the file system has read the file data.
Windows ships with many filter drivers in these and related stacks. Figure 2 shows a complete picture of the filters provided with a clean Windows 10 1703 installation.
The number of filter drivers present on a default installation of Windows amplifies the issue with the call through model. In order to get an I/O request to the storage adapter, we need to create a call chain that includes eight different drivers!
Boilerplate Legacy File System Filter
Given our discussion thus far, let’s talk about how someone would write a file system filter driver in this model. Along the way we’ll highlight some different challenges we might face just in our interactions with the system, let alone in whatever other value add processing we might try to provide (e.g. A/V scanning).
The first goal is we’re going to need to get a filter Device Object attached to the file system stack. Once our filter is attached, we’ll see all the I/O requests coming from the application before (“pre-”) the file system (Figure 3).
As mentioned previously, the file system filter may also see the operations after (“post-“) the file system’s processing.
There are several problems that quickly happen when you talk about putting a filter in the file system stack. We’ll highlight some of these here.
There is only one file system filter in this figure. However, if there were multiple there is no architecturally defined order for how multiple filters in the stack are layered. The OSR Filter might be above another other filters in the file system stack, or it might be below the other filters. The best we can do is try to choose the right load order group for our filter and hope that works well enough.
Dynamic Loading and Unloading
Filter drivers can only be dynamically attached to the top of the stack. For example, I could not take a running system and insert another filter driver between the OSR Filter and NTFS. Additionally, filter drivers cannot safely, dynamically detach from an active Device Stack.
The only way to properly layer the filter or detach the filter (and thus unload your driver) is by destroying the Device Stack and rebuilding it with the filter in place. In the case of your system volume, this means instantiating anywhere but at the top of the device stack or unloading your filter driver requires a reboot.
Mechanics of Attachment and Detachment
HOW the file system stack instantiates itself and tears itself down is a deep rat hole filled with weird edge cases. Just managing the attachment and detachment of the filter to the stack requires a significant amount of arcane knowledge that no one should need to know (including special case handling for when you eject a floppy diskette and then re-insert it!).
Propagating I/O Request Packets (IRPs)
Once a filter Device Object is attached to the Device Stack, it is responsible for processing all I/O Request Packets sent to the Device Stack. Even if a filter driver is only interested in processing read requests, it still must intercept all requests supported by the lower drivers in the Device Stack.
Trouble with Fast I/O Data Operations
As mentioned earlier, Windows uses a packet based I/O model. For example, each call to ReadFile in user mode creates a new, unique I/O Request Packet (IRP) to represent the I/O operation. The IRP is then passed around from driver to driver until the request is completed.
Long ago, someone made an interesting observation: lots of IRPs only make it as far as the file system before they are completed. This is particularly true in the case of cached file I/O. The file system simply copies the data out of the file system cache and returns it directly to the user. In these cases, all the work to build the IRP and pass it along was wasted.
Thus, the idea of “Fast I/O” was born. The file systems provide something called a Fast I/O Dispatch Table, which contains entry points to, for example, read data from a cached file. Instead of building the IRP, the I/O Manager bypasses the Device Stack entirely and calls directly into the file system to retrieve the data.
Now for the fun part: the I/O Manager always calls the Fast I/O Dispatch Table located at the top of the Device Stack. Once a filter driver is attached at the top, the filter driver receives the Fast I/O requests. If the filter driver does not register a Fast I/O Dispatch Table, then Fast I/O processing is disabled for the Device Stack and we lose the benefit.
Thus, it is the responsibility of the filter driver to “pass” Fast I/O requests down the stack by calling the lower driver’s Fast I/O Dispatch Table. There is no support for this processing provided by the system, thus every driver must invent and provide this code to not lose functionality in the system.
Trouble with Fast I/O Non-Data Operations
You would be foolish to think that the Fast I/O Dispatch Table only contains callbacks related to I/O!
To maintain a consistent locking hierarchy between the file system, Cache Manager, and Memory Manager, there are also some callbacks in the Fast I/O Dispatch Table related to acquiring locks in the file system. For example, the Memory Manager call’s the file system’s AcquireFileForNtCreateSection callback when a user attempts to memory map a file.
Unlike the Fast I/O callbacks related to data operations, the locking related callbacks are not sent to the top of the Device Stack. They are always sent directly to the file system.
Sounds great! Finally something we don’t need to deal with…However, a file system filter might actually be interested in these callbacks. There are many cases when a filter might want to know that an application is memory mapping a file. If the callback bypasses the file system filter, how do we get that notification?
On XP and later a mechanism was added: FsRtlRegisterFileSystemFilterCallbacks. This allows the filter to receive notification (and even fail) locking requests related to various Cache and Memory Manager activities. Note that these callbacks do not need to propagate execution to lower filter drivers, the operating system takes care of the layering for us. They also take an orthogonal set of arguments from the standard IRP and Fast I/O based callbacks.
Trouble with Recursion
File system operation is recursive by nature. For example, imagine an application sends a cached read request to the file system. This I/O request will flow down the file system Device Stack (as discussed) before reaching the file system.
The file system will then ask the Cache Manager to please copy the data from the cache into the user data buffer. But, what if the data is not in the cache? The Cache Manager must then perform a non-cached read of the file data to retrieve the data from disk. Of course, it does this by submitting a new read request to the top of the file system stack. Thus, a file system filter may pass a read request down the stack and, before it returns, may see another read request arriving from the same thread (Figure 4).
While recursive operation is a way of life in the file system stack, file system filters can amplify the problems of recursive processing.
First, remember that his is all done using a call through model. Thus the call stack here can be enormous. It was not at all uncommon to see stack overflow bugchecks when there were multiple legacy file system filters present.
Secondly, if a file system filter generates its own recursive I/O then the filter must be careful to not deadlock itself (or other filters). For example, imagine and anti-virus filter that does not want to allow an application to open the file before it has been scanned. If the file system filter generates recursive I/O operations to read the file, the A/V scanning filter will see its own read I/O requests arrive at the top of the device stack. It must be sure to allow those reads to occur, even though the file scan has not yet completed.
A Lot of Work to do Nothing
To give you a sense of scope, old versions of the Windows Driver Kit shipped with a “Simple” legacy file system filter driver. This filter did absolutely nothing except attach a filter Device Object, pass all IRP and Fast I/O requests without modification, and register for the file system filter callbacks. The total line count for this “do nothing” filter? 6,800!
Now Let’s Get to Work!
Now that we have almost 7,000 lines of boilerplate code, we can get to work! There are lots common things that almost every file system filter driver needs to do.
Contexts, Contexts, Contexts
One of the things that we will undoubtedly want to do is attach context to various different things. For example, we might want to create a context for the underlying file system and volume that we’re filtering. Are we filtering FAT? NTFS? Is the drive removable or fixed? Or are we filtering the network?
We might also want context associated with the files we are filtering. For example, in our write request handler how do we know which file is being written?
We might also want context associated with the individual streams of a file. For example, in our write request handler how do we know which stream of the file is being written?
We might also want context associated with the individual opens of a stream. For example, in our write request handler how do we which open of the stream was used to write the file?
Retrieving File Names
Of course, if you’re in a file system filter driver then you probably want to know the name of the file being read, written, renamed, deleted, etc.
Unfortunately, something that is seemingly so obviously necessary is a very involved and complicated task. There are unwritten rules on when you can and cannot query the names of a file for fear of a deadlock. The overhead of this activity can also be significant, especially when querying name information over the network.
Also, in the legacy filter model every filter that wants a name must perform its own name query. Thus, we have a very involved and complicated task being performed by every file system filter in the stack.
Communicating with User Mode
It is not uncommon for a file system filter to work in conjunction with a user mode service. The user mode service might be used simply to send control or configuration information to the driver, or to consume logging activity generated by the driver. In many cases, the user mode service can be used to offload complex processing that is much better suited to user mode development (e.g. binary analysis prior to execution).
A Ton of Work to do a Little
Because I love counting lines of code, we’ll look to the old FileSpy sample to get a sense of scope in the leap from “do nothing” to “do the common things you probably need.” The FileSpy sample creates file system and volume context, as well as per-stream contexts. It retrieved file names for individual file operations, and logs file system activity to a user mode console application.
The grand total for the filter only (i.e. not including user mode components): 16,200!
Clearly Needs Fixed…
So, there you have the state of file system filter development in the year 2000. The highlights include:
· No ability to dynamically load and unload the filter
· Non-deterministic behavior in terms of the layering of filters
· The simple act of attaching the Device Object to the stack is needlessly complicated
· Increased execution stack utilization for each and every filter added
· Filters intercept requests in three different ways: IRPs, Fast I/O, and File System Filter Callbacks
· When there are multiple filters loaded, there is a significant amount of duplicate work, especially around querying file names
· Filters that generate recursive I/O requests need to perform extra work to avoid blocking their own execution. Filters must also be prepared to handle recursive I/O from lower filters
· Filters that need to communicate with user mode must create all their own communication code
· A filter that does nothing takes 6,000 lines of code
· A filter that kind of does something takes 16,000 lines of code
This is all in addition to the fact that the Windows file system I/O interface is inherently complicated. Much of this is due to the long history of the interface and the many edge conditions that have been created over the years. As mentioned, there is a significant amount of recursive activity generated by the file system. During these recursive I/O operations, locks may be held by the lower file system, thus making it dangerous to attempt incompatible operations. The file systems are also deeply integrated with the Cache and Memory Manager subsystems in Windows, which have their own rules and own locking requirements.
Lastly, each Windows file system is different than the other by way of implementation. Thus, the behavior that you see while interacting with a particular file system may be architecturally similar but different in its observed behavior.
At this point, a group of very senior file system and kernel architects and developers at Microsoft (i.e. “very smart people”) decided this needed fixed. Not because they had nothing better to do, but because a large percentage of Windows crashes were being blamed on file system filter drivers. While you could try to blame the filter developers for this, the filtering model was objectively broken. File system filter drivers were clearly an afterthought in the original design of NT. It was time to sit down and architect and design a way to write a file system filter driver on Windows.
Enter Filter Manager!
And this is where Filter Manager and the Filter Manager Framework come in. The team behind Filter Manager wanted to make it easier and more reliable to write a file system filter driver on Windows. In doing so, there were two significant requirements that had to be met:
1) The “new” way to write a file system filter must be as flexible as the existing model. If there was a filter type that could be written the old way, but not in the new way, then the new way was a failure
2) The underlying file system architecture in Windows could not be radically changed. In other words, the goal of simplifying file system filter development could not be met by simplifying the file system interface
Thus, the ultimate solution to the problem was to create a framework for writing file system filter drivers. The framework provides the one legacy file system filter driver necessary in the system, and consumers of the framework plug in as “minifilters”. As I/O requests arrive at the Filter Manager legacy filter Device Object, Filter Manager calls the minifilters using a call out model. After each minifilter processes the request, Filter Manager then calls through to the next Device Object in the Device Stack (Figure 5).
However, Filter Manager does not simply pass the native operating system operations to the minifilters. Instead, Filter Manager creates its own abstractions of the native operations (we’ll see the benefit of this shortly).
What Filter Manager Does…
Filter Manager is great and does a lot of things for you. This includes but is not limited to the following.
Allow for Dynamic Load and Unload
Filter Manager itself is a legacy file system filter driver, thus it cannot dynamically load and unload. However, it provides support so that minifilters can dynamically load and unload (if they choose).
Deterministic Layering Behavior
Filter Manager introduces the concept of altitudes, where higher altitude filters are called before lower altitude filters for operations en route to the file system. Altitudes are groups by functionality (e.g. AntiVirus, Encryption, etc.) and Filter Manager ensures that the filters are called in the correct order based on their altitudes.
Simplifies Filter Attachment
Filter Manager still needs to deal with all those annoying details of how the file system stack instantiates and tears down. However, it does not expose the minifilter driver writer to all this non-sense. Instead, Filter Manager creates an abstraction called an Instance. An Instance is an instantiation of a minifilter within a file system stack (e.g., in Figure 5 there are three Instances). Instance setup and teardown follows a sane set of rules and requires minimal code (and no special case floppy code!).
As an added bonus, a single filter is even allowed to have multiple instances within the same stack! This has some useful applications, including breaking a complex filter up into multiple different instances. For example, in our File Encryption Solution Framework (FESF) we use separate instances for data transformation (i.e. encryption) and managing the on disk structure of our encrypted files.
Decreases Stack Utilization
Filter Manager’s use of a call out model to the minifilters significantly decreases the stack utilization of I/O operations. It also means that adding more filters does not instantly increase the stack utilization of every single I/O request.
Provides a Coherent Context Model
As mentioned previously, file system filter drivers undoubtedly need to attach context to things such as volumes, files, streams, and even File Objects (i.e., open instances of files or streams). Filter Manager provides a consistent context API for attaching and retrieving context for these objects. Filter Manager also provides additional support for attaching context to instances, transactions, and memory mappings.
Provides a Coherent Callback Model
Legacy file system filters needed to deal with IRPs, Fast I/O, and FsRtl filter callbacks to capture all possible file system operations. Filter Manager rationalizes all these different callbacks into a single, unified callback using a Callback Data structure. This Callback Data structure provides a consistent view of file system operations and is the one structure that minifilters need to understand to filter any file system operation.
Provides Name Query Support (with Cross-Instance Caching)
Filter Manager provides a consistent interface for querying file names for any given file system operation. In addition, Filter Manager understands all the situations in which it is not safe for a file system filter to query name information. So, instead of a difficult to diagnose deadlock in your filter, Filter Manager simply returns an error if it is currently unsafe for you to query name information.
Even better, Filter Manager caches the results of name query operations and shares the results amongst minifilter instances. Thus, if one filter queries the name for a given I/O request, other filters can benefit from the cached result.
Provides Support for Avoiding Recursive I/O
Filter Manager allows minifilters to target I/O requests at specific altitudes. Thus, a minifilter can choose to send an I/O request only to those minifilters that are at lower altitudes. This means that a minifilter does not need to write defensive code to detect its own recursive operations.
Provides Support for User/Kernel Communication
Given that the majority of file system filter drivers communicate with user mode, Filter Manager provides a nice Communication Port package for bi-directional communication. User mode applications can easily send messages to the minifilter, and the minifilter can easily send messages to user mode (with or without a response).
Significantly Decreases the Size and Complexity of Boilerplate Code
Clearly a major goal in the creation of Filter Manager was to reduce the amount of boilerplate code in a filter. If every filter needs complex attachment code, then Filter Manager should handle that. If every filter is going to perform name queries, then Filter Manager should handle that. If every filter is going to communicate with user mode, then Filter Manager should handle that.
What Filter Manager Does Not…
Filter Manager is so well considered and so well executed, that it almost makes writing filters easy! A simple sample is only ~900 lines of code (mostly comments) and there are lots of examples provided on GitHub. It’s all very approachable and, in fact, quite easy to get something demonstrably working in a short period of time.
Even with all that, writing filter drivers is still hard in 2019. You can see this from the level of questions that show up on NTFSD. Everyone thinks that their filter is pretty much working, but there’s one or two little problems to solve. Unfortunately, those problems are things like their filter doesn’t work with Notepad. Or the system hangs when they try to save an Excel document. We see it all the time, folks think they’re just a few weeks away from having a product and they go right off a cliff. Unfortunately, those one or two issues could easily take months to resolve.
So, what’s happening? Are the samples not good enough? Is the documentation not good enough? Or is it that Filter Manager failed in its goal to make file system filter drivers easier to write and maintain?
The answer to this question goes back to fundamental requirement #2 in the development of Filter Manager:
The underlying file system architecture in Windows could not be radically changed. In other words, the goal of simplifying file system filter development could not be met by simplifying the file system interface
Thus, the biggest hurdle to learning to write a file system filter in 2019 is the same hurdle you faced in 1999: understanding the complexity of the underlying file system interface. Sure, the mechanics of how you assemble a filter are important, but the devil is in the details of the underlying interface. This understanding greatly influenced the development of our minifilter seminar, where we spend lots of time talking about the why of the Filter Manager callbacks instead of just focusing on the how.
We’ve worked really hard to create a seminar with the right information, hopefully you’ll get a chance to join us and find out!