Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Windows 8 Preview: File System Changes

With the recent release of the Windows Developer Preview (i.e., Win8), I took the opportunity to review the changes that they’ve made public that will affect file system and file system filter driver developers for the Windows 8 Platform.

While one can certainly look at the documentation, my approach in preparing this article was to look at the changes in the critical header files: ntifs.h and fltKernel.h, as these suggest much of the current scope of changes.

With that said, my experience with previous pre-release versions of Windows is that everything is subject to change.  Thus, it is unwise to count on any feature or change that we describe in this article until the final release of Windows 8.

New File Systems

The header files suggest the presence of new file systems as part of the Windows 8 release.  This includes the Protogon file system (although there is no new Protogon file system that I could see in the Developer Preview).  The headers (Figure 1) indicate:

Speculation in the press is that this will be the database-as-filesystem model previously promoted as WinFS.  If so, I suspect it will be a substantially different implementation (likely with a strong emphasis on performance).  Then again, if it wasn’t far enough along to include in the Developer Preview, it might be one of those features that doesn’t make the initial release.

In addition, there is quite a bit of new information about something called CSVFS (perhaps the “cluster shared volume file system” to work with “cluster shared volumes”?  If so, this may just be exposing new information to an existing technology).  Again, it seems a bit premature to speculate, especially given that if it is related to clustering, it is not likely that this feature would be present in the client release.

Emphasis on Data Verification

Many of the changes we observed relate to data verification support; whether this is an NTFS only feature, or if it will be present in other file systems (e.g., Protogon).  There are two new FSCTLs (FSCTL_{GET,SET}_INTEGRITY_ INFORMATION) that appears to be related to these changes, and a structure used to control this information (as shown in Figure 2).

Frankly, this makes quite a bit of sense: modern disk drive technologies are vastly more complicated than they have been in the past and can (and do) suffer from real issues with respect to data integrity.  Thus, it is quite possible to read data back from the drive and receive something other than what was originally written.  While it doesn’t happen frequently, it does happen and most applications are not written to withstand that sort of data corruption. Some applications, particularly database applications, are highly susceptible to this type of data corruption – the relationship information between the components can be lost when critical database information is lost.  While transactional database models will protect against some forms of failure, they do not protect against incorrect data being returned from the underlying storage media.

Data Deduplication

Another intriguing hint from the header files are suggestions for supporting data deduplication techniques.  For example the following new FSCTL operations for data deduplication are present (see Figure 3).

Thus, this suggests using a scheme by which file hashes are being queried and data deduplication is being done for those blocks with identical hash values (for good deduplication hash algorithms, the likelihood of a data hash collision is very low).

Offload Support

The header files strongly hint at new support for “offloading” I/O operations.  While we are speculating a bit, if this is similar to other forms of offloading, it would suggest the use of hardware to perform some operations (e.g., I/O operations as well as computing hashing).  This might be used, for example, for intelligent disk drives to allow them to perform additional high level processing, such as is done in Object Storage Devices.  When combined into file systems, such devices can actually provide specialized support and can even split data and meta-data across multiple drives (local and remote).  Whether or not that is what is envisioned here is still uncertain (after all, this is just based upon header file information).

There are two new FSCTL operations for offload (Figure 4).

What is particularly important about this new support is that it is disabledif file system filter drivers do not support it, according to the comments within the header file itself (Figure 5).

Thus, it is important for those building file system filter drivers to be both aware of this new functionality and to ensure that they support it.  Otherwise, it runs the potential risk of breaking some other product’s functionality (or at least degrading performance) without supporting it.

Advanced FCB Header Changes

File system filter driver writers won’t notice this change as much as those of us building file systems, but the Windows 8 version of the advanced FCB header has changed.  The Advanced FCB header has gained a new field (Oplock) and the definition of FsRtlSetupAdvancedHeader (implemented in the header file) has changed to initialize the header properly.

The comment from the header file is fairly clear on this one (see Figure 6).

 

Note that this declares there to be a new FCB header version (2), set in the Version field of the common FCB header.

Oplocks

This new oplock field appears to tie into a new round of oplock support changes in Windows 8.  For example, we have several new support functions: FsRtlCheckLockForOplock Request and FsRtlAreThereWaitingFileLocks.  Interestingly, these new support routines have been optimized so that byte range locks on regions of the file outside the allocated range will not conflict with oplocks (recall that byte range locks and oplocks are traditionally inconsistent).

Logically, this makes sense.  The rationale for not allowing both is that applications using byte range locks want to obtain coherent copies of the data, which is incompatible with the basic premise of caching.  However, byte range locks on regions of the file where there is no data are used by applications as a means of inter-process communications (potentially on different computers) and not related to data coherency.  Thus, these routines should actually permit the use of oplocks in a broader range of situations.

New ECP Types

Another interesting change related to oplocks is the new support for extra create parameters (ECPs) for tracking oplock “ownership” state for not only the target of an open operation, but also the parent.  Note the definition of the new dual oplock key:

Presumably, this would be useful for rename operations (for example) and would allow operations to proceed without forcing oplock breaks (much like the GUID_ECP_OPLOCK_ KEY was used previously to keep from breaking oplocks with associated opens to the original oplock holder).

Beyond this, we also have a solution to the frustration of receiving back an error whenever a reparse occurs inside a call to IoCreateFileSpecifyDeviceObjectHint (or any function that it calls it, including the various Filter Manager APIs for opening files).  This error (STATUS_MOUNT_POINT_NOT _RESOLVED) is difficult to resolve cleanly in a file system filter driver.  Having done so (by iterating through the path until finding the reparse point, then opening it to query the contents of the reparse information) it is encouraging to find that there will be a solution to this problem for future releases of Windows.

Indeed, the comment from the header file fairly clearly describes the model for this new ECP:

Cache Manager

There are a number of Cache Manager changes present in the Windows 8 header file as well.  One that has me a bit mystified is the exposure of a new structure (Figure 7):

 

There is some interesting information in here, but the existing interface does not seem to provide any mechanism for us to obtain or directly modify these values.  Perhaps we will see something in a future release, or some clarification as to the ultimate use of this structure.

There are a number of additional new functions exported from the Cache Manager as well: CcCopyReadEx, CcScheduleReadAheadEx, and CcSetAdditionalCache AttributesEx.  These new functions introduce the concept of an “I/O issuer” – these new functions take an extra parameter, a PEPROCESS pointer.

Filter Manager Changes

There are a number of Filter Manager changes visible in the new header files as well.   Perhaps the most interesting is that there is now a section context – so that file system filter drivers can associate specific state with a section (in addition to the other context types already supported by the Filter Manager).  From the material available (including preliminary documentation) it appears that the purpose of this is to allow mini-filters to synchronize changes to section objects.  A number of new APIs have been introduced to support this, including FltGetSectionContext, FltSetSectionContext, FltRegisterForDataScan, FltCreateSectionForDataScan, and FltCloseSectionForDataScan.

The registration structures have changed to accommodate the new section contexts.  Note that section contexts are only present in Windows 8 and more recent.

Beginning with Windows 8, Filter Manager now supports filters for the named pipe file system (NPFS) as well as the mail slot file system (MSFS).  To filter these, mini-filters must indicate their interest as part of their registration information.   To further aid in supporting these, the Filter Manager now provides new functions for opening named pipes (FltCreateNamedPipeFile) and mail slots (FltCreate MailslotFile).  Presumably these are wrappers around the existing OS calls.

The Filter Manager APIs have been extended to support new features, such as the reparse point support (see GUID_ECP_FLT_CREATEFILE_TARGET).

The MDL interfaces are now also available as part of FltReadFileEx and FltWriteFileEx.  While not fundamental changes to the model, they should simplify development for those mini-filters that wish to use the capabilities of existing OS interfaces that have not previously been available via the Filter Manager interface directly.

There are also new Filter Manager APIs for invoking the fast I/O MDL operations directly, as well as Filter Manager wrappers around the get/set interface for quota information.

Finally, there is now support for retrieving multiple contexts simultaneously using the new FltGetContextsEx.  There is a corresponding “bulk release” operation in FltRelease ContextsEx.

Conclusions

There are other changes (notably in the security area) that we haven’t been able to cover, but it is clear that while there are a number of interesting new changes for file systems and filter drivers in Windows 8, they are more along the lines of evolutionary, rather than revolutionary.

Stay tuned as we watch Windows 8 evolve – after all, as we said before, none of this is known until the final version ships.

Summary
Article Name
Windows 8 Preview: File System Changes
Description
Here, OSR checks out potential changes in the Windows file system space prior to the release of Windows 8.
Author