If you’ve been reading The NT Insider for a while, you probably already know that here at OSR we’re keenly interested in all things to do with storage and file systems. Over the years, we’ve worked closely with both Microsoft and a number of storage vendors to help evolve these disciplines. So, when NVM Express started to evolve out of the previous NVMHCI working group that we had been a part of, we were eager to see the outcome.
When the first version of NVMe was released in 2011, we realized it was an important starting point. But we had seen cool technical specs come and go over the years, so we weren’t about to get overly excited. But by the time the NVMe V1.1 spec was approved in 2013, we had already realized this technology was going to be significant.
Clearly, Microsoft agreed with our assessment, because Windows 8.1 (which RTM’ed in August of 2013) had a fully functioning NVMe driver included.
The importance of Microsoft’s early support cannot be overstated. Microsoft has always been loath to jump on the bandwagon when new technology specifications are released. This is particularly true when it comes to storage. While those of us who like to live on the bleeding edge might bemoan this approach, it makes perfect sense: If support for some up-and-coming technology is included in Windows today, that support is going to have to remain in Windows for a good long while. And the technology has to be widely enough available, and sufficiently well tested, that it can be used by potentially millions of users world-wide. Say what you want about versions of Windows, but even the most ardent penguinite can’t complain about the extensive hardware compatibility support that Windows provides.
So, Microsoft’s support for NVMe in Windows 8.1 signaled that this technology was not only significant, but that it was here to stay.
It was about this time that we started hearing from colleagues and technology partners that they wanted us to provide them with an NVMe driver. Why did they want OSR to write an NVMe driver for them, when Windows was shipping with an NVMe driver as part of Windows 8.1? Well, it turns out there are a lot of reasons:
- Most users out in the world aren’t running Windows 8.1 – Even forward-thinking users who’ve done recent upgrades are running Windows 7 (or Windows Server 2008 R2, AKA S08 R2). Microsoft doesn’t provide an NVMe driver that’s compatible with older systems.
- The Windows driver doesn’t provide support for all the (optional) NVMe features that most hardware vendors and many large data centers need. NVMe might only have a few required commands, but there are a lot of really cool optional features. Folks want to be able to take advantage of these features. Today.
- The Windows driver can’t be customized (obviously) to support vendor-specific optimizations and behaviors. Sure, NVMe is a standard. But there wouldn’t be any competition if each developer of an NVMe device didn’t try to gain some advantage for their technology implementation over those of others. Tailoring a driver to the specifics of how a device is implemented is an excellent way to maximize a device’s potential. And, no surprise: Hardware vendors want to do this.
- In addition to the above, the Windows driver doesn’t support an NVMe pass-through capability, beyond that which can be mapped from a limited set of SCSI pass-through operations. NVMe pass-through is vital for those who want to perform unique commands, especially management and administration commands. In fact, we’ve never talked to a client interested in NVMe technology that didn’t think NVMe pass-through was vital.
- Not every NVMe device vendor or large data center wants to invest the time and money (mostly the time) to develop their own in-house NVMe driver.
- Even if they have a custom driver, most storage vendors or data center teams don’t want to maintain it. Unless driver development is part of their core skill set, they’d rather not be bothered.
- Folks told us that if OSR wrote it, they’d be confident that the driver would be both well designed and reasonably well performing. This made us happy.
About this same time, work was also progressing on an open source NVMe driver for Windows. Some of the early promoters of NVMe worked on this driver collaboratively as part of the Open Fabrics Alliance (OFA), prior to a Windows driver being available from Microsoft. Some colleagues, clients, and partners told us that they did not trust the stability of this driver sufficiently to use it in production. For an example of two very different public assessments of this driver, see the NTDEV thread at http://www.osronline.com/showthread.cfm?link=253628).
Our own review and testing of the open source OFA driver (which started in early 2014) concluded that the driver suffered from some unusual design quirks, was uneven in its implementation quality, and was in general not production ready. It also suffered from having undergone multiple revisions, and needed a general clean-up. Our ultimate conclusion was that, to produce the best possible driver, a clean start was necessary. We thus decided not to contribute to the OFA effort, or even use that code as the basis for our own work, but to rather undertake our own, unique, built from scratch, NVMe driver project.
That’s how we here at OSR made the decision to develop our own Windows NVMe driver. We took what we heard from vendors and data centers, and we brought all our knowledge of the Windows storage stack, I/O Subsystem architecture, and all our experience in StorPort driver development, to focus on the problem.
That driver is in internal test right now, and is scheduled to be released to external Beta Test on 12 January 2015.
The OSR NVMe driver seeks to meet the needs of those companies – NVMe device vendors or data centers – that don’t want to use either the Microsoft or the OFA driver. With our NVMe Driver Solution Kit, clients get a driver that works out of the box on Windows 7 (including S08 R2) and later systems. The kit is royalty free, and comes with full source code and the rights to make any modifications a client may desire. On request, we’re able to customize the driver, adding advanced NVMe features or custom features unique to a particular use. And, of course, we can provide on-going support, maintenance, and upgrades for the driver – including support for new OS versions – long into the future.
What’s been most gratifying is that our careful design resulted in the highest performance and lowest CPU utilization of any NVMe driver we’ve tested to date in our lab. We handily beat both the inbox Windows 8.1 driver and the OFA Windows driver in our tests.
So, what’s supported? At Beta release, the driver will support all the basic functionality in NVMe 1.1b and the NVMe SCSI Translation Reference. The driver will pass all Windows Hardware Certification Kit tests for the driver (that is, the tests that test functionality of the driver itself and not the actual NVMe hardware device being tested), including SCSI compatibility tests. The driver also supports MSI-X, and an almost entirely lockless model using a dedicated pair of submission and completion queues per processor.
The distribution contains both built (and signed) executable images, as well as full source code. The driver builds using the latest Windows 8.1 WDK.
By the Final V1.0 release of our driver, we currently are committed to support at least the following:
- Windows 7 (and Server 2008 R2) through Windows 10 Technology Preview
- All mandatory features specified in NVMe V1.1e
- All mandatory features in the NVMe SCSI Translation Reference
- Comprehensive NVMe pass-through capability, including namespace formatting via pass through.
- MSI and MSI-X. Where MSI-X is available, support for dedicated submission and completion queue pairs per CPU.
- Windows Hardware Certification Kit tests (relevant to drivers, not to an NVMe hardware device) all pass, including SCSI compatibility tests.
We’re taking a unique approach to development between the Beta and Final releases: Clients who are members of the Beta program will have the opportunity to influence the standard NVMe features that will be included in the 1.0 release, over and above those listed. We’re open to considering the addition of features described in any currently published version of the NVMe specification (even NVMe V1.2).
We’re also willing to customize the driver for a particular client’s needs. These customization can include specific interfaces, specialized hardware handling, or even inclusion of advanced NVMe features that we would not otherwise have time to include in our standard 1.0 release.
NVMe hardware vendors might choose a customized driver to ensure specific, advanced, features are supported that are advantageous to their device. Similarly, they may choose to customize algorithms, such as by specifying algorithms for read/write optimizations.
Data centers deploying NVMe might want us to customize the driver to provide features that are important in their environment (such as namespace hot-plug), or to ensure that diverse vendor hardware can be used via a uniform, standardized, interface.
It’s been a long an interesting journey for us, as we decided to undertake this project and started to make it happen. We’re excited to finally be working with clients as part of our beta program, and are looking forward to releasing our work to the world.
For more on OSR’s NVMe Solution Kit and the Beta program, visit www.osr.com/nvme-driver/ or contact our sales team (sales@osr.com) to describe your specific NVMe needs or to get your questions answered.
OSR Open Systems Resources, Inc.