Moving NTDEV, NTFSD and WINDBG — What’s Next

Community Members: This is the first in a short series of preparatory articles about moving OSR’s peer support forums, NTDEV, NTFSD, and WINDBG from their long-time home to a new and thoroughly modern platform.  In this post, we discuss the history of the lists, our search for a new home, and the features we’ll be supporting in future.

If you’ve been reading the OSR Discussion Lists over the past year or so, you know that we’ve been working on a long-term project to migrate the Lists to a new home.

List History — How We Got Here

Back in early history, NTDEV and NTFSD were founded by Atria Software as listserv mailing lists.  Artria were the folks who brought the ClearCase version control system to the world, which was the very first non-Microsoft developed NT file system. The Lists grew, as did Atria’s business.  At the same time OSR’s dedication to working with the community became evident.  As a result, Atria offered stewardship of the Lists to OSR.

When we brought the Lists in-house here at OSR, we hosted them using Lyris ListManager.  This was a commercial super-efficient discussion list email system, and we hosted it on a dedicated server in OSR’s server room.  We got a T1 connection (blazingly fast at the time, at 1.544Mbps… and more than a little expensive at more than $3K/month) to support the lists.

Several years later, OSR Online was invented and as part of our initiative to create “a home page for driver developers” we custom crafted the forum-style front end interface to the Lists that you see today.  We licensed the “look and feel” and layout code from vBulletin, which — at the time — was the standard for forum interfaces.  The lists were all modern and fancy. We were very proud.  We leased a second T1 and bonded it to the first yielding 3Mbps throughput (WOW!), which we primarily dedicated to the Lists.  And we added a WINDBG list to the venerable NTDEV and NTFSD.

Fast forward about 15 years, to 2018.  We’re still hosting the Lists locally on Lyris, in the OSR Server Room.  The Lists (and OSR Online) are the only remaining services we host locally, having moved the rest of our operations To The Cloud.  The interface that seemed to us so exciting years ago looks old and clunky and (quite frankly) depressing and embarassing, and the server software (Lyris, SQL, and the rest) is tired, out of date, and no longer supportable.

It’s time to migrate the lists to a new home.

The Search For a New Platform

One thing that hasn’t changed in all these years is OSR’s dedication to working with the community.   Sure, we could have made life simple for ourselves and just moved the VM that hosts Lyris to a cloud service provider.  But we wanted to bring the Lists fully into the 21st century (even if we can’t do the same with actual Windows driver or file system development, but that’s whole other article, really).  So, we undertook a search that took more than two years and entailed our doing tests using almost a dozen potential providers.

A key feature of the Lists has been that community members can participate using email, the forum interface, or a News Reader (NNTP).  Some of you probably haven’t even heard of NNTP, but you can Google it if you care.  You can still use all three of these modes to access the lists today, and people do.  While most interaction with the Lists is via the Forum interface, about 25% of the people (including some of our most valued members) participate via email, and a dedicated group (including some folks here at OSR) still use NNTP.

During our search, one thing became clear very quickly:  Our needs were unique.  There is no modern package or hosting provider who could offer us email, forum, and NNTP access.  Well, that’s not strictly true.  There are providers who have ugly, barely usable, forum interfaces to email discussions lists (think Mailman V2).  And there are providers who have reasonably nice forum interfaces and an email system, but they either can’t deal with the volume of members or email that the Lists generate, or they’re run by “one guy in his basement” and can’t be counted-on to be a long term, reliable, home for the Lists.  One of the contenders was actually a company with one employee (a self-described “serial entrepreneur”), who made it very clear (through his lack of attention and support) that he didn’t much care if we chose to pay him to use his service or not.  So we chose “not.”

For a long list of reasons, we also quickly discovered that continuing to offer List access via NNTP was not going to be feasible.  We’re sad that we can’t maintain the long tradition of providing this choice for List access.  But, time marches on!

The search took at least two years, but we’ve finally found our new home.

Our New Home

I won’t point you to our new site yet, or even tell you who the hosting provider is, because we’re not ready to accommodate you there.  But it looks a bit like this (this is a sample, from a trial migration we performed some weeks ago, and doesn’t represent the exact look/feel of the lists… but it’s close):

I can tell you a few key facts about our move (and I’ll repeat and perhaps refine these in a later post, as we get closer to the actual move).  Some basic facts, in random order:

• We plan to migrate in one “big bang”… we will not maintain using the old and new Lists in parallel.
• We’re only moving NTDEV, NTFSD, and WINDBG — We’re not bringing NTTALK with us, thank you very much.
• When we do the move, we’ll move all the history (threads and posts) starting from 1 January 2010 to the present.  We will not move any history before that starting from the beginning of time.  The Community talked us out of our initial thought to move only part of the history.
• We will keep the existing List forum site up in “read only mode” for some time — The actual shutdown of the forum interface is tied to the future decommissioning of OSR Online… and we don’t have a concrete plan for this at the present time.
• When we move, we will no longer support NNTP access.  Like I said, there are a lot of reasons.

Here are some more facts:

The new Lists will be fully modern in every way.  You can post in HTML, you can post images, you can have an avatar, you can have a signature, you can use markdown, you can post code segments that get syntax highlighted.  You can even edit your posts (for a period of time).  You can subscribe to threads that interest you, so you don’t have to keep checking-back to see if you’ve gotten an answer to a topic.  The mods can pin topics to the top of the page.

We will migrate every user who has a valid account on any one of the Lists that is being moved. This includes migrating their email address and their existing usernames.  Usernames that have spaces in them (such as “Peter Viscarola (OSR)” or “Tim Roberts”) will have the spaces replaced with underscores (so the previous names will appear as “Peter_Viscarola_(OSR)” and “Tim_Roberts”).  Note that in the picture above, we didn’t do this substitution (that’s why you don’t see it).  There are multiple reasons why we’re doing the “substitute spaces with underscores” thing… but one reason is that the new forum supports (the now traditional practice of) @ mentions.  So, you can refer to @Tim_Roberts and Tim can, somehow, see that.  I guess.

Users will have to reset their passwords after the move.  Every user who wants to access the Lists will need to go to the new site, and go through the password reset procedure.  This entails you the usual getting an email, clicking a link, and specifying a new password.

Any user may post to the List, and receive replies to their posts, via email.  As of today, our goal is to keep the email addresses for the Lists the same that they are now.  We’ll see if we can do that.

Any user can subscribe to a given thread, to be notified of posts to that thread.  Again, the new system is like any other modern forum that you’ve used.

Any user who belongs to one List, will belong to all Lists.  A very annoying feature of Lyris, which is a holdover from its listserv roots, is that you have to actually join each list.  And, you can have a different password and username on each list.  We’ve tried to hide that as much as possible behind the OSR Online membership system (ugh, another topic for another day).  Thankfully, we’re getting rid of all that nonsense.  In the new world, once you join you’ll be able to post to NTDEV, NTDEV, or WINDBG.

The Lists will be a “forum first” interface.  That means the primary mechanism for engaging with the Lists will be via the interactive forum interface.  There will be specific features that are designed to only work through the web interface and that are not available via email.  You may have to occasionally visit the forum with your browser to do thing (like, I don’t know… set your password initially).

The Lists will no longer support the old listserv commands.  #SorryNotSorry

The Lists will no longer reveal users email addresses.  Lyris was born at a time when everybody was happy to share everything.  Now we have lots of security threats, restrictions, and stuff like the GDPR.  As a result, we will no longer be revealing the email addresses of members unless they affirmatively choose to share it.  We will also no longer reveal members email domains (as we do today via the forum interface)… though we are open to the possibility of changing this decision after the migration has been completed, based on user feedback.

Initially, only a subset of users will be able to receive every post and every reply to a given List via email.  Because of technical limitations on the hosting platform, it is not possible for us to continue sending every post that’s made to every List to every user who’s ever subscribed as an email member of one of the Lists.  That’s thousands of members.  We tried, but we just can’t do it. And, after looking into it, it doesn’t even make sense.  There is at least one List member who I know to be dead, and who still has an active email subscription to NTDEV.  I don’t think he’s still reading the List (though I can’t be certain).  Again, to be clear: Anybody can post via email.  Anybody that posts interactively or via email can “subscribe” to a given thread and see all the replies to that thread.  But NOT everybody will be allowed to sign-up to arbitrarily receive every post made to one of the Lists.  This privilege will initially be reserved to a subset of users.

That subset of users who CAN receive every post to a given list via email will automatically include all those who have recently participated via email, and others by request. What we’re doing is creating two groups of members: The default group, that cannot elect to receive every post via email, and another group who can elect to receive every post via email.  This latter group will get email from all the Lists by default, and can elect to NOT receive email from one or more lists if they wish.  I haven’t actually finalized the criteria yet for who gets put into the email-enabled category by default yet.  But whatever that criteria is, we’ll be able to manually override it and add people who want to get email posts.  So, don’t freak out.

We will eventually enable all users to receive every post and every reply via email, if they choose.  This really does have to do with a technical limitation of our hosting provider.  We expect to extend every member the ability to receive every post (new thread and reply) to every list within the next year.  That’s an expectation, though, not a promise.

Some things will get screwed-up, and we will try to fix those things.  It’s inevitable.  We’re dealing with tens of thousands of posts, thousands of users, a pile of ancient technology, and well-meaning but imperfect engineers.  If it’s broken, we know you’ll let us know.

The move will happen in August.  Yes, of 2018.  We’re pretty sure.

Let us know if you have questions or concerns. You can post to NTDEV, NTFSD, or WINDBG.  But it’s better if you write to one of us here at OSR directly.

We’ll try as hard as we can to keep you informed and involved.  Believe me, there are more fun things for me to do with my Saturday morning than writing this blog post. But here at OSR we’re really and truly dedicated to making the lists the best place for a dev to find answers to their Windows system software questions.  So, let us hear from you.

Thanks in advance for your patience.  We’re going to make the move as painless as possible.  The folks who are doing the migration are service professionals who’ve done many hundreds of these migrations in the past… even migrations from listserv-type lists like ours.  We trust them.  But we know that at times your patience will be required.  And we thank you in advance for that patience.  And remember:  We’re always here to help.