You might’ve noticed we had a serious outage that took down our website, forums—everything.
Some people have speculated on what could be so catastrophic as to take us down for WEEKS?
For legal reasons, I can't get into the details other than to say it wasn't ransomware but it was a total data loss. This catastrophe wiped out everything at our data center, including the on-site backups, so we lost over three decades of data in one hit.
Fortunately, we run nightly offsite backups, but they’re enormous—about 34 terabytes. That’s 34,000 gigabytes, all of which has to be downloaded, scanned, extracted, and then reuploaded to new servers. Just the download alone took over a week. Then came the challenge of figuring out which parts needed immediate restoring, in which order, and whether we should rebuild them piece by piece, create entirely new services, or move to a cloud-based infrastructure to avoid having a single colocation ever again.
We’re talking about a giant library of websites, databases, skins, themes, icons, wallpapers, videos, and more. Some of it’s ancient, from when we started well before Google or Facebook existed. Imagine sifting through tens of thousands of gigabytes to find a single legacy web service, built decades ago, that needs to run on a specific OS. It’s a painstaking process.
This outage has been extremely difficult. Everything from old box art for our products to OS/2 programs I wrote in my college days—gone, at least until offsite backups did their job. We had fallback backups on hard drives, DVDs, tapes, and so on, but for a while, it wasn’t entirely clear how much of that would be usable.
A fun fact some may not know: we have one of the oldest continuously used forums around, migrated from Usenet eons ago. That entire environment was wiped, so we’re rebuilding it from offsite storage. Not everything is back yet, and it looks like a few forum user accounts will be lost. That’s not related to customer data, but still worth noting.
We appreciate everyone’s patience. Getting services running again has been the top priority. It’s been a monumental effort, but we’re seeing real progress each day, and the community’s understanding means a lot.
Thanks for sticking with us,
-Brad (Founder & CEO)
