On rotting software

Last night, I installed a PC game from my childhood.

My toddler keeps going through a big ol’ basket of CDs I have on the floor in my office, handing me various jewel cases she finds interesting. Naturally, a CD with a grinning, green Centipede on the cover is more appealing than my Gateway Computer edition of Microsoft Bob.

While Microsoft Bob, Encarta 95, and a slew of other precursors to everything done online nowadays…and absolutely littered with ads. don’t have much use in $CURRENT_YEAR, what made Centipede fun in 1998 is likely to be true in 2022.

This particular game was one of my favorites: Fast-paced and adrenal. A thumpin’ electronic redbook audio soundtrack accompanied you on your journey around the sprawling worlds in need of protection from alien centipedes, spiders, and fleas.

It installed without issue on a Windows 10 laptop. Configuration of a PC game in the 90s, however, was a bit more complex than the initial installation. Given the sheer number of competing graphics cards with different ideas about a) how they should be programmed and b) what sort of capabilities they had, the game’s programmers included a standalone utility that scans your graphics card and figures out which graphics modes your particular card can actually use. 640x480 at 16-bit color, or an eye-watering 1024x768 with 32-bit color? 25 years after it was programmed, without even having to use modern Window’s explicit “run this application like it’s 1995” mode, the game and its configuration utility both run. It plays perfectly–at 1280x1024x32, no less!

Easy old things, now difficult

As my trusty Dell XPS laptop doesn’t have an optical disc drive of any sort, I have to use a portable/USB CD drive to read the game’s CD. Which is annoying. But I can’t rip the CD to my hard drive and mount it on a virtual drive as I was unable to find any (free) software that understands how to mount a multi-part disc image. There’s 280M of data on the disk, but another ~300M of redbook audio Windows doesn’t know what to do with.

A virtual disc drive capability native to Windows 10 handily mounts a copy of the basic CD data (track 1 on the disc), but not any of the 16 audio tracks that come after it.⊕To be fair to Windows’s team, my favorite piracy ISO tools from 2002 [which are still being ported to modern Windows!] were similarly stumped by the multipart CD. The game will run with a virtual, audio-less CD in the “drive,” but the experience is sorely lacking without the Jon Holland (Vectorman)-esque gritty acid techno soundtrack.

Old difficult things, now easy

Lengthy aside: I was legitimately impressed that they used 110M of space on the CD for the Smacker-encoded animated videos on the disc. 110M/280M–what a budget!

Smacker, and its successor Bink, are old media formats created expressly to be used for FMV in video games. It turns out my ~5MB copy of Media Player Classic can play Bink/Smacker videos. In full-screen. On a 32-bitThe game itself can’t run Smacker videos if you choose a 32-bit color. Smacker uses 256 color, so I’m not really sure what technical limitations lock the videos to 16-bit color modes only, per the readme and the game code itself. color 1080P display. As a child, I felt like a legitimate hacker when I found a standalone Smacker/Bink player buried in a dev tools directory on a Maxis game from the same era. Finally, I could access all these strange .smk and .bik files in my game directories.

Who knew off-the-shelf media software could handle such an obscure format 25 years later? That’s legitimately impressive.

End aside.

Why does Centipede work?

Microsoft has poured more time and money than any of us can imagine into ensuring applications written for the Windows of Yesteryear run in newer Windows: Vista, 7, 8, and 10If this laptop didn’t come bundled with 10, I’d never use anything newer than 8.1. 11, and its upgrade nag screens it randomly hits me with upon booting my computer, can go off to die in a fire.. The “Application Programming Interface” (API) that Windows applications interface with to Get Stuff Done™ has remained impressively stable, or been faked in compatibility modes, for years.

But I suspect the game’s robust qualities come not from Microsoft’s efforts, but from the men who programmed it. Five devs with two interns. That’s it; the entire team to create a medium-sized 3D game with mid-90s tech. ⊕They even outsourced 100% of the sound and visuals to a separate company–stick with what you know? The reason that this experience Just Worked™ was because a small team created a well-written piece of software. The only dynamic library it uses is DirectX 5.0 from 1997, everything else seems to be static. With few external dependencies and good abstractions between the main game logic and the hardware, it runs on hardware that’s 10- to 100-times more powerful than what it was designed for, on an operating system wildly different from its initial target.

In comparison, most company’s modern software won’t even exist in 25 years, let alone be able to run. To be pedantic, I suppose you can install most modern apps on your current iPhone, stick it in a freezer for 25 years, then pull it out. The app will still be there and launch and show you a login screen that you’d never be able to pass. Because the company, or service, or both no longer exist. The end. I hope you exported PDF copies of all those Turbo Tax Online tax returns…

There is another famously modular game architecture. The original killer app: ID Software’s DOOM. The DOOM game engine was primarily programmed by John Carmack. Carmack’s personal workstation for DOOM was a NeXTThis is the company Steve Jobs cofounded after being fired from Apple. They made expensive Unix-based workstations before being acquired by Apple in the late 90s for many hundreds of millions of dollars. No, there is no Orwellian Inner Party in Big Business. What are you talking about? Unix box. DOOM was a program designed to sell exclusively on MS-DOS, but developed on an expensive Unix workstation. This is akin to using a Nissan GT-R as your test mule for a new type of headlight eyelashes for VW Beetles. Carmack had modular code that abstracted the specifics of talking to the audio, disk, and graphics drivers. That left him with two OS targets for that specific code: NeXT to actually run what he was programming and MS-DOS for the final release that customers would use.

Complexify all the things

In comparison to Centipede’s presumably very straightforward architecture, or the clean, modular approach Mr. Carmack used in DOOM, modern SAAS software is disturbingly complex and disgustingly fragile. Yet I’m surrounded by dozens of chipper young-ish devs who are convinced that This Is The Way™ and we can never again build software like Centipede 3D, because… something something maintainability? It’s inexcusable to only have a team of 1-8 developers on a big project, so instead we must adopt architectures that are easier to have 200 2,000 20,000 people working on at once, because otherwise we won’t be able to build features quickly enough? Or spend VC money fast enough? Or we’ll lose our first-mover advantage and associated synergies?

Rotten code

A related topic is “old” code. That Centipede code is old. My retail copy is version 1.1, so it got at least one patch after the gold master was released and my “Scholastic” edition was purchased two years later. So it hasn’t received any bug fixes, security patches, or new features in ~25 years. But it works flawlessly for me. Flawlessly. ⊕As does my copy of Tetris from 1989, which also seems pretty… old.

Contrast that with boring libraries that do some basic, trivial thing. For instance, libraries for receiving communication from a barcode scanner. That’s a fixed standard that’s been around for decades. What new bug fixes and features does a barcode reader library need, once the basic use cases are discovered and implemented in the first year or two of its existence? Frequently, however, I see this anxiety about using “old” code at my current employer. And the previous one. And the one before that. And even Lawn Love, where I was ostensibly responsible for such stressors. Nearly everywhere I’ve worked has had someone stress out about the age of a software package we’re thinking of including.

The messages are usually posted to all of engineering and go something like this: “Hey, there’s this useful small library that can do X for us, but it hasn’t been updated in 6 years. I can’t find anything else that does it. Is it safe to use? Can the security team review it for me?” …because all code somehow rots after it’s written, it seems. Without ever using it and seeing whether it meets our needs, that small library is deemed too risky, because it hasn’t gotten a security patch or bug fix in 6 years! If it’s targeting a fixed or otherwise mature platform, who gives a crap? I’ll grant you, if it’s for a mobile OS, maybe that’s a recipe for disaster seeing as those APIs are changed every ~6-9 months for no reasonI mean to sell new phones that are no different from your old phones. Er, whoops, I mean, to “provide small improvements and bug fixes” that were missing from your current phone.. If you ask most programmers these days, programming instructions that tell a processor to move data from memory to CPU registers for mathematical operations turn brown and mushy like last week’s bananas.

Never update your software?

When running this rant by a friend, he commented that stable is unchanging, by definition, but doesn’t that mean that un-updated software is more vulnerable to bugs or security issues?

Maybe.

I used to be more sensitive about “BUT SECURITY PATCHES!!” but I’m now deeply skeptical that bug fixes and security patches are net beneficial. How many of the bugs that are fixed and security holes that are patched are because of bugs in an earlier bug “fix” or security patch!? And the two are almost always intermingled, because no one is going to spend the time to create a standalone security patch on some “old” branch in the version control software. So it’s impossible to only receive “security” patches without also having a larger vectors for receiving additional bugs and, worse, “improvements” to the software.

On the flip side, unchanging software can’t have new bugs added to it, though researchers are always trying to figure out new ways to break your company’s awful Nodejs Javascript packages, updates or no.

Never update my software

When I find software that meets my needs, I want to use it. Forever. I’m not impressed that your MP3 tagging application has new updates–I’m happy with the existing features. I can no longer spare any cares for the incessant releases of JetBrain’s code editors, either: They’re already miles ahead of vim and emacs. If I have my choice, I’ll keep using the same version of MP3Tag and PyCharm and SoundConverter, because they… just work. Too many times have I been burned by the ol’ 2.0 version of a program, promising me the world, only to ruin the desktop environment so thoroughly (GNOME 3.0 I AM STILL UPSET WITH YOU), that I switched environments entirely… To a more stable one, XFCE, that doesn’t jack with my settings and its look and feel every few months.

The world, however, is seriously annoyed that I insist on running the same version of [insert software here] past its supposed expiration date. Every internet-capable device I own has begun a war against me. One used to politely decline to receive automatic updates and the operating system, package managers, and applications themselves would respect your request. In 2022, I’m beginning to go slightly insane as I encounter ever more software with the view that Opt-outs Are Considered Harmful™. Windows 10 forcibly installs things, which was annoying but not a big deal. Until I booted my laptop one day and it went to Windows 10 instead of Fedora. Er.. Did a Windows update just blow away my Linux boot partition..? Oh, hey, it changed the UEFI settings to boot Windows and not the Fedora bootloader. Wait. What? A Windows update can make changes in my machine’s firmware!? I did some cursory research to figure out how to make it stop and the answer seems to be “Never connect it to the internet.” Wonderful.

In addition to the glorious, nebulous “performance improvements and bug fixes,” when I booted my laptop into Windows this weekend, the updates it “configured” upon boot was to show me marketing screens to convince me to install Windows 11 on the spot. Hard. No.

Back in the day, I would have no doubts that X.org patch #307 was a cut and dry security fix, with no other impacts. But I don’t trust that patch for a second on any modern software.

But, uh, speaking of dynamically linked libraries… we couldn’t release software at my current employer when this post was written because a logging/alerting library updated its CLI in some Docker dependency, which breaks our build pipeline, which means nothing can be packaged and sent to the production servers. And there was no way to install the old one.
ᕕ( ᐛ )ᕗ

What do we do?

So what do we do about this situation, how can we make more Centipede 3Ds? I haven’t the faintest idea. Maybe teaching more young developers about what possibilities existed when we had binaries with statically linked libraries? Dazzle them with stories of how small teams of people shipped amazing software before you needed a master’s degree in Google, Facebook, and Amazon Web Scale technologies to run your Helm Argo K8s clusters of virtualized computers on top of virtualized computers on someone else’s server? Teach your company CFO about the real cost of small teams running on big iron, rather than believing Amazon’s marketing that The Cloud™ is categorically cheaper with its horde of Cloud Engineers and DevOps hires you need to keep anything online? Deciding to buy real computers that you can own and modify to your heart’s content, rather than severely-restricted phones and tablets that Google and Apple begrudgingly lease you?

Awareness is the first step on the road to recovery.

Further reading

• Managerial failings: complification – Scott Locklin blames our failed managerial class on letting this complification run amok in the first place.
• Game Engine Black Book: DOOM – Seriously, Carmack’s architecture is top-notch.

Future improvements to this post

A writer I deeply respect recommended either making this post narrower and focusing on Microsoft’s backwards compatibility, or making it “bigger” and tackling some common architectural patterns that lead to this soup. The latter is far more interesting, but I need more Stuart Hinsons in my life to argue on behalf of the current state of things. Any takers? Drop me a line at e@lawler.io.

You can’t escape it

Amusingly enough, I wrote this amendment from my original Linux laptop, which I apparently hadn’t booted since late 2019. Naturally, the version of Firefox and Slack on there had rotted⊕Is it because my laptop didn’t receive the mRNA injection?. Slack prevents me from messaging Stuart as “This version of Slack is out of date.” Ah, of course. This must be pretty serious, because it won’t let you past the nag screen informing you of this dreadful situation. A quick update to Slack via the Fedora package manager results in… Oh, wait, Slack signed me out. Ah, of course, why would it remember what workspaces I was using between major versions?

No matter, I’ll click the link to sign in in the browser and send Stuart that messa–wot? My browser, Firefox 69, is unsupported? To sign in on a basic webpage and immediately be redirected back to the desktop application no longer works in Firefox 69? No matter, I’ll update Firefox via the package manager. Cool, now I’m at Firefox 82, which is “old,” but newer than 69, so it’ll… nope, no dice. Minimum Firefox to sign in to the thin wrapper around a Chrome browser that Slack, Inc. calls an application is Firefox 88, apparently. So close!

No matter, I’ll fake the user agent string to have Slack’s website think I’m actually running Firefox 89, that’s one more than the minimum! That lets me through the first barrier, then slaps me with a non-functional page from Firefox 82’s lack of newer root certificate authorities that validate the SSL certificates littering the modern web. For my own “protection” Firefox fails to load a ton of scripts the page needs to function, so I just get a nice white screen and can’t sign in. Sigh.

Everything worked fine when I last used this machine in 2019, but these applications can’t run in 2022 because of artificial logic bombs embedded in them. Lovely.

I’ll leave you with a final comment from a reader of the first draft:

I find myself torn in terms of how I want to develop my career. On the one hand the best pay seems to be in whiz bang tech and complexification, on the other hand I want to build things that can stand the test of time.

Amen. The struggle is real.