Once upon a time, around 2009, a company CEO annouced a new product. For the first time in recorded history, customers could stream high-end PC video games to their underpowered PC over a DSL connection, and only have to download a 1 megabyte (MB) to their computer for the privilege. None of these things were strictly true; however, it became necessary to make them true-enough before the release date.

The real client download was an absolutely whopping 6 MB. In order that the user experience matched the promise, we decided to add a layer of indirection. The installer would be under 1 MB, and it would quietly download the ever-so-slightly larger client program in the background. Most users with an internet connection capable of streaming video games would not notice this sleight-of-hand. After all, the client speed test exchanged 25 MB of data with our servers in a few seconds every time the user logged in, and the video-game streams themselves required several megabits per second.

The Problem

Because this company was on the bleeding-edge of video compression at the time, the video protocol was tightly coupled between the client and server. In order to connect at all the client and server had to be on the exactly the same protocol version. Also due to bleeding-edge optimizations, the install of the client had to run as a priviliged account. In order to avoid scary dialog boxes on the first install, the client and the installer had to be signed by a private key blessed by Microsoft. At the time, Windows had recently strengthened these dialog boxes, so making sure most users never had to click “Allow” was a big deal for a brand new service looking to attract users.

This set of requirements is not so technically difficult; however, the company infrastructure made this simple set of requirements virtually impossible. The service was instanced into entirely self-sufficient sites which were collections of game servers that were co-located and shared almost all internal infrastructure. A site was guaranteed to be on a particular version of the client/server protocol. Each site had its own separate client update URL, served by that same site. The official build process for the client and installer was entirely agnostic to any notion of sites, and was the only way to access the signing keys.

The problem arose when building the installer via this site-agnostic by-design build infrastructure. Once built and running on a customer’s machine the installer had to be able to reach out to the particular site it was downloaded from in order to download the correct client. However, at the moment the installer was built and signed, the sites it would be deployed to was unknowable. Once built it could not be changed without creating an invalid signature and unpleasent warning dialogs designed to discourage the user from installing the software. There was no public discovery service or similar information source that would solve this bootstrapping problem. In order to create the user experience promised by the CEO, the installer simply had to be signed already containing all of the bits necessary for it to download the correct client from the correct site.

The Organization

This issue existed because of the interaction of the organization with the technological requirements. The build infrastructure and the production infrastructure had a strict set of agreements about what information passed between them. The build infrastructure was not supposed to ever know lists of sites, or their URLs, and the production infrastructure was not ever supposed to have sensitive information like the signing keys. There were reasons for this separation, mostly tied to scale the company never reached. Relaxing either constraint in the name of shipping would have allowed a simple implementation of a cromulent self-contained signed installer. Alas, this was not to be.

Fresh out of school at the time, my boss impressed upon me that the 1MB installer was non-negotiable with the CEO. Having consulted all stakeholders, I brought the organizational dimension of this problem to him in private seeking his help. In a 1-on-1 that is still memorable over a decade on, he told me to “figure it out” and left the room abruptly. This maneuver skipped over half of the time scheduled for us to speak regularly, and exposed the gulf between his allegience to company edicts and doing the work of leading his team. Being ignored in this way was formative to my management style. I hear my team out even when its bad news. I also make visible effort to bring the task priorities and organizational priorities into alignment, even when those are tough conversations.

In the end though, the strategy of ignoring the problem paid off for him in the short-term as it often does. I went back to my desk and started on the only thing that could possibly work: A program that would modify a signed windows executable.

The Hack

I read the spec for the Windows Portable Executable file format, and began signing executables and looking at them in a hex editor to understand what was going on exactly. I learned:

  1. Signing records (and their offsets and sizes) are exempt from signature checking and all other checksums.
  2. Multiple code signatures are allowed; however Windows signing tools will only alter the first signing record.
  3. Windows (2000, XP, and Vista) only verify the first signing record. Anything could be written in the second signing record, and Windows security checks would ignore it.

Thus at the time it was possible to modify a signed windows binary, so long as your modification was adding a second signing record. I wrote a short, and heavily commented, C program to add a bogus signing record. This program would take a URL and a windows executable at the command line and create a second signing record simply containing the URL as a c-string. Building the installer would also build this modification program, targeted to run on the production servers. When the installer was installed to the webhost, a script would fetch the appropriate site URL and invoke the modification program to inject the URL into the installer. Then all future downloads from that site would have an installer imprinted with the URL.

The installer was written in the Nullsoft Install System, which has facilities to call arbitrary DLLs and read files. With these capabilities, the installer used the Win32 API to determine its own file name, and then read the URL out of its own second signing record. After reading the URL the installer would be able to download the correct client in the background. This process skipped scary warning dialogs and kept the visible download size under 1 MB.

Aftermath

Once deployed, the program to change signed binaries worked without incident for the rest of my time at that company. I was the only person familiar with it’s internals, though it was heavily commented just in case. The program depended on undocumented and unsupported behavior across several versions of Windows. Had Windows changed any of the behavior this hack relied on, it would have degraded the new user signup flow and forced an urgent review of all the above issues. To my knowledge, this never happened.

Seeing the writing on the wall after the company’s poorly performing launch, I left for greener pastures about 6 months before they went bankrupt the first time. The code that modifies signed executables is likely languishing in a source archive at Sony, who bought up the business after it’s second bankruptcy.