How Bad QA Killed 2.2 Million Imperials: A Death Star Post-Mortem
The Death Star. It was to be a weapon to bring order to the galaxy through fear. Larger than any other space station ever created in that galaxy a long, long time ago, the most terrifying weapon that the Galactic Empire ever unleashed was brought down by a plucky space farmer who liked to shoot womp rats back home and his friend, his space dog and their space Winnebago. Then, a scant 4 years later, they lose ANOTHER Death Star. The saddest thing about the Death Star (besides the whole blowing-up-Alderaan thing) is that with just a little bit of buy-in and ingenuity, the whole fiasco could have been prevented.
We’ll be exploring some mistakes the Empire made while constructing and deploying their Evil Space Globe, from both a QA perspective and just a general software development working in an Agile environment perspective. You can group these failings into some general categories:
- Not Seeking Buy-In
- Fast and Loose QA Practices
- Not Changing the Process and Expecting a Different Result
Death Star QA Mistake 1: Not Seeking Buy-In
Let’s start at the beginning. The Death Star was the brainchild of Director Krennic, who might go down as the worst product owner in the Empire. This is clearly a guy who does not care about adhering to good process and transparency. His project scales up too quickly and he realizes that he has a problem with brain drain, so he dips back into his well of experience and decides to coerce his lead engineer back into the fold. The good Director’s failing, however, wasn’t giving that lead engineer the usual, “Hey, come on back and we’ll give you a lot of stock options!” kind of deal that we see in our tech industry. Instead, he goes the, “I’m going to murder your space family” route.
That lead engineer, Galen Erso, faces a tough situation here. Krennic isn’t asking for his buy in and Galen faces the following issues:
- He’s not interested in working on the product.
- He has serious misgivings about the business logic of the product.
- He really hates his boss and has already been proven to be a flight risk.
The next thing Krennic does is moving forward with his product without the buy-in from either of his bosses, Grand Moff Tarkin and Darth Vader. Tarkin sees the Death Star as a waste of materials and resources, being led by a guy that he thinks is a big ol’ idiot dumb-dumb. Darth Vader seems to only care that the Death Star is causing problems that he doesn’t want to deal with, particularly with his busy schedule of floating in a tube tank of goo in his castle, built on top of a volcano where Obi-Wan Kenobi chopped off his limbs and left him for dead (Vader is nothing if not a little dramatic).
To recap, Director Krennic wants to push his vision and his business practices on three people who he hasn’t solicited actual buy-in from: his direct report, his boss, and a Dark Lord of the Sith. Getting buy-in on your project is a holistic advantage and you should be seeking it not just from your key stakeholders but also from the development team that is going to be working with you. When everyone is aligned on the vision and end result of a project or feature, it reduces uncertainty and allows everyone to focus on a greater vision: quality. Because Krennic is neglecting managing the expectations and processes for the Death Star project, he’s created an under-performing team with low morale coming right out the gate and his bosses think he’s incapable of actually delivering (after all, he’s already been majorly delayed because he let Erso go to begin with).
If there is a win in this mess, it’s that Krennic is at last making forward progress on finishing the Death Star. However, with time pressure coming from his bosses, the urgency to deliver leads to the Empire’s second mistake.
Death Star QA Mistake 2: Fast and Loose QA Practices
Whoever the QA manager on Krennic’s team was was probably ripping his hair out everyday.
The first instance we see of some truly bad QA practices is when Krennic wants to use the Death Star to destroy not just Jedha City, but the entire planet of Jedha at the end of the first act of Rogue One. Thankfully Tarkin reins him in on this plan, but how many times have you had a producer or executive that wanted to go way overboard on your first actual test? Krennic wanted to release worldwide, whereas Tarkin wanted to release to a small test market first.
In real life, you should be defining your quality processes and defining someone to be the invested “owner” of quality. For larger companies, maybe you already have a QA manager who is in charge of overseeing a team’s QA process and making sure that releases are going out with confidence. For smaller companies or teams, maybe that owner is a product manager. It tends not to really matter who has the title here, it’s just that someone needs to care about the testing process and can hold the team accountable for decisions that undercut the QA process.
So we’ve seen that in Rogue One, the Empire has some pretty weak testing processes. Instead of testing their superweapon on a dead asteroid or something, they go straight for wiping out an entire city. What if firing the main gun at 100% (remember, they destroy Jedha City with a single-reactor ignition) caused a chain reaction that destroyed the whole thing? The Death Star was filled to the brim with kyber crystals, the fuel that powered the main cannon. How much did the Empire really know about the reactionary properties of kyber crystals? What if hitting a kyber stockpile on Jedha caused enough destructive potential to take the Death Star out of orbit? Given how close the Death Star is to Jedha when they fire, it is not inconceivable to think that they could have been pummeled with debris ejecting out of Jedha’s atmosphere.
The Empire had no way of knowing this stuff actually worked. However, with some forward thinking, they could have had QA teams embedded with their development teams to review the specs and stories being generated, catching edge cases and problems long before a guy was welding some scaffolding floating around in space above an Imperial blacksite. It’s so much cheaper to bring QA early into the fold, looking for issues just like this and also developing regression test suites that can be used as a safety net later on, either being executed by a QA team, a development team, or even a separate QA group that only does regression tests while the embedded QA work on sprint-level work.
In fact, having someone reviewing the Death Star plans and testing that their infrastructure was solid might have even caught the fact that Galen Erso, our disgruntled lead engineer, intentionally designed a flaw that would allow the Rebel Alliance to destroy the Death Star.
The Empire was so confident in the quality of their build that they thought it was inconceivable that a one-meter wide port would lead to the instant destruction of the Death Star that they powered on anyway. At the Battle of Yavin, a mere handful of days after the destruction of Jedha City, Tarkin is even informed that there is a weakness and he chooses to ignore it! Tarkin is taking a large gamble at that point, deciding to test the Death Star on production. But unlike the rest of us who have released untested content onto production and had to spend a weekend working on hot fixes, Tarkin’s gamble led to Luke Skywalker, a nobody (or so we think) from nowhere, shooting two torpedos down a single exhaust port, leading to the instantaneous deaths of 1.1 million Imperial Fleet staff and Stormtroopers.
This is a teachable moment. Krennic and Tarkin made some grave mistakes:
- They didn’t have an owner for quality;
- They weren’t testing early and often;
- They were testing on production, a lot, and on a Friday too.
One would think that this would lead Vader, the Emperor and the entire Galactic Empire to sit down and have a retrospective on what went wrong and how to prevent it for next time. But because the Empire is built on incompetence and arrogance, they did the opposite of that and built a second Death Star.
Death Star Mistake 3: Not Changing the Process and Expecting a Different Result
Maybe the Empire did learn something. For all we know, maybe they fixed the whole exhaust-port-doomsday thing. It’s hard to tell because a solid third of the Death Star II hasn’t even been built. At the beginning of Return of the Jedi, the station commander is clearly beyond stressed that Darth Vader says that he needs to speed up the development process.
The solution? Just throw more bodies at the problem. They don’t even bother testing the gun on this one; they build a shield generator on the Endor moon and decide to invite the entire Rebel Alliance fleet, the largest such fleet to stand against the Empire around, and have a climatic battle for the fate of the galaxy.
- Doesn’t have a quality owner.
- Isn’t testing anything, it seems.
- Rushed to worldwide production.
And how did that work out for the Emperor?
- The Imperial Stormtrooper garrison, an elite battle force, on Endor was defeated by the Ewoks, a flesh-eating race of teddy bears (remember, the first thing they try to do to Han when they take him to the Ewok village is they try to cook him).
- Vader’s Super Star Destroyer crashes into the Death Star II.
- Lando and friends, once the shield generator is brought down literally just flies into the Death Star II and blows up its main reactor directly. Maybe the exhaust port was better than nothing.
The Death Star was a project that was seen as a resource sink by its administrators, had bad QA support and was rushed to production by management. The Death Star II had the buy-in from management but had bad QA support and was, again, rushed to production by management and didn’t account for the issue of team storming, where adding more people to an established team will create a disruption in productivity for a sprint or two until velocity begins to increase again. In the end, it allowed a ragtag group of rebels and their converted warships to take out the most powerful symbols of the Galactic Empire (the Emperor, Darth Vader, and the Death Star 2) in one swoop.
If there was a QA team on the Death Star 2, I think we all know what ended up happening to them: