Why Continuous Deployment Sucks Without Continuous QA
Trying to get to continuous deployment (CD)? You’ll be fighting an uphill battle if you’re still using outdated testing methods. In this post, our CEO Fred Stevens-Smith talks about how continuous QA practices ensure that you can get up to the speed of CD without sacrificing quality.
How We Arrived at Continuous Deployment
Let’s take a quick walk back in time to the bad old days of waterfall development. Remember that? Probably not. Good for you. As a refresher, waterfall was a development methodology built around the idea of boxed software releases every 12 months: slow and steady, with an incredibly high cost for releasing buggy software.
Next we had agile development. This is still the main trend in software today, and in fact many larger companies are still trying to transition to agile from waterfall. Agile sees the world in sprints, each typically two weeks long. The team figures out what you can realistically cram into a sprint and then starts working, shipping at the end of the two weeks. A huge improvement on waterfall. Fairly quickly, however, we realized the disadvantage of a two-week schedule: it’s arbitrary. Some smaller features and fixes take way less than two weeks to finish; some bigger projects take many months before the MVP is ready. So why two weeks? Because someone thought that sounded good, that’s why.
Enter continuous deployment (CD). CD is basically the idea that you release as soon as “it’s ready” — meaning that rather than release based on an arbitrary cycle time, you release whenever the feature or fix you’re working on will add value to your customers. The fundamental idea of CD is that releasing small chunks of isolated code is the least risky way to deploy since, to put it simply, fewer code changes means less risk of unforeseen bugs and easier debugging if bugs do appear.
CD is now taking the software world by storm, starting with the bleeding-edge, early-stage startups in Silicon Valley and beyond. We do it at Rainforest QA, and in fact a large part of why we started Rainforest QA is because we saw that the traditional ways of doing QA weren’t all that compatible with CD. You can think of Rainforest QA as QA for the CD age.
Optimizing for Minimal Risk
Let’s dig a bit deeper into why CD minimizes risk.
With CD, we release code in very small chunks. We’re talking like one-line changes here. Rather than rolling these bug fixes and smaller features up into a larger release, we release each one individually to production. Since we do thorough pre-release testing (through unit testing and of course the Rainforest platform) and monitor everything that’s going on in production, if the change causes any failure we simply roll the release back to investigate.
Rolling back a single-line change is trivial, fast, and very low risk. The developer can analyze what went wrong, fix it, and re-release all in a matter of minutes.
Compare that to the old way of doing things, where you bundle many changes into a single release. Everything is harder. QA before the release is harder — there are now many developers whose many changes are all combined together. Sifting through these and figuring out what is breaking things is a hugely complex task, since the complexity of modern systems and development frameworks means that small changes in one part of the application can often introduce regressions into other parts. So QA is harder and more time consuming.
Then, once you’ve identified the inevitable issues in the release, it’s also slower to fix them. Since you’re bundling many different changes into one release, the responsible developer is likely now working on something different. This means that they need to context-switch back to the code they wrote a week ago to try to figure out what went wrong with it. This is far less efficient both in terms of debugging and in terms of distracting them from their current project.
So CD really helps you minimize risk, which might seem somewhat counterintuitive. After all, CD can conjure up terrifying images of “moving fast and breaking things.” But done right, with proper monitoring and QA processes in place, CD is a far less risky way to deploy software than any other method.
Bringing QA Up to the Speed of CD
Speed is probably the most important thing to the customers I talk to today. When I speak with other CEOs, they’re obsessed with it. It’s easy to see why: speed is what literally defines a modern technology company. In the age of minimal-to-zero capital requirements to build software, what separates great companies from good companies is the ability to consistently push their products forward. So speed is important.
Enter, again, CD. As you’ve probably guessed by now, CD is incredibly good at speeding up your process. Doing CD well creates tight iteration loops for your product teams, meaning that they can get to the “right” feature faster. This keeps you ahead of the competition and allows you to hire and retain the best talent, who are all obsessed with immediate impact.
One thing we hear all the time is that companies are held back from truly embracing CD by their QA processes. It’s the old “move fast and break things” conundrum. Do I move fast and break things or do I move slow and break fewer things? To understand the answer to this question, it helps to dive more deeply into how testing is typically done in a CD workflow.
The Testing Bottleneck in CD Workflows
There are two parts to a continuous deployment workflow: continuous integration (CI) and CD.
CI is the process of running unit tests and automated interface tests on every code change (typically each commit). This is great because it provides near-constant feedback to your developers, so that if they break something during development they know about it immediately and can fix it before it gets further down the line to a review phase.
CD, as we know, is pushing code to production as soon as it’s ready.
However, there’s a gap between the two, and this is what makes so many people scared of CD (and see the world in the false dichotomy of move fast/slow and break things/nothing). While CI helps you understand whether the code-level changes are causing code-level breakages, in general, it doesn’t help you figure out whether the user interface has been affected. To do this, you do manual testing, as various people across your organization use your software in a browser, mobile device, or whatever in order to figure out if anything is broken. So far so good. The problem, however, is that manual testing is slow. Slooooooow. We have to get a team of humans to act as users exhaustively through the application, testing all the different things that a user may do. As you may recall, speed is the essence of CD. So manual QA isn’t really compatible with CD. It’s a bottleneck that prevents you from truly doing CD, and causes the gap between CI and CD.
Now you may be asking “why not automate this interface testing?” Good question. Automated interface testing is, generally, fast and accessible through the APIs that your CD system speaks. Unfortunately, automating interface and user testing is pretty difficult since one is trying to get a computer to behave like a human. As we all know, computers are dumb. So the price you have to pay is learning to talk in “computer speak”, by writing the interface tests in code. You also have to endure the many incorrect results that your computer arrives at, since it doesn’t understand an interface in the way we do, but rather looks at the underlying code that makes up the interface and tries to figure out if it looks the same as last time. A simple non-breaking change can cause a test like this to fail, leading to what I’m sure many of you have experienced: flip-flopping tests that are unreliable and which your team gradually comes to ignore.
So you need something to bridge the gap between CI (which is generally awesome) and CD (which is generally very hard), which leads us to the Rainforest platform.
How Rainforest QA Makes CD Better
Enter Rainforest QA! We built the Rainforest platform to be the best of both worlds: speed of automation with the quality and reliability of humans. In fact, Rainforest QA offers a giant (50,000 people) QA team from all over the world, that responds incredibly quickly and reliably in a way that no internal QA team ever could.
To give you an example, when one of our customers runs their Rainforest tests as part of their CD, the Rainforest platform returns their full interface regression results within about 20 minutes. This is fast enough to sit in their CD flow, meaning that when the developer is ready to deploy, they kick off a deployment and sit back while their testing is all taken care of. They can sleep safely knowing that if anything in the code or the interface is broken, their tests will fail and the deployment will be blocked. Most importantly of all, their customers will continue to have a great experience.
Trying to get to continuous deployment this year? Join Rainforest QA on January 26th for our first webinar, "Getting to Continuous Deployment," for insights from CEO Fred Stevens-Smith and CircleCI Developer Evangelist Kevin Bell. They'll be discussing more on why manual QA can kill your ability to deploy quickly, how to discover false positives from automated testing, and more. Register now to make sure you get this insiders' look at getting to CD without screwing it up!Filed under: