With more traditional develop-first-then-test models, the rate at which you can merge or ship codes depends on how fast you can test it. Every new or updated feature must go through the cycle. As a result, you can only move as fast as your QA process allows. That’s why in order to do continuous integration and continuous delivery (CI/CD), you need to adopt an equally continuous QA process.
Trying to get to CI/CD? You’ll be fighting an uphill battle if you’re still using outdated testing methods. In this post, we'll talk about how continuous QA practices ensure that you can get up to the speed of CD without sacrificing quality.
Let’s take a quick walk back in time to the bad old days of waterfall development. Remember that? Probably not. Good for you. As a refresher, waterfall was a development methodology built around the idea of boxed software releases every 12 months: slow and steady, with an incredibly high cost for releasing buggy software.
Next we had agile development. This is still the main trend in software today, and in fact many larger companies are still trying to transition to agile from waterfall. Agile sees the world in sprints, each typically two weeks long. The team figures out what you can realistically cram into a sprint and then starts working, shipping at the end of the two weeks. A huge improvement on waterfall. Fairly quickly, however, we realized the disadvantage of a two-week schedule: it’s arbitrary. Some smaller features and fixes take way less than two weeks to finish; some bigger projects take many months before the MVP is ready. So why two weeks? Because someone thought that sounded good, that’s why.
Enter continuous deployment (CD). CD is basically the idea that you release as soon as “it’s ready” — meaning that rather than release based on an arbitrary cycle time, you release whenever the feature or fix you’re working on will add value to your customers. The fundamental idea of CD is that releasing small chunks of isolated code is the least risky way to deploy since, to put it simply, fewer code changes means less risk of unforeseen bugs and easier debugging if bugs do appear.
CD is now taking the software world by storm, starting with the bleeding-edge, early-stage startups in Silicon Valley and beyond. We do it at Rainforest QA, and in fact a large part of why we started Rainforest QA is because we saw that the traditional ways of doing QA weren’t all that compatible with CD. You can think of Rainforest QA as QA for the CD age.
Let’s dig a bit deeper into why CD minimizes risk.
With CD, we release code in very small chunks. We’re talking like one-line changes here. Rather than rolling these bug fixes and smaller features up into a larger release, we release each one individually to production. Since we do thorough pre-release testing (through unit testing and of course the Rainforest platform) and monitor everything that’s going on in production, if the change causes any failure we simply roll the release back to investigate.
Rolling back a single-line change is trivial, fast, and very low risk. The developer can analyze what went wrong, fix it, and re-release all in a matter of minutes.
Compare that to the old way of doing things, where you bundle many changes into a single release. Everything is harder. QA before the release is harder — there are now many developers whose many changes are all combined together. Sifting through these and figuring out what is breaking things is a hugely complex task, since the complexity of modern systems and development frameworks means that small changes in one part of the application can often introduce regressions into other parts. So QA is harder and more time consuming.
Then, once you’ve identified the inevitable issues in the release, it’s also slower to fix them. Since you’re bundling many different changes into one release, the responsible developer is likely now working on something different. This means that they need to context-switch back to the code they wrote a week ago to try to figure out what went wrong with it. This is far less efficient both in terms of debugging and in terms of distracting them from their current project.
So CD really helps you minimize risk, which might seem somewhat counterintuitive. After all, CD can conjure up terrifying images of “moving fast and breaking things.” But done right, with proper monitoring and QA processes in place, CD is a far less risky way to deploy software than any other method.
Speed is probably the most important thing to the customers I talk to today. When I speak with other CEOs, they’re obsessed with it. It’s easy to see why: speed is what literally defines a modern technology company. In the age of minimal-to-zero capital requirements to build software, what separates great companies from good companies is the ability to consistently push their products forward. So speed is important.
Enter, again, CD. As you’ve probably guessed by now, CD is incredibly good at speeding up your process. Doing CD well creates tight iteration loops for your product teams, meaning that they can get to the “right” feature faster. This keeps you ahead of the competition and allows you to hire and retain the best talent, who are all obsessed with immediate impact.
One thing we hear all the time is that companies are held back from truly embracing CD by their QA processes. It’s the old “move fast and break things” conundrum. Do I move fast and break things or do I move slow and break fewer things? To understand the answer to this question, it helps to dive more deeply into how testing is typically done in a CD workflow.
There are two parts to a continuous deployment workflow: continuous integration (CI) and CD.
CI is the process of running unit tests and automated interface tests on every code change (typically each commit). This is great because it provides near-constant feedback to your developers, so that if they break something during development they know about it immediately and can fix it before it gets further down the line to a review phase.
CD, as we know, is pushing code to production as soon as it’s ready.
However, there’s a gap between the two, and this is what makes so many people scared of CD (and see the world in the false dichotomy of move fast/slow and break things/nothing). While CI helps you understand whether the code-level changes are causing code-level breakages, in general, it doesn’t help you figure out whether the user interface has been affected. To do this, you do manual testing, as various people across your organization use your software in a browser, mobile device, or whatever in order to figure out if anything is broken. So far so good. The problem, however, is that manual testing is slow. Slooooooow. We have to get a team of humans to act as users exhaustively through the application, testing all the different things that a user may do. As you may recall, speed is the essence of CD. So manual QA isn’t really compatible with CD. It’s a bottleneck that prevents you from truly doing CD, and causes the gap between CI and CD.
Now you may be asking “why not automate this interface testing?” Good question. Automated interface testing is, generally, fast and accessible through the APIs that your CD system speaks. Unfortunately, automating interface and user testing is pretty difficult since one is trying to get a computer to behave like a human. As we all know, computers are dumb. So the price you have to pay is learning to talk in “computer speak”, by writing the interface tests in code. You also have to endure the many incorrect results that your computer arrives at, since it doesn’t understand an interface in the way we do, but rather looks at the underlying code that makes up the interface and tries to figure out if it looks the same as last time. A simple non-breaking change can cause a test like this to fail, leading to what I’m sure many of you have experienced: flip-flopping tests that are unreliable and which your team gradually comes to ignore.
So you need something to bridge the gap between CI (which is generally awesome) and CD (which is generally very hard), which leads us to the Rainforest platform.
It’s much cheaper and faster to fix bugs that are recently introduced, since the developers will still have the context of that particular bit of code in their memory. Faster recall > faster fix > less dev time > less expensive! The consequences of not doing QA is that your customer becomes your QA team, and what you save in time and headcount during development you will “pay” with customer churn.
The role of QA in continuous delivery is to accelerate deployment, rather than hold it back. Aside from the fact that it’s cheaper to do QA continuously, it’s also much faster. Continuous testing creates a “safety net” that increases dev confidence in making changes and shipping updates. There’s a much less prominent risk of shipping code, because if something breaks they can rely on QA to catch it.
Don’t try to test everything. Don’t try to cover everything up front, just make sure critical paths are covered (shopping cart checkout, landing page, etc.). Focus on having a solid base of unit tests, functional tests, and anything automatable or deterministic to keep your testing as fast and efficient as possible. At Rainforest, we run unit tests in feature branches, and use Rainforest tests in develop which serves as our functional/end-to-end/QA tests.
Build regression tests incrementally. When making a bug fix, add a test for that bug. Adding this to your process will build up the regression suite over time in the areas you need it most.
Don’t rely on manual testing for anything that you don’t have to. Use automation and crowdtesting to offload things that don’t have to be tested manually. These are significantly faster than testing manually, which will speed things up and help you recover time from QA activities.
True continuous deployment requires a testing strategy that can keep up with rapid development cycles. Many companies are turning to continuous testing as a method for bringing QA up to speed with CI/CD processes. When utilized effectively, continuous testing can have a significant positive impact on your business:
Continuous testing can require significant infrastructure and procedural changes for your team. If you want to implement continuous testing successfully, here are some key actions you’ll have to take to get the most out your new testing processes.
According to the 2015-2016 World Quality Report, 41% of QA and testing executives cite getting the right test coverage as a major challenge to application development. Knowing what test cases you should include in your QA strategy is key to implementing continuous testing. Whether you currently use manual testing, automation or a combination of testing resources, audit your existing testing strategy and identify where your weak points are. Finding the sources of quality breakdowns and release bottlenecks -- from QA activities that your team spends too much time on, to the types of bugs you often see in production -- can help you identify where you need to bulk up your QA processes.
Continuous testing should not wait until the end of development. To reap the benefits of having a tight feedback loop between development and QA teams, continuous testing should start during development, with unit tests and functional and regression testing. For example, productivity software company Atlassian exemplifies this practice of “shifting left” with their QA practices, and regularly involve QA testers in development activities and place a focus on empowering developers to write more effective unit tests early on. As a result, they’ve been able to speed up development cycles and keep their QA team lean and efficient.
Continuous integration, or CI, gives you the advantage of a consistent build and a consistent run of your tests. Leveraging a CI server to streamline integration and execute unit tests can help your team find regressions more effectively. When used in conjunction with proper code branching strategy, CI tools like CircleCI and Jenkins can significantly reduce the bugs and provide immediate feedback on the quality of your codebase.
Automation is a critical part of scaling up QA practices. But many teams make the mistake of trying to go all-in with automation premature. When products and features are still in development, test parameters are generally in flux. Testing automation suites often require too much time and too many resources to set up for tests that many change quickly. As a result, testing automation is can be an inefficient use of resources for certain tasks. Rainforest can function as a stop gap for features that require more testing than your team can do manually, but that aren’t ready for full-blown automation. Because Rainforest tests are written for and executed by human testers, your test cases will be less brittle, less flaky, and easier to maintain than with testing automation for this stage of development.
Siloing your QA process, whether that’s with a particular team, in a specific timeframe during development, or overseas with an outsourcing firm, significantly slows down the feedback loop and delays releases. Isolating testing also increases the cost of quality; one study by IBM found that the cost of finding a bug during integration testing is double what it would be if found during implementation.
In order to keep up with fast-moving development cycles, QA must be able to work within your coding process and integrate smoothly into your development workflow. The Continuous Testing Manifesto advises that QA tools must be “pluggable,” or integrate well into your development process. Here are three kinds of integrations that allow for easy, deep integration of QA testing into your development process:
The most effective QA process is one that is part of a larger continuous integration (CI) strategy. In addition to speed, one of the key benefits of CI is consistency, allowing teams to catch bugs as early as possible in development. Verifying your work regularly with an automated build helps catch bugs quickly, before they can snowball into larger issues. By using CI, dev and QA teams can spend less time on testing overall.
Manual testing can’t scale to meet the demands of fast-paced development and often becomes a bottleneck, so maintaining an automated test suite is often the best option for developing a scalable CI process. Each release, whether manual or automated, should be automatically run and reported on.
For more on integrating functional testing into your continuous integration process, check out this blog post on setting up Rainforest to integrate with your CI server.
Issue reports come from a variety of places -- QA tests results (ideally), your team, customers (not-so-ideally, but still important!). By centralizing all reported issues into an issue tracking platform like JIRA or PivotalTracker, your team will be able to track and prioritize bugs appropriately. Using a bug tracker also decreases the all-important “time-to-fix” metric by ensuring that developers have access to the information they need, when they need it.
The key to using bug trackers effectively is context. Make sure that your process includes tracking things like state, screenshots and logs (http, server and console). Rainforest test results include everything making them a great resource for developing stronger bug tracking.
Your team might not be sitting on your CI server watching every result come in, but that doesn’t mean they shouldn’t be aware of quality snafus. Keeping QA top-of-mind even for members of the team who aren’t in the weeds of issue tracking and CI runs every day helps keep quality higher overall, and keeps key members of the team informed more consistently.
Setting up automation notifications for your team when something breaks keeps the feedback loop tight and prevents unexpected issues from slipping through the cracks. Simple email notifications when a test fails can help keep everyone on the same page. Communication apps like Slack or Hipchat are a great place to integrate updates, especially if your team is among the 3 million daily users who hang out in Slack every day.
Enter Rainforest QA! We built the Rainforest platform to be the best of both worlds: speed of automation with the quality and reliability of humans. In fact, Rainforest QA offers a giant (50,000 people) QA team from all over the world, that responds incredibly quickly and reliably in a way that no internal QA team ever could.
To give you an example, when one of our customers runs their Rainforest tests as part of their CD, the Rainforest platform returns their full interface regression results within about 20 minutes. This is fast enough to sit in their CD flow, meaning that when the developer is ready to deploy, they kick off a deployment and sit back while their testing is all taken care of. They can sleep safely knowing that if anything in the code or the interface is broken, their tests will fail and the deployment will be blocked. Most importantly of all, their customers will continue to have a great experience.
Continuous testing tools make it possible to deploy software updates multiple time a day while keeping quality high.
How the Rainforest team ships code, and what we're doing to make our deployment strategy better, faster and stronger with continuous delivery.
A less obvious cost is your CD development is effort required to keep your application CD-friendly. This post covers one of the thorniest problems of the continuous delivery world: deployment race conditions, and two alternatives for solving this issue.
Testing earlier helps teams catch bugs earlier and move faster. Here are three ways that three different companies approach testing to help speed up development and deploy more confidently.