What is the ideal QA process?
A good QA process doesn’t just break through quality bottlenecks; it can accelerate delivery cycles and greatly improve the confidence of the both team and customers in their product. To stay ahead of the curve teams must ask themselves, “What does my ideal QA process look like?”
The Characteristics of “Good” QA
For many teams, knowing what they should aim for is challenging. There’s no “right” or “wrong” way to approach product quality, and what works well for one organization may not be the best fit for another. An effective QA process can come in many different shapes and sizes, but it always has these central characteristics
The quality of your product impacts every facet of the business. As a result, every team — from product and dev to marketing and sales — all have a stake in your QA process. Cross-functional interests requires cross-functional input. Good QA processes are designed to help all stakeholders have input and insight into product quality
More QA doesn’t mean better quality, and what’s the right level for a startup of 10 won’t look like the right level for an enterprise of 1000. Good QA can happen at orgs of any shape and size, but it always has to fit the company’s needs
Automated & Integrated
QA happens across the application’s lifecycle, so it needs to weave in and out of the many platforms and tools that different teams use. QA is traditionally somewhat manual, from test execution to results triage. Good QA is well-integrated into the tools and processes that are essential to the team’s workflow, and whatever can be effectively automated is automated.
Flexible & Adaptive
Needs and processes change over time; QA shouldn’t have to be entirely reinvented every time the org structure changes or the company’s needs change. Good QA anticipates growth and is designed for scalability, as well as flexibility. At the high-level (overall process) as well as the test case level (not over-investing in things like brittle Selenium test scripts)
You can’t have good insights without data from practice. Traditional QA metrics often take a “spray and pray” approach to test writing, or follow edge cases down rabbit holes. Good QA takes a step back to think strategically about overall product quality and process performance, measures key metrics and uses that data to drive decisions about the team, product and process
2. What does a good QA process look like?
There’s no such thing as a one-size-fits-all QA process. Product quality standards will look different from organization to organization, and will change over time. But from examining and developing the workflows of many successful organizations, there is a core workflow that helps teams to work as quickly as possible, while using testing resources as effectively as possible:
Adapting the Ideal Workflow for Your Team
What this workflow looks like in practice will depend on how fast you’re moving, how big your team is, and what your quality goals are.
Note that the ideal process doesn’t have a time frame, nor does it have team ownership. Our concept of the ideal QA workflow is designed to be adaptable to the goals, structure and workflow of the team using it.
To explore this adaptability more deeply…
What does QA look like when my dev team owns QA?
Many teams — especially those who use CI/CD or Agile processes, are experimenting with not having an in-house, full-time QA team at all. The key concern for this model is scope creep; it’s easy for devs to end up spending more of their time triaging issues and managing test cases than they do developing new features.
In this case, the QA process should be adapted to be as well-integrated with code writing as possible.
What does the ideal QA process look like if I only have a single QA manager?
Time management is paramount here, and the product team and dev team will realistically play a larger role in the QA process than they would otherwise. The focus here should be on offloading test case execution in a scalable, sustainable way.
Is your lean QA team focused on automation? If so, you may want to reassess the value of test automation across the whole of your QA process, as it realistically represents a small percentage of the total test cases.
What does QA look like when my product team owns QA?
Product teams care a lot about product quality, and feature quality often starts with the QA team. If product owns QA, you’ll want to make sure they start drafting test cases as they write out the product spec, and stay involved during development.
But your product team isn’t doing test case maintenance long term, especially for older features. In this case, minimizing the long-term upkeep requirements of test cases and smoke suites is key.
Deep Dive into the Ideal QA Process
In this section, we’ll dive deeper into each stage of the QA process and take a look at what each part of the process should look like for optimal performance.
Feature Design & Spec
Key Takeaway: Start the QA process when you start the product spec
There’s no “over the wall” testing here. Quality questions start as soon as the feature or app in question is being designed. This is test-driven development at its core. Start by QA’ing your spec just as you would QA code. Teams should also draft out the initial user flows and test scripts as early as possible.
There are two major benefits to this. The first is that developers gain additional context to pull from as they implement the feature; you can “fix” many issues by never implementing them in the first place. This practice of early test writing also saves time and effort later in the process, as the “bones” of the key test cases are already there.
Questions to Answer:
- What are the core user “flows” through the feature being designed?
- Who should write these first iterations of tests?
Key Takeaway: Unit tests are essential to long-term product quality; use them, and try to be organized about it.
Unit testing traditionally sits solidly in the realm of developers, and this probably won’t change any time soon. While it adds a bit more effort to the coding process, devs should be writing unit tests alongside code. Unit testing ensures that the team gets extremely fast, extremely granular feedback continuously as the feature shapes up.
If your system allows it, leverage a tagging system for unit tests to make insights stronger, running subsets more effective, and to make the entire test database more adaptable as the code base grows.
To get greatest leverage from your unit tests, run them in the following order:
- The tests for files changed.
- The entire feature changed. Structure or tag tests by feature to help here.
- All tests. Usually this should be for every push and run via CI only, due to speed.
Ad hoc & Exploratory Testing
Key Takeaway: As the feature shapes up, you’ll have to adapt your initial test specs to match the actual product. Use ad hoc manual and exploratory testing to refine your functional test flow.
As soon as there’s enough of a feature in existence to test, start testing. At this point things can still be a bit muddy, so manual testing should be your method of choice. Ad hoc manual testing and exploratory testing sessions will help you spot issues without creating QA-based technical debt that trying to script automated tests right now would do.
Don’t forget that you’ve already drafted your initial test cases during Design, so now it’s a matter of refining those test cases to match what your developers have built, and integrate feedback from manual testing. By the time the feature is ready for “Testing,” you should have a healthy set of tests, informed by the feature itself, ready to go.
Version control for your QA tests comes into play starting now (and continuing on forever). Having a record of how test cases have evolved over the application’s lifecycle will help you pinpoint issues that are due to faulty tests, and prevents you from having to do extra work if you need to roll back a feature; simply roll back to the test suite that matches the version of the feature you’re testing.
The benefits of version control for your test cases:
- Always knowing what the expected behavior is for that code.
- Being able to review and accept tests using standard code-review processes.
- Keeping a history of tests.
Test Execution & Environments
Key Takeaway: Kick off as many tests as possible automatically, integrating things like CI, crowdsourcing and automation into each other.
Centralizing the testing “engine” helps keep execution time down and minimizes risk of things falling through the cracks. Good QA process is always part of a wider Continuous Integration (CI) process. The primary benefit of CI is not speed, but consistency. Continuous Integration provides a safety valve for quality as early as possible in the development process. Verifying your work regularly with an automated build helps catch bugs quickly, before they can snowball into larger issues. As a result, less time needs to be spent on testing overall. Each release, whether manual or automated, should be automatically run and reported on.
Maintaining an automated test suite is the best option for developing a scalable CI process. Human-based testing often can’t scale to your needs, so integration may become a bottleneck. Further, if things aren’t automated, they’ll be missed. But it’s important not to automate test cases too early, as they are more brittle, flaky and generally require closer maintenance. Finding a balance of the execution methods that will streamline testing without creating additional technical debt or work is critical.
To learn more about the components that go into a streamlined testing process, check out Getting to Continuous Deployment. This session featuring Kevin Bell from CircleCI covers the infrastructure and processes needed for fast, efficient test execution.
Key Takeaway: Not all issues are created equal; focus on triage strategy
Your highly-effective QA tests just found an issue, just as it was designed to do! Just as it’s easy to get caught up in testing all the things, it’s a slippery slope towards fixing all the things. Not all issues are created equal however, and it’s important to take a strategic approach to triaging issues as they surface to prevent your team from chasing small potatoes and ignoring big ones.
Set the expectation for triage before a bug hits. A bug tracking platform to keep track of this process, and individual issues, is essential. A labelling system for triage is extremely useful, allowing your team to flag non-show stopping issues and move on to more pressing issues, without feeling like they’re ignoring quality concerns.
Whether the issue is a bug, an environmental issue or a broken test, be sure to include test case refactoring into the checklist of activities that must be done before the issue is considered “resolved.”
Questions to Answer:
- Who is responsible for tracking and triaging issues?
- What are expectations around turnaround time for show-stopping issues?
Testing & Maintenance
Regression Test Cases
Key Takeaway: Don’t let regression testing become edge case testing; focus on show-stopping regressions to optimize impact.
As your team goes through the process of checking and testing the feature, you’ll start building a list of regressions to include in your test suite. These should not come up again in production once the issue is resolved. Add a test case to your suite to ensure that it doesn’t.
Your regression test cases may not be a static group of tests. As you discover new issues you’ll likely need to add regression tests, and in some cases you may be able to “retire” any that have become obsolete.
It can be tempting to write automated test scripts for your regression test cases immediately. However, if the feature is still being changed frequently, automated regression test cases will be too flaky to be reliable. Resist the temptation and stick with manual or crowdsourced execution for now.
Key Takeaway: Streamline your smoke suite for speed and ease of execution.
The main goal of a smoke suite to have insight into critical breakage as quickly as possible, without taking resources away from other projects for excessive testing. Your smoke tests should be fast to execute and focused on areas that are most essential to how your users interact with the product.
Smoke tests should be run for every new build. Once a feature is in production, the smoke suite can be run at regular intervals to ensure that everything is still functioning as you expect it to. Many teams kick off their smoke test at the same time every night (or every week) to provide a baseline of quality insight on an ongoing basis.
Prioritize your test suite for maximum impact and speed. Minimize edge cases wherever possible!
Key Takeaway: Test case automation is a powerful tool when used for stable tests in production, but a distraction elsewhere
Test automation is a valuable component of mature QA strategies, and can ensure that test execution doesn’t slow down development. But automation isn’t a silver bullet for every quality woe, and comes with its own costs related to test maintenance and hiring for specializing skills. Teams that want to use test automation should implement it strategically. Instead of trying to use as much automation as possible, as early as possible, focus instead of automating production test cases that are stable and not likely to change any time soon.
It’s also helpful to think of automated tests not as a final end state for test cases, but as part of a continuum of execution methods. Teams with the most robust test automation strategies have a workflow in place for rolling broken or unreliable automated tests back to a manual state (and back to automation again) to provide uninterrupted quality insights.
To dig deeper into the subject of test automation, listen to these sessions on optimizing test automation for impact:
Additional Resources: Improving Your QA Workflow
For most teams, the path to their ideal QA workflow will be an iterative process. To learn more about developing a QA strategy and workflow that will help your team excel, check out these additional resources:
- 90 Days to Better QA
- Continuous Testing Manifesto Guide
- Applying Continuous Testing for Continuous Delivery
Level Up Your QA Process
Want to empower your team to deliver better quality results? Sign up for updates on exclusive QA University content. We’ll only email you when new content launches.