What QA Metrics Should I Be Measuring?
Gathering insights that will help you understand how your QA process is performing is essential for continued success. But knowing what QA metrics to measure can be a hurdle in itself.
Identifying Your Essential QA Metrics
Pinpointing the metrics that map to your business goals and help you understand where your QA process is performing and where it needs improvement will give your team more confidence in their quality initiatives.
As a baseline, the quality metrics your team tracks should be:
- Trackable over time
- Maintained and updated regularly
- Tied to business goals
1. Test Coverage
The number and spread of tests across the code base. This provides insight into where your resources are being used. Mapping test coverage to usage trends will help you determine where to focus your testing activities for the greatest impact.
Be careful when using test coverage as a metric, because it can easily tell an incomplete story. By itself test coverage is not a measure of the quality of the tests or the thoroughness of those tests. It should only be used to work out which areas are completely untested.
Broken or unreliable tests that aren’t providing useful quality feedback. Flaky tests not only waste time and resources, but also reduce overall confidence.
Especially when using automation, you should track tests which pass or fail intermittently. This indicates possible poor test quality, poor choice of system or poor execution.
Execution problems can be the result of human QA tester or test environment failures. Monitoring when this happens will expose patterns, allowing you to fix the root cause. Avoid being only reactive here.
The amount of time it takes to run and report results for a set of tests. This helps teams understand how to make testing cycles as efficient as possible.
Keep in mind that the time it takes to execute tests can be offset by the time required to maintain and refactor them over time. Comparing these two metrics is especially important for teams using test automation.
The amount of time between when something breaks and when it is fixed, giving insight into QA and developer communication. Measuring time to fix answers how a development team is able to use the output from QA to triage and fix bugs.
5. Escaped Defects
The number of defects that reach customers. This is one of the most direct measures of QA success!
Tracking the percentage of customers affected by a given defect will help you gauge the overall impact of the defect on your business.
NPS (Net Promoter Score) can be a useful indicator of overall quality, but can be challenging to map to specific quality issues.
While it’s a great end-measurement for your entire product, it’s ultimately a trailing indicator that can be influenced by product improvements, customer success effectiveness, and other factors apart from QA.
Use Test Coverage for:
- Understanding which product areas are undertested
- Determining where your team should focus their efforts
Measure Flakiness with:
- Tests that pass or fail intermittently
- The root cause of failures (poorly written tests, human error, etc.)
Measure Time-to-Test with:
- How long it takes to run smoke and regression tests after a new build
- How long it takes to run new feature tests
Measure Time-to-Fix with:
- The process used to triage and resolve bugs
- The source of reported issues (customers, internal testing, etc.)
Measure Escaped Defects with:
- Number of defects that reach production
- Percentage of affected customers
- Product area
- Time to fix
Use NPS for:
- Assessing customer satisfaction and product performance
Infographic: 6 Essential Quality Metrics
More of a visual learner? Check out the 6 essential quality metrics in infographic form:
QA Metrics to Avoid
Not every measure of QA is created equal — some metrics simply can’t provide the depth or breadth of knowledge that teams need to move the meter on quality. Even worse, some metrics can encourage bad habits and negatively impact productivity.
Quality metrics should not…
- Be Too Vague: If metrics are too vague, they won’t provide any actionable insight. Focus on metrics that are quantifiable and measurable.
- Be Too Specific: If metrics are too specific, they will be impossible to impact. Your quality metrics must be broad enough to represent a real difference made to overall product quality.
- Measure Individuals: If your metrics measure individual performance, it stops being a quality metric and starts being an employee performance metrics. These tends to be poor indicators of actual product quality, and can be easy to game.
Level Up Your QA Process
Want to empower your team to deliver better quality results? Sign up for updates on exclusive QA University content. We’ll only email you when new content launches.