For teams who update their product and features frequently, maintaining a high level of product quality can be extremely challenging. Organizations frequently spend too much time on essential but time-consuming regression testing instead of focusing on developing new features or improving user experience for existing ones.
Crowdsourcing is one solution for improving product quality without slowing down deployment time. Crowdsourcing provides a bandwidth boost that can help improve teams improve product quality, while also freeing up in-house resources for other quality-driving activities.
Here are three ways that implementing crowdsourced testing as part of a QA strategy has helped companies like ConsumerAffairs improve product quality.
A large factor in assured product quality is how comprehensively it is tested before going to production. But increasing testing coverage traditionally requires scaling out your QA team, or devoting more resources to automation. Crowdsourced testing provides a more flexible, cost-effective means of increasing testing bandwidth.
Crowdsourcing is especially well-suited for increasing web and mobile browser coverage, as functional and regression tests can be run across multiple browsers simultaneously. For websites and applications with a large number of pages -- such as ConsumerAffairs, who must confirm that thousands of pages are functioning correctly across multiple browsers -- crowdsourced testing allows for much faster testing than an in-house team can achieve.
UX-focused testing helps uncover issues that will matter to users faster. Crowdsourced testing puts your product in front of users earlier, tightening the testing feedback loop for developers. “The comments we get from testers are the most valuable pieces of data we have for improving the quality of our product,” said Max Spankie, a software engineer at ConsumerAffairs. In addition to providing confirmation that features are working correctly, the subjective feedback that ConsumerAffairs receives from Rainforest testers helps them discover previously unknown bugs.
Crowdsourcing also gives QA teams the advantage of having real people test your product without previous experience using it -- a key scenario for testing usability for new users. The constraint of presenting test cases to crowdsourced testers coherently can help developers and UX designers understand how user-friendly their product is.
“We realized that if it takes too much effort to write a test that the testers will understand, that’s a red flag for us about the usability of that feature,” says Max of writing tests for Rainforest testers.
Many teams now have developers own or partially own QA. This can take developers away from quality-driving activities, such as feature development. In other cases, QA teams can spend too much time on repetitive functional and regression testing, instead of working on exploratory testing and other quality-driving activities. IBM’s survey found that 65% of QA testers spend a large amount of their time on functional verification testing.
Crowdsourced testing allows teams to offload repetitive functional and regression tests and allocate their in-house resources more effectively. ConsumerAffairs saves approximately $1.3MM annually by using Rainforest for their functional testing. “We are saving dollars from our in-house developers, who were previously spending more time testing. Our developers rely on Rainforest to do a lot of testing work for them,” Max says of the benefits of using Rainforest for crowdsourcing. By leveraging crowdsourcing, the developers at ConsumerAffairs spend less time involved finding and resolving bugs and more time developing and improving features.
By using Rainforest QA to access 50,000 crowdsourced testers, ConsumerAffairs has been able to speed up testing and spend more time improving their product, resulting in smoother deployments and fewer bugs in production. “I’ve noticed a lot fewer problems when we push,” says Max. Read our case study for more on why ConsumerAffairs says that “Rainforest has become our functional QA department,” and how their team saves testing resources with Rainforest. View the case study here.