For teams who update their product and features frequently, maintaining a high level of product quality can be extremely challenging. Organizations frequently spend too much time on essential but time-consuming regression testing instead of focusing on developing new features or improving user experience for existing ones.
Crowdsourced testing is one solution for improving product quality without slowing down deployment time. Crowd testing provides a bandwidth boost that can help improve teams improve product quality, while also freeing up in-house resources for other quality-driving activities.
Here are three ways that implementing crowdsourced testing as part of a QA strategy has helped companies like ConsumerAffairs improve product quality.
Have a scalable, expert workforce
One of the key benefits of crowdsourcing is having a large number of workers available to carry out tasks on a piecework basis. This means that it is likely you can get the crowd to start work on your tasks almost immediately. It’s also likely that you will be able to have a large number of tasks completed quickly, providing a great way to handle bursts of activity without having to employ people full time to deal with the peak load. As a result, you can reduce the need to hire a team that either sits idle when they aren’t needed, or is taken off other activities when you have a spike of work.
Because crowd-sourcing platforms generally source workers from all over the world, they also provide good 24×7 coverage of workers. You can provide services even when you are outside of your normal working hours locally — an increasingly important requirement when more and more organizations provide services online to national and international customers, who may be many time zones away from you.
In short, the crowd is on-demand, scalable, and always-on. As a result, it can be easy to assume that leveraging crowdsourced work is no different than adding a new software tool to your workflow. But while there are clear advantages to using crowdsourced work, it isn’t a perfect solution, and there are significant problems that you have to take account of before using crowdsourcing.
Increase bandwidth for more comprehensive testing
A large factor in assured product quality is how comprehensively it is tested before going to production. But increasing testing coverage traditionally requires scaling out your QA team, or devoting more resources to automation. Crowdsourced testing provides a more flexible, cost-effective means of increasing testing bandwidth.
Crowdsourcing is especially well-suited for increasing web and mobile browser coverage, as functional and regression tests can be run across multiple browsers simultaneously. For websites and applications with a large number of pages — such as ConsumerAffairs, who must confirm that thousands of pages are functioning correctly across multiple browsers — crowdsourced testing allows for much faster testing than an in-house team can achieve.
Get actionable feedback faster
UX-focused testing helps uncover issues that will matter to users faster. Crowdsourced testing puts your product in front of users earlier, tightening the testing feedback loop for developers. “The comments we get from testers are the most valuable pieces of data we have for improving the quality of our product,” said Max Spankie, a software engineer at ConsumerAffairs. In addition to providing confirmation that features are working correctly, the subjective feedback that ConsumerAffairs receives from Rainforest testers helps them discover previously unknown bugs.
Crowdsourcing also gives QA teams the advantage of having real people test your product without previous experience using it — a key scenario for testing usability for new users. The constraint of presenting test cases to crowdsourced testers coherently can help developers and UX designers understand how user-friendly their product is.
“We realized that if it takes too much effort to write a test that the testers will understand, that’s a red flag for us about the usability of that feature,” says Max of writing tests for Rainforest testers.
Offload repetitive tasks from in-house team
Many teams now have developers own or partially own QA. This can take developers away from quality-driving activities, such as feature development. In other cases, QA teams can spend too much time on repetitive functional and regression testing, instead of working on exploratory testing and other quality-driving activities. IBM’s survey found that 65% of QA testers spend a large amount of their time on functional verification testing.
Crowdsourced testing allows teams to offload repetitive functional and regression tests and allocate their in-house resources more effectively. ConsumerAffairs saves approximately $1.3MM annually by using Rainforest for their functional testing. “We are saving dollars from our in-house developers, who were previously spending more time testing. Our developers rely on Rainforest to do a lot of testing work for them,” Max says of the benefits of using Rainforest for crowdsourcing. By leveraging crowdsourcing, the developers at ConsumerAffairs spend less time involved finding and resolving bugs and more time developing and improving features.
Challenges with crowd testing
In our experience, the greatest problem with using the crowd is getting the correct results for your task. Simple tasks that can be completed quickly are likely to be completed correctly. But more complicated tasks, which require greater reading comprehension, more analytical skills, and more effort — in short, tasks that are harder — will have a greater risk of being done wrong.
At Rainforest, we tackle this by standardizing how our customers’ QA tests are presented to the workers. This reduces the cognitive load workers are under when they are testing sites and letting them concentrate on the tests. We also train them with frequent refresher courses on the behavior we expect. Making communication between the customers and the workers clearer and the workers responses more consistent and reliable when carrying out tests ensures better quality results as both sides learn to speak the same language.
Even well-trained workers are only human and can make mistakes. This is problematic with crowdsourced tasks, because generally the primary reason for using the crowd is that the task you want done is hard to automate, but easy for a human to complete. This means it’s difficult to tell the difference between right and wrong answers automatically. So how do we give our customers quality test results? We give each test to several workers, and use algorithms that have been validated over hundreds of thousands of tests to determine whether the overall result is correct or not. These algorithms and the data they rely on is being constantly refined and analyzed by our data science team to improve results quality.
Another source of errors in results are from workers who are engaged in fraud and have no intention of attempting to complete your tasks in good faith. Again we use the large data set of workers completing tasks in our environment to determine which workers are honest and which are attempting to defraud us.
Remember: To Err is Human
Finally, it’s important to remember that although workers from the crowd can appear robotic and dehumanized by the internet, they are people with desires and needs. Most are genuinely trying to deliver the best work they can. That’s why we spend significant amount of time listening to our workers, as well as our customers, to discover what we can do to improve in helping them to succeed.
At Rainforest, we continually refine how we use crowdsourced work in order to get the best results possible, from using machine learning to rapidly validate test results to creating a tester community to help support our crowdworkers. All of these efforts come together when we deliver fast, reliable QA results for customer’s sites.