Delivering B2C applications that consistently impress users requires a careful balance between the speed of innovation that consumers demand and the quality that keeps them engaged.
Development teams need to adopt a QA strategy that stays agile while providing the coverage required to stay competitive in a market saturated with options for consumer apps. In this post, we’ll dive into how teams can scale QA coverage for consumer apps to keep their testing processes fast and effective.
Seed Stage: Stay Lean & Leverage Customer Feedback
Focus On: Functional Testing, Dogfooding, Headline Case Testing
In the consumer app market, the users of a seed-stage company’s product are there for the thrill of getting their hands on an early release. They expect the core functionality to work, but aren’t looking for all the bells and whistles to be there. That means you don’t need to worry about any of the edge cases, or even the kinds of issues exploratory and informal testing might surface. Even integration testing has limited value at this point; your product is going to go through so many iterations and change so significantly that these tests will mostly be wasted effort.
Focus on internal quality metrics to ensure that building more functionality into later releases won’t mean rewriting everything you’ve done so far. Test-driven development, unit tests and code reviews are key. Use your own product team to the greatest extent possible to verify functionality (dogfooding your own product is key to understanding headline cases at this stage. It’s also beneficial to realize that the early adopters who use your product now will understand it’s barely beyond beta, and will effectively do functional testing for you. Keep communication channels open with your early users and solicit feedback often.
If the product’s functionality is brand new, even a minimum working release will leave users hungry for more. But if you’re entering a crowded market, your product needs to have something that makes it stand out—in a good way. Because these features are so new, and so new to your product, they’re likely to contain serious headline cases that will impact your product’s functionality. More importantly, they will impact the perception of your product in a crowded market. Make sure you probe for any headline cases in your most novel features; doing so may mean the difference between major positive press for the unique value your product brings to the consumer, and major negative publicity that will stop your momentum before you’ve made it out of the gate.
Series A/B/C: Focus on UX for a Broader Audience
Focus On: User Acceptance Testing, Regression Testing, Integration Testing
At this stage, companies have released successful early versions of their product and found a ready customer base. The next goal is to build on that with a solid product that will start getting wider acceptance in the marketplace. Now’s the time to start including regression tests to make sure functionality doesn’t get broken from one release to the next. It’s also time to add integration tests.
Since you’re starting to have mass-market usage, your product should be pretty stable in its structure. Making sure system components hook together properly will ensure basic issues are found and eliminated before the app gets to user acceptance testing. You also need to start user acceptance testing at this stage. You can do exploratory and informal testing if you want, but to end up with a product that’s ready to be used by millions of consumers, you’ll want a more structured UAT process that covers the major usage scenarios.
Implementing a scalable QA process is important at this stage of growth. Make sure you have a plan for tracking and addressing the bugs that are identified so your next releases can draw in even more customers.
IPO: Prepare for the Masses
Focus On: Load Testing, Penetration Testing, Edge Cases
Once you’re ready to go public (or have already gone), you’re playing with the big boys. That means matching the quality levels of a major corporation. You don’t necessarily need to be Six Sigma, but if you want to be part of consumers’ daily lives like Facebook or Instagram, you need to be up and working when the customer is.
That means adding load testing to make sure you can handle the additional capacity demands. Protecting customer’s personally identifying information is also important, as your larger customer base will likely make you a more tempting target for hackers. Depending on the nature of your business and the kind of information you collect and store, penetration testing might be valuable to you. Even without penetration testing, testing application security features is important now.
Now that you have a reputation to protect, check potential edge cases in addition to the headline cases. Some of them may be important enough to matter at this point (you’ll need to decide based on your understanding of your business and your customers). Besides the formal acceptance test, bring back exploratory and informal testing. You can afford to spend time on that now, and the “just poking around” method of testing can surface issues that “comprehensive” test plans miss.
Focus on Customer Needs for Better Quality Consumer Applications
No matter what stage of maturity your company is currently at, focus your QA strategy on testing activities that will have the most impact on your customers. By keeping your tests focused on customer needs, you’ll be able to use your QA resources as efficiently as possible.
