QA = Money + Time. But how much???

Picture of Paul Burt
Paul Burt, Monday May 18, 2015

You need QA, but how much? In the world of software there are two big types of risk: death by 10,000 papercuts and death by catastrophic failure.

QA is a hedge against risk. Both types of failure are grim, so most folks hedge against both when making their investment. We'll dive into specific differences between cuts and catastrophe in a later blog.

For now, how much should you invest in QA?

Planning your hedge

How much time, money, and energy should you spend? There are a lot of factors, but a small handful really determine your investment. Consider your industry, your market (B2B or B2C), and the size of your business.

Since a lot of this advice is specific, we've created a handy table for you to choose your own adventure. Find where you fit, and read on!

Stage B2C B2B
Seed stage
Series A - C
IPO!!!

(Pro-tip: Click on your stage to jump to that section on the page)

A caveat about our recommendations: we're listing quality processes that you'll want to do every release. Exploratory testing, for instance, is a great tool at any part of your app's lifecycle. However, only at a certain point is it worth doing Exploratory testing every release.

B2C Seed

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

If you're just starting out, your target customers tend to be early adopters. That's great, because early adopters are generally tolerant of quality issues. This is true as long as your product delivers on it's core promise.

Speaking of core promises, what is it that you're making? Early stage products are liable to change rapidly. As a result, a lot of 'best practices' around testing either don't apply. That, or you'll need to do some translation before you see real benefits.

For instance, it might sound a little crazy to forsake integration testing. No integration tests? It's hard to get a lot out of integration testing when your product is amorphous. So, what should you do instead? The highest quality yields are likely to appear in the form of better engineering practices. That is, code review, TDD, a smooth build pipeline, etc.

At this stage, you'll also probably want to delay hiring QA.

Caveat: This advice primarily applies to startups developing novel products. If you're in a saturated market, your quality bar is slightly higher. Think of dating, social, and messaging apps. We're much less forgiving of the new when comparing them to glossy products like OkCupid and Facebook. If you have a higher bar to hit, consider a higher investment in quality.

B2B Seed

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

Getting a foothold in the B2B space is a bit more demanding. The early adopter is still your main customer, but faults in your product can lead to faults in their product. Tolerance for risk is low.

That said, a lot of the same wisdom of B2C applies in B2B. Companies in this segment still need to iterate quickly to find their ideal fit, so investments in integration testing (and the ilk) are likely to be more hindrance than help.

The need for quality is still a bit higher here. The cheapest and quickest way to get a little more quality in each release is exploratory testing. This might mean using a testing service. This might mean using your peers as informal QA. Delaying your first QA hire is recommended, if you can swing it.

B2C Series A to C

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

Yay! As a company raising your A-round of funding, you've probably found a solid customer base. You have a good idea who your customer is, and what kind of revenue you can pull in from each on average.

The amoeba like lack of definition disappears. You've defined the shape of your product. As a result, iterations will probably be less radical. Good integration tests will save you real time. Your organization is also bigger now. That means there's a lot more room for miscommunication. Combat it directly with acceptance testing.

Here is where most consider their first QA hire. There are two schools of thought on how to do this.

The first is having developers own testing. This is the Netflix / Amazon / and Facebook approach. If your product and culture are amenable, then engineers own test writing. That means QA fills the role of bug triage + building frameworks to support dev testing efforts. Your head count is likely 1 to 18 for QA to Developer.

Having a strong enough engineering culture can make that hard to pull off. The alternative is if you're hiring manual or manual/automated hybrid to support your team. Then your head count is probably more in the range of 1 to 3-6 for QA to Developer.

B2B Series A to C

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

In addition to all of the B2C notes above, B2B adds additional reasons to be risk averse. You're selling up market now, which means selling to the big dogs in the enterprise field. The word 'enterprise' has certain connotations for quality and polish.

As part of your sale, you might even begin to make a few quality guarantees. That means solid uptime. That means high polish. That means load testing and solid test infrastructure.

B2C IPO!!

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

Think of Facebook, Twitter, and other giants in the space. These are your peers. Privacy is a big concern. Up-time is a big concern. There's still some tolerance for bugs, but overall your product needs to be bright and shiny.

Weird edge cases? Sure, those are fine. Maybe the submit button stops working if you have cookies off + refresh the page. No biggie.

Failure on an every day actions? Unacceptable for customers at this stage. You're serving hundreds of millions at this point. A failure that affects a small percent of your customers still means millions are affected. Rubbing salt in the wound, your failures are also publicly reported on. Big name news journals like Forbes, BusinessWeek and others love the schadenfreude.

In short, minor edge cases that only affect a few hundred customers are quite fine. Anything else is tomorrow's News headline.

B2B IPO!!

Coverage: Testing Stack:
Headline cases Unit tests
Regression cases Dogfooding
Edge cases Integration testing
Exploratory + Informal
Acceptance testing
Load testing
Penetration testing

Space agencies, banks, and airlines. In each, even the smallest details can lead to catastrophic failure. As an enterprise B2B company, you can expect any failure to cascade similarly.

Remember the Challenger disaster? The O-rings (rubber like seals) failed when subjected to both pressure and sub-freezing temperatures. The tiniest product defect can cost serious dollars. The stakes here are livelihoods, and depending on your market, lives.

With that context, investigating edge cases becomes a core part of every release.

**Want to learn how companies like Etsy and Facebook scaled up without spending a fortune on QA? Check out our ebook on Scaling QA without Scaling Your QA team for the inside scoop on smarter testing strategies for growing orgs.