In-house QA is killing your business

Picture of Fred Stevens-Smith
Fred Stevens-Smith, Monday March 10, 2014

Quality Assurance has traditionally been handled by teams of in-house testers who do exhaustive manual and automated testing. This worked great when you deployed every 6 months - but now modern companies like Amazon, Facebook and Google ship hundreds of times every day. It's almost impossible to ship fast and frequently while maintaining quality with in-house QA, and this is hurting your business.

It is slow, inefficient and expensive. It's slowing you down and preventing your developers from being happier and more productive. It's making it harder for you to attract top talent. It's keeping you from becoming the next Amazon. I believe that keeping QA in-house is harming your business, and I will show you how.

Wait - you're saying I should have no QA?!

Before you go out and fire your entire QA team, don't! I'm not advocating that you should have zero QA headcount.

It's important to make the distinction here between 'testing' and 'checking'. Checking is rote checking of functionality against multiple browsers based on test scripts. This is what most of us think of as 'QA'. Testing, on the other hand, is the far more leveraged and nuanced act of trying to expose unknown bugs in an exploratory manner.

Simplifying, one relies on smart humans who do need to be intelligent and inventive to do their job, and one simply relies on humans with a godlike ability to do repetitive menial tasks day-in and day-out. These checkers are not the kind of humans that you want to be hiring.

However, you need testing and checking to deploy quality applictions. Unfortunately, checking is the deployment bottleneck (exhaustively checking functionality and appearance for all the flows that matter takes time), and is also likely the majority of what your QA team spends their time on.

You should outsource the checking of your app to a dedicated service, so that you can move faster, save money and make your customers happier.

Let's explore why you shouldn't have in-house checking. (For the rest of this article I'm using the terms QA and checking interchangeably. We will cover the optimal way to integrate QA testers into your workflow in a later blog post.)

In-house QA is too slow

The core problem with in-house QA is that it's simply not fast enough to keep up with the deployment strategy your business needs to stay competitive.

It's so slow to do in-house because QA is done sequentially: I check signup in IE8, then in IE9, then in IE10. So there are very low returns to scale when it comes to your QA team, because adding headcount will only ever increase throughput linearly.

Essentially, in-house QA represents a process problem. It's incredibly slow and painstaking, and it ends up becoming a bottleneck in your development process as soon as you start to try and ship frequently. This is becoming more and more relevant to your business as the industry gradually embraces continuous deployment.

In-house QA teams don't fit how your developers want to work

Demand for QA tends to be spiky. When you release frequently, you want all your QA done in 5 minutes, and then nothing until your next release an hour later. Unfortunately, human teams don't handle spikes very well because they are inherently difficult to scale.

Aside from the difficulty of actually hiring people, it's hard to decide how large the team should be. Scaling a human team where marginal returns are linear is hard, since there is always a tradeoff between total output from the team at peak demand and number of bodies sitting idle during dormant periods.

From a personnel standpoint the optimal solution is to have sufficient QA headcount to have everything done just in time for the next release. This minimises downtime and locally maximises efficiency. However, this also makes it impossible to deploy fast since your releases won't go live until the next one is ready. This will scupper your continuous integration plans.

So you have to choose between quick QA with massive overcapacity or slow test cycles. It's a lose-lose. Your developers want QA done fast. Your team of in-house checkers can't provide that.

Doing in-house QA without a QA team is even worse

I'm at how often I talk to really smart people who work for companies I admire and we get around to talking about QA, and they say they have no in-house QA. Amazing! How do they do it?! How do they maintain great production quality while having no QA team? Magic? And every single time, I quickly realize that they have no in-house QA team because they just get everybody in the company to do it. PMs, engineers, designers, secretaries. The 'full company QA run'.

This is much, much worse. You simply cannot afford to have your entire team stop work to manually check new code before it gets released to production, yet many smart companies are still doing this.

Entire teams shut down for each release, and are expected to manually check their app and report bugs. These are teams of highly skilled, educated and trained workers where the average wage is well north of $100k, and where the company has invested significant time, cash and brand value into attracting them in the first place. So why are these incredibly valuable resources being wasted on unskilled work and prevented from doing their jobs, even in otherwise awesome and forward-thinking companies?

It's actually fairly logical. These companies know that in-house QA sucks. So they decide not to hire an in-house QA team. But then they wonder what to do next. One of their dev team suggests investing in automation. They play around with it, create some tests, maybe even put someone on 'writing integration tests' (this is in quotes because making 'writing integration tests' a task, much like 'writing code', is so nebulous as to be irrelevant). They already believe in dogfooding. And with the total lack of decent alternatives to in-house QA, it makes sense to them to extend the concept of dogfooding to encompass hours and hours of exhaustive checking. And I would agree it makes sense at some scale. But if either customers are paying for your product or you're paying your staff, you are already too big to be making mistakes like this.

The point is, no company arrives at this process through considered planning: it's more like an unhappy accident. And it unfolds like a slow-motion car crash. You don't realise how much everyone hates 'company QA runs' until your best employees start leaving. And you can be sure that no developer ever joined a company due to their excitement about doing QA.

Don't worry, many of our paying customers came to us from this exact process. They tried it, realized it doesn't work and decided to hire Rainforest for the job.

So how can I move fast without breaking things?

What we need is a way to do QA that doesn't rely on fixed teams of humans. Ideally this solution would also be easy for anyone on the product team to use, and be simple and quick to get setup.

Unfortunately automated testing is out, because you need to be a programmer to write automated tests, and the majority of your product stakeholders are not actually programmers. They are UX designers, Product Managers, marketers, Ops guys. And even if you have tons of programmers to spare, you shouldn't be wasting their cycles on creating test scripts.

Another way to do this is to use Rainforest.

Introducing Rainforest

We built Rainforest because we were frustrated with how hard it was to ship fast while maintaining high quality code, and we use Rainforest every day to test our various projects (including Rainforest!).

Nearly every one of our paying customers found us because they were unhappy with their existing in-house QA setup. Here's why they think Rainforest is awesome:

  • Rainforest tests are written in plain English so anyone on your product team can create and maintain tests.
  • Rainforest is fast so your team can ship more frequently.
  • Rainforest abstracts away the complexity of managing and scaling human teams so you can focus on what matters to you as a company.
  • Rainforest has a first rate API and CLI and is super easy to integrate with your existing CI processes.
  • Rainforest is SaaS, so there's no infrastructure setup and you can be testing in 5 minutes.

I guarantee that if you're experiencing the problems outlined in this article you won't find a better solution than Rainforest.

I spend all my time thinking about how to make QA painless and beautiful for companies like yours, so if you want to chat just hit me up on Twitter or email me.