No QA team? 4 ways to automate your tests and what to consider

Is your startup operating with no QA team and trying to figure out your options for improving software quality?

Perhaps you’re proactively thinking about your QA process before quality becomes a problem.

Or maybe quality issues have forced you to finally address your QA process (or lack thereof).

Either way, implementing end-to-end (E2E) automated software testing is almost always part of the answer. Compared to manual testing, it’s cheaper and faster to execute, more reliable (humans get distracted and bored!), and is a better fit with modern DevOps pipelines featuring CI/CD.

But when you don’t have a dedicated QA team, how do you make the transition from manual testing to automation?

You have four different options, each with its own tradeoffs:

• Have your developers use open-source frameworks
• Hire a QA engineer
• Use low-code/no-code tools
• Outsource QA

In this piece, we explain the pros and cons of each option so you can make an informed decision for your software team — and get to a point where you’re releasing fast, with confidence.

The information we’ll share is based on a decade of our team’s experience in this market, plus surveys we’ve run and hundreds of conversations we’ve had with startup engineering leaders.

1. Have your developers use open-source frameworks

One way to start automating tests is to have your devs do all the QA. In addition to unit testing, they write and maintain their own E2E tests in an open-source framework like Selenium, Cypress, or Playwright.

This approach is easier on your budget than hiring a full-time specialist to manage these tests, but the costs show up in other ways. Specifically, you’ll pay for it with less productivity from your expensive software developers and lower velocity in your release pipeline.

It’s painful and time-consuming for developers who have to dig around in automation code to maintain E2E tests instead of doing what they’re incentivized to do, which is shipping code and new features.

And they’ll be doing a lot of test maintenance, because tests in these frameworks tend to be quite brittle. That means even small, non-functional changes to your app can break things in your test suite, requiring investigation and maintenance.

“We have some Selenium tests. I’m sure you guys are very familiar with just how painful and time-consuming it is to build Selenium tests. And so we don’t build nearly enough of them.”

Startup CTO

In fact, based on our survey of 625 software developers and engineering leaders, 55% of ‌teams using open-source testing frameworks spend at least 20 hours per week on maintaining automated tests.

And, contrary to what you might expect, using AI isn’t correlated with saving any time on test maintenance.

You’ll also need to consider the financial and productivity costs of the tooling and infrastructure you’ll need to provision and manage.

• 3rd-party plugins to extend functionality (e.g., to get more detailed test results or to add visual validation to your tests)
• Test management software to manage and organize your test suite(s)
• Machines on which to run your tests, whether the machines are real or virtual. Want to run tests in parallel? You’ll need more machines. Manage them yourself or pay for a 3rd-party test grid like BrowserStack or Sauce Labs.

Tl;dr

Having your developers use open source test automation frameworks requires tradeoffs in productivity and velocity (and often, developer satisfaction). This approach tends to be an anti-pattern for startups interested in moving fast and keeping their engineers focused on delivering code.

If your goal is continuous delivery, you should consider another option.

2. Hire a QA engineer

Hiring a QA engineer means adding a full-time headcount who applies specialized skills to writing tests and maintaining tests in an open-source framework. Hiring this specialized contributor allows your development team to execute faster, without the drag of E2E test management.

Plus, if you can find an experienced QA person as your first hire, they can implement a testing strategy and process, help you define quality metrics, and otherwise help you implement best-practice methodologies.

A good candidate integrates tightly into your dev team’s workflows to keep releases moving as smoothly and quickly as possible. They’ll develop a deep knowledge of your product, which allows them to be more efficient and effective. And, as part of the team, they’ll have a sense of ownership and accountability that’d be difficult to find in outsourced headcount.

In short, finding the right QA engineering hire for your team can be a boon for both product quality and your peace of mind.

But can you afford it?

The main drawbacks of hiring a QA engineer are the costs of your time, energy, and budget

Time and energy costs

Finding good people is hard — in QA or otherwise. In a survey of startup software engineers we ran, the number one response to “​​What are the main challenges when it comes to hiring QA?” was “Finding good candidates.”

Hiring can eat up a lot of time and energy for any hiring manager. ​​A typical process involves writing and posting job descriptions, screening and interviewing candidates, negotiating, onboarding, training, and management. Once you decide to add headcount, it can take months before someone’s adding value in that position.

Plus, managing a QA person creates a new type of labor for engineering leaders. For them, managing contributors from a non-engineering discipline requires a different set of knowledge and a different way of thinking. QA contributors have different goals, viewpoints, and communication styles.

​​Budget costs

If you do manage to find a good hire, you’ll pay for the privilege. Publicly-available salary databases show that hiring an experienced QA engineer in the U.S. tends to cost north of $100K. 

If a QA hire doesn’t work out, that’s an expensive mistake.

Other hiring risks and costs

And that’s not the only risk when you hire a QA engineer. There’s also single point of failure risk, where the success of an entire function — in this case, quality assurance — depends on the continued contributions of a single person. If you lose your QA engineer, your test suite doesn’t get updated and you lose whatever confidence you had in your test coverage.

Finally, you still have to pay the time, money, and velocity costs of using open source automation frameworks:

• The costs of inevitable bottlenecks caused by test maintenance. (Remember how over half of software teams using open-source testing frameworks 20+ hours per week keeping their automated test suites up-to-date?)
• The costs of provisioning and managing additional tooling and infrastructure.

Tl;dr

If you’ve got the budget and are comfortable with the risks, hiring a QA engineer to automate your tests and lead your QA process can be a winning move.

If you don’t have the budget or appetite for hiring, you still have reasonable options. The difference between hiring QA and hiring in some other categories — like HR or marketing — is that the market is full of practical alternatives for QA. (Specifically, the ones covered in this post.) There’s less need to hire to solve challenges with QA than there is in other disciplines.

So, if hiring a QA engineer isn’t for you, keep reading to learn about your other options.

3. Use a no-/low-code test automation tool

The promise of no- and low-code tools is that anyone on your team can quickly create and automate tests without any technical expertise, at a low cost. 

No- and low-code test automation tools can seem pretty compelling to engineering leaders who don’t have the budget for hiring or outsourcing, don’t want their software developers to get bogged down in open-source frameworks, and don’t want to deal with managing test infrastructure.

They think, “With a codeless testing tool, we don’t need to hire an expensive QA engineer to automate our tests. Our devs can use the tool and move a lot faster than they can with open-source tools because it’s low-code. Maybe we can even have our non-technical product managers help out.”

And, broadly speaking, this is all true. But there’s a catch.

You’ll spend less time on maintenance using no-code tools, but the time spent is still consequential

Our survey data show that teams using no-code to automate their E2E tests spend a lot less time on test creation and maintenance than teams using open-source frameworks.

But even with the best no-code tools, test maintenance costs don’t fall to zero. Someone on your team will still have to dedicate a number of hours per week to keeping your automated tests up to date.

Let’s use our solution, Rainforest QA, as an example. Rainforest started as a self-service, no-code testing tool. Even though it’s quite intuitive and easy to use (just ask our customers), some of our Agile customers shipping fast and frequently found test maintenance to be a time-consuming burden. (Even though they were spending much less time on maintenance than they otherwise would in an open-source tool like Selenium or Cypress.) 

In fact, when teams adopt no-/low-code tools, they tend to underestimate the costs of maintenance. They think they’re saving money by not hiring QA engineers, but their developers spend more time than expected keeping the test suite up to date. So they end up having to hire QA headcount to own test automation in these tools and to free up their development teams to focus on delivering code.

That’s a big part of why we introduced done-for-you test automation services — to take the distracting pain of test maintenance completely off your team’s plate. (More on that in the next section.)

Other common downsides of low- and no-code testing tools 

“Low-code” doesn’t mean “easy to use”

Not all low- and no-code tools are created equal. Some, like Rainforest, are designed from the ground up to be intuitive. So anyone can jump in and quickly confirm or update its plain-English test scripts, no training required.

Other tools, however, are quite complex and difficult to learn. Your team will spend more time trying to get things done with these tools, so look for one where it’s easy to get things done quickly.

Some tools claim “codeless,” but still require coding

Some tools claim to be “codeless” but still require you to code. Specifically, for any test step requiring even a bit of complexity, you’ll need to use custom JavaScript. This excludes non-technical users from creating and maintaining anything but simple tests.

What you want is a simple tool that gives you the option to use code for certain situations, but doesn’t require it.

Most low-/no-code tools test what a computer sees, not what your users see

Most no-/low-code tools — just like all open-source frameworks — interact with and evaluate what happens in the DOM. The DOM is the code behind what happens in the user interface (UI) of your web app.

The DOM approach is a problem because it ignores the visual appearance of what happens in the UI of your app. That is, the approach of these tools ignores what your end users will experience in your app. Instead, they test a proxy for the user experience (i.e., the behind-the-scenes code), which often fails to accurately represent what users will see.  ​​

Some tools offer visual validation through a 3rd-party plugin like Applitools, but in other cases you’re stuck with DOM-based software testing.

Unlike many other tools, Rainforest takes a visual-first (not DOM-first) approach to interacting with and evaluating your app. It tests exactly what your users will experience. We use machine learning to simulate how a human QA tester would evaluate your app — if a visual change is so small that a human QA tester wouldn’t notice or care, Rainforest will pass it, avoiding annoying false-positive test failures.

Plus, our visual-first approach means you can test anything that appears on the screen of a Windows or Mac virtual machine, not just what appears in the browser.

Many tools claim “AI,” but not all deliver on velocity

In testing, “AI” should be considered a marketing ploy until it’s proven to speed up the slowest parts of the automated testing pipeline: test creation and maintenance.

Some AI-powered tools remove more of the creation and maintenance burden than others due to how thoughtfully they’ve integrated the benefits of generative AI (genAI).

For example, using a novel, patent-pending approach, test steps in Rainforest can automatically update themselves — or “self-heal” — to reflect intended changes to your app. This avoids some of the need for humans to intervene and spend time maintaining tests. To our knowledge, Rainforest is the only AI test automation platform where an entire series of test steps created with a genAI prompt can automatically update themselves when your app changes. 

Further, when your Rainforest tests run, if the system can’t find an element based on its default identifier (e.g., a screenshot or DOM selector), the system will look for the element based on its auto-generated AI description (e.g., “Pricing” roughly in the top middle of the screen).

This helps avoid annoying false-positive test failures and maintenance steps whenever you make minor, intentional changes to your app.

Tl;dr

If you don’t have the budget for hiring or outsourcing, no-code tooling can seem appealing. But if you’ve never used these tools before, you’ll almost certainly underestimate the amount of time-consuming test maintenance your team will need to do to keep your automated regression suite up-to-date.

4. Outsource QA

Outsourcing could mean hiring a firm like Applause, or it could mean hiring individual contractors on Fiverr, Upwork, or similar platforms.We surveyed software engineers at startups that have outsourced QA, and they characterized outsourcing as a flexible and scalable way to get more testing done at an affordable cost (while freeing up coders to focus on shipping).

While outsourcing is a faster and cheaper way to scale up your QA resources — relative to making a full-time hire — it comes with some notable drawbacks.

The biggest downsides of traditional QA outsourcing: communication and service quality

The same survey also revealed the biggest challenges with QA outsourcing: communication issues and the (poor) quality of services delivered.

Communication issues

The survey respondents cited two issues around communication: working across distant time zones and language barriers.

Some software teams value around-the-clock testing coverage that comes with having testers spread across different time zones. But most startups don’t like the negative impacts to velocity that come with overseas QA partners. When you have to wait overnight for a response to a request or question, it puts a serious bottleneck in the development process.

And language barriers can make the problem even worse. What if the response you get from your outsourced QA contributors isn’t clear, or maybe they didn’t understand your request or question in the first place? (A survey respondent described it as “a breakdown in understanding.”) Then you have to wait yet another day ‌for a round of back-and-forth to complete.

Service quality issues

Meanwhile, since one of the main value propositions of outsourcing is the relatively low price, outsourcing firms don’t necessarily hire the most skilled or experienced people, but instead the most cost-efficient ones.

This might explain why our survey respondents described service quality issues including “poor work quality,” “a problem meeting deadlines,” and “reporting things that weren’t actually issues.”

Traditional outsourced QA personnel don’t have enough context

That last critique points to another issue reported in the survey: these distant QA contributors don’t tend to join the software team’s meetings or embed in the team’s workflows, so they don’t have the context necessary to meaningfully understand how the team’s product works, let alone the team’s priorities.

That means training and onboarding take longer, ‌contributors require more oversight and corrections, and the work itself isn’t as useful.

You see these critiques — and others, including security issues — reflected when you ask these surveyed engineers to describe the benefits of hiring QA vs. outsourcing.

New QA outsourcing models address the issues with traditional outsourcing

In recent years, a new type of QA outsourcing model has emerged. It aims to deliver the benefits of traditional QA outsourcing (cost and scalability), without all the downsides (issues with communication, service quality, and context). 

The two main companies offering this new model are QA Wolf and Rainforest QA. (See all the best QA Wolf alternatives.)

At Rainforest QA, we dedicate one or more experienced Test Managers to your account to take test creation and maintenance completely off your team’s plate, allowing your devs to focus on shipping. Pricing starts at a quarter of the cost of hiring an experienced QA engineer.

Here are some of the ways Rainforest QA’s test automation services are different from a traditional QA outsourcer.

• Context: Your Test Managers are dedicated to your account so they can deeply learn about your product and priorities. They embed in your workflows and comms tools (like Slack, Microsoft Teams, or Jira) and end up feeling like a part of your team.
• Service quality: These QA professionals have each worked with us and our platform since at least 2017, and have been highly rated by other customers. 
• Communication: For efficient and effective communication, your assigned Test Managers work in or near your time zone and speak fluent English.

You just tell them what you want tested, and they take care of the rest, including screening any test failures. All the work happens in Rainforest’s all-in-one, no-code testing platform, so anyone from your team can quickly verify and even update your test coverage. There are no black boxes. 

“The fact that [our Test Manager] is very specialized, very efficient, knows Rainforest incredibly well, and knows our product incredibly well means she can just focus on this one area and do an incredibly good job of it.”

Robert Guillaume, QA Manager at YNAB

Tl;dr

Outsourcing is an affordable and flexible way to ramp up your startup’s QA resources, at least relative to hiring full-time contributors. The trick is to use an outsourcing partner who won’t undermine your velocity and confidence with communication issues, service-quality issues, and open-source frameworks.

Final thoughts

When you’ve got no QA team, we’ve specifically designed Rainforest to have the big upsides of your other QA options, without their big downsides.

• It’s a lot more affordable than hiring, starting at a quarter of the cost of hiring a QA engineer in the U.S.
• It’s designed to avoid the communication and service-quality downsides of traditional QA outsourcing
• It takes time-consuming test creation and maintenance completely off your plate, so your devs can focus on shipping
• It includes all the features and infrastructure you need to manage your test suites, run them massively in parallel, and get detailed test results for easy debugging

We even offer exploratory testing services to find bugs off the “happy paths” covered by your automated tests.

If you’re looking to solve QA for your startup, we invite you to talk to us about how we can help.