Since 2020, we’ve worked with hundreds of startups transitioning from manual software testing to test automation. Over that time, we’ve repeatedly seen teams underestimate the costs of test automation maintenance.
Specifically, teams don’t anticipate how much time they’ll have to spend on maintenance, and how disruptive it’ll be to their productivity and release pipelines.
To quantify these costs, we surveyed over 600 software developers and engineering leaders about their experiences with test automation. We’ll share the results in this post.
But first…
A quick primer on test maintenance
When you push a code change to your app, it can break your tests.
For example, if you change your “Sign in” button labels to “Log in” button labels and some of your tests are still looking for “Sign In” buttons, then those tests will fail.
These broken tests present false-positive failures, because the failures reflect a problem with the tests, not a problem with your application. These failures can result from any sort of change to your application that isn’t represented in your tests.
Test maintenance is the work done to update automated tests to reflect the latest, intended version of your app. The more you update your app, the more someone needs to either (a) proactively update your tests or (b) investigate and then fix your failing tests.
If you don’t perform regular maintenance, your automated test suite will quickly fall out of date. Then it won’t effectively do its job — finding bugs and other issues in your app.
As the data will show, test maintenance can represent a time-consuming chunk of your automation efforts. And, between finding the root cause of test failures and then updating test automation scripts, it’s no-one’s idea of fun. It can take your engineers away from shipping code, and it often bottlenecks release pipelines.
The data: How much time does test maintenance actually take?
Creating and maintaining tests in open-source automation frameworks like Selenium, Cypress, and Playwright can be a drag on agile development teams aiming to ship fast and frequently. (Selenium is notoriously the most painful to maintain.)
They all require someone with technical skills — like a developer or QA engineer — to dig around in code to find the offending part(s) of the test to fix.
In fact, software developers often deliberately neglect to update their end-to-end (E2E) automated test scripts in favor of delivering more code. After all, most devs are incentivized to ship, and software teams have to ruthlessly prioritize their time and resources.
No- and low-code test automation tools can make maintenance a bit easier, but they still carry maintenance costs. (More on that, shortly.)
According to our survey results, of the teams using open-source testing frameworks like Selenium, Cypress, and Playwright, 55% spend at least 20 hours per week creating and maintaining automated tests.
But that’s looking across all teams. In practice, the more complex your app is (measured by the number of non-QA engineers on your team), the more testing and test maintenance you’ll need to do.And, at least for the moment, the data show that using AI doesn’t provide any relief for teams automating tests with open-source frameworks.
In fact, open-source teams using AI spend slightly more time on test maintenance!
What about using no-/low-code tools?
The data show that using an intuitive, no-code tool like the Rainforest QA platform is up to 3x faster than using an open-source framework.
Teams using no-code create and maintain automated test coverage in much less time than teams using open source.
For teams using open-source frameworks, the survey data showed that small dev teams in particular struggle to keep their automated test suites up to date. (Probably because small teams are the least likely to have the necessary procedures and policies in place. When maintenance isn’t enforced, it can get ignored in favor of shipping code.)
Because it’s easier to create and update tests in a no-code tool, small teams are more able to keep their test suites updated, which makes the suites more reliable.
See more new research on the test automation technologies and practices that increase shipping velocity.
Of course, even when you’re using an optimal no-code tool, the cost of test maintenance is non-zero. Someone(s) on your team will still need to dedicate some number of hours every week to keeping tests updated.
If anything, the tendency to underestimate the costs of test maintenance is worse when a software team is adopting a no- or low-code solution. Just because a tool is easier to use — much easier to use, in some cases — doesn’t mean the costs of maintaining automated tests go away.
We saw this struggle all the time over the several years we sold a no-code test automation tool (before we enhanced it with AI and test automation services). We even have customers who signed on for our test automation services because they couldn’t stay on top of maintenance themselves with the version of our easy-to-use, no-code tool lacking AI.
It’s also important to note that not all no- and low-code tools are created equal — some are much more complex than others, making them more difficult to learn and use. According to our customers, TestComplete and Katalon are among the most complex. (On the other hand, you can read for yourself what they have to say about Rainforest.)
How to avoid the costs of test maintenance
Clearly there are significant time costs involved in test automation maintenance. And if you don’t have a QA team, there are other ways you want your engineers to spend their time and attention.
So here are the ways you can automate your end-to-end tests while keeping maintenance to an absolute minimum.
1. Use a test automation tool that makes maintenance easier
As we’ve established, test maintenance is generally easier and faster in no-/low-code tools than it is in open-source frameworks.
But “no-code” isn’t the same as “intuitive,” so make sure you’re choosing a solution that’s as easy as possible for your team to learn and use. We’ve designed Rainforest to be the most intuitive no-code test automation solution on the market. Anyone from your team can automate, interpret, and maintain automated tests without any training.
Rainforest QA is an all-in-one test automation platform. It includes everything you need for the testing workflow, including an API and CLI to integrate tests into your CI/CD pipeline.
2. Let AI handle maintenance for you
There are several ways in which AI-powered testing in Rainforest QA helps you avoid unnecessary maintenance efforts and increase your release velocity.
Reliably identifying elements in your app
In most automated testing tools and frameworks, a slight change to an element in your app — like a button or form field — can break the relevant tests. Whether the element’s visual appearance or naming convention in the behind-the-scenes DOM code has changed, a test relying on a single identifier (or “locator”) for the element can fail when it can no longer find a match for that identifier. (Automated tests that frequently fail due to minor changes like these even have a name: “brittle tests.”)
Tests in Rainforest are more robust because they rely on three different types of identifiers to locate elements in your web application. These include visual appearance, an automatically-identified DOM locator, and an element description automatically generated by our AI. A change in your app in any one of these aspects won’t break your test.
Automatically updating tests
When you make intended changes to your web app’s appearance or functionality, Rainforest’s AI can often automatically update — or “heal” — the relevant test steps so you don’t have to.
Watch a six-minute demo of Rainforest AI, including its auto-healing capabilities.
Determining the app changes that matter
Rainforest’s visual validation algorithm has been trained via machine learning to simulate the judgment of a human tester. If a visual change in your app’s user interface is so minor that a human tester wouldn’t notice or care, Rainforest will allow it to pass.
So, inconsequential visual changes to your app won’t trigger false-positive test failures.
3. Write fewer, shorter test scripts
This is a core component of the testing best practices and methodologies we recommend to our customers.
Fewer tests
Designing automated test coverage is largely about prioritization. Apply the Snowplow Strategy and focus your automated regression tests on the most critical user paths and functionalities in your app.
Once tests cover the most important flows, you’ll see diminishing returns on additional tests. At some point, you’re adding future, recurring maintenance costs in return for having confidence in functionalities that aren’t all that important. As part of your testing process, recognize when that tradeoff isn’t worthwhile.
Some rules of thumb:
- Don’t waste time testing functionalities that you wouldn’t fix right away if they broke.
- Treat your test coverage priorities like a product roadmap: “nice to haves” go in the backlog.
Shorter tests
The longer a test is, the more points of potential (false-positive) failure it presents.
So, make each test script as long as necessary and as short as possible. You’ll enjoy faster test executions, an easier time debugging tests that fail, and ultimately, lower maintenance efforts.
4. Don’t use E2E test automation for new features still in active development
Unit testing and integration testing are useful (and sensible!) early in the software development life cycle. But it doesn’t make sense to create automated E2E tests for a new feature until it’s largely stable.
Otherwise, you’ll spend a disproportionate amount of time updating your tests to keep up with the feature. For these test cases, the flexibility of manual testing makes it a better choice.
5. Look for opportunities to re-use test snippets
Most test automation frameworks and tools will offer a way to create “snippets” of test scripts that you can re-use across different tests. You can update a snippet once, and the change will replicate across all the tests where the snippet is embedded.
In Rainforest, we call these “reusable snippets,” and they’re helpful for cutting down the time needed to create new tests and maintain existing ones. Common test cases for them include testing login and account creation flows.
Without these snippets, if you updated your login sequence, for example, you’d have to update every automated test case that covers your login sequence. With snippets, you only have to make an update in one place and it gets applied across the board.
As you’re writing your tests, be mindful of flows that tend to repeat — these will be good opportunities to save time with reusable snippets.
6. Make debugging easier with detailed test results
Part of the pain of the test maintenance process starts with debugging — figuring out what needs to be fixed.
Detailed test results will give you enough information to quickly figure out where and how each test is failing, and even suggest why it’s failing.
Whatever test automation solution you use, we recommend — at a minimum — making sure you have access to the types of test result details we provide in Rainforest, including:
- Steps to reproduce failures
- Video recordings of tests
- Browser and HTTP logs
7. Use a done-for-you test automation service
If you want your team to avoid the costs of test maintenance altogether, Rainforest QA’s test automation service will handle 100% of test creation and maintenance for you. So your team can focus on higher-value work.
As part of the service, we’ll dedicate one or more experienced Test Managers to your account. Simply tell them what you want tested — type a message or record a video — and they’ll take care of the rest.
- They’ll embed in your workflow and comms tools (e.g., Linear, Jira, Slack, Teams), deeply learning your product and priorities.
- They work in or near your time zone and speak fluent English, so communication is easy and efficient.
- They’ve all worked with us and our platform as quality assurance professionals since at least 2017 and undergo regular training and evaluations.
- They’re all consistently highly rated by customers — otherwise they wouldn’t still be with us.
The Test Managers also proactively review your test failures, updating the tests that need it and sending you the legitimate bugs.
Plus, they work within Rainforest’s no-code, AI-accelerated platform, so they’ll be able to create and maintain your automated tests as quickly as possible.
Pricing starts at a quarter of the cost of hiring an experienced QA engineer. Talk to us to learn more and get a personalized demo.
