Regression testing—when done well—gives software teams the confidence that their entire application works properly after a code change. But doing regression testing manually is time-consuming, costly, and difficult to scale.
As their applications grow in complexity, many teams end up having to throw more and more resources into regression testing—hiring more QA specialists and waiting longer for them to complete testing with each release cycle.
Automating your regression test suite can help you scale up testing without adding more headcount because you can complete tests in a shorter amount of time and run multiple tests simultaneously.
But choosing the right regression testing tool is tricky.
In this post, we’ll share what we’ve found to be the most important factors to consider when choosing a regression testing tool. Then, we’ll look at how 11 tools address these factors, starting with our tool, Rainforest QA.
Regression Testing Tools
Ready to automate your regression testing? Sign up for Rainforest QA—you can run up to five hours of no-code regression tests for free every month. It’s only $5/hour after that.
Software teams use regression testing to make sure any new code changes (e.g., a new feature or design change) haven’t adversely affected other existing functionalities in the application. When you do regression testing manually, you (or a team of testers) simply open the application in a browser and perform a set of actions to make sure everything works properly and looks right. Each test typically follows a script and is designed to test a specific feature or user path.
When you switch to automation, you ideally want to find a tool that helps you mimic this process at scale, and that won’t take more time and effort to set up and maintain than it saves in test execution time.
Here are six factors to consider to determine if a tool will actually save your team time—and help you catch more bugs:
When a person performs a regression test, they interact with the actual UI—the visual representation of the code. This allows them to perform true end-to-end testing because they can check both the functionality (do the buttons work?) and the appearance of the app (do the buttons have the right colors, fonts, shapes, positioning, etc.?).
However, most regression testing tools don’t interact with the visual layer of the UI. When testing functionality, these automated regression testing tools primarily interact with the DOM, or the underlying code of the application, while assuming that the visual layer (what the user sees) is rendering properly.
We’ve written a lot more about this in our article about UI testing tools, but the important thing is there are a variety of situations where an automated test that only interacts with the underlying code will miss a visual bug that would be glaringly obvious to a user (or a human tester).
Since regression testing is often the last layer of testing before release, it’s particularly important that your regression testing tool can test both what the end-user will actually see and the functionality of the app for every test step.
Traditionally, you had to have programming skills to do automated regression testing because the only tool available was Selenium, a framework for writing coded test scripts. But now there are a variety of no-code and low-code tools that let you create Selenium test scripts without writing the code directly. Most of these tools let you record yourself interacting with an app and then translate your actions into test scripts.
Unfortunately, these record-and-playback tools are really limited in the kind of tests they can create. Most teams end up needing to have a programmer who’s familiar with Selenium to create more sophisticated tests.
Luckily, there’s a third option: A no-code automation tool that doesn’t use Selenium at all. Skip to the Rainforest QA review if you want to learn more about that right away.
Automating your regression test suite is a great way to save time and money, however, not all test cases can be automated. Some test scripts will always require human interpretation and some features may be in an evolving state that would cause an automated test to fail nearly every time it’s run. In these cases, manual testing is the better option.
Having one platform to manage automated and manual tests can save a lot of time (as well as money, in most cases).
When an automated regression test fails, it doesn’t always mean the software has a bug. Sometimes, it just means the test itself is out-of-date. These are called false failures. But figuring out why a test failed can be one of the most time-consuming tasks of automated functional testing. When using a code-based tool, teams often have to sort through many lines of code to find the error.
There are a few different ways tools try to make it easier to debug failures, but we’ve found the most useful option is to watch a video replay of the UI during the test.
While it’s very helpful to have a tool that makes it easier to figure out why a test failed, it’s even better to have fewer false failures in the first place. Mitigating false failures is one of the biggest challenges of test automation, and each tool has different features to try to prevent tests from breaking.
Some tools are more susceptible to breaking due to any change—no matter how small—in the underlying code, and others will be somewhat immune to minor code changes but more sensitive to changes in the UI.
Each organization has different needs, but we think it’s usually better to use a system that reliably catches changes in the UI, even if that means putting up with a few false failures because of intentional design changes. We’ll explain why in detail below.
Just about every tool provides a way to write and edit (or “maintain”) test scripts. But you still need a way to execute those tests.
For that, you need infrastructure to run those tests on. Many tools require that you engage a third party’s testing grid (i.e., set of devices upon which you can run tests) if you want to run your tests on anything other than devices that you manage locally.
Many testing platforms also require that you adopt a separate solution if you want a way to manage your test suites. (Which you definitely do want.)
It’s much easier to get started if you’re using an all-in-one tool that includes all of these features, but note that some all-in-one testing platforms charge you separately for each feature.
Once you’ve determined that the testing platform includes the necessary features, you’ll want to consider whether you’ll be locked into a contract. Many tools require monthly or yearly contracts that allow for a set amount of testing each month.
For example, a package might provide 500 tests per month. If you only use 250 tests one month, you’ll still pay for 500 tests. If you need 510 tests during a different month, you’ll either have to opt in for a higher package or skip some tests.
If you want a flexible pricing structure that can respond to weekly, daily, or even hourly changes in how much testing you need, look for a tool with usage-based pricing.
Next, we’ll show how each of the following regression testing tools handles the factors mentioned above.
Rainforest QA is a no-code automated UI testing tool for creating regression tests that mimic how users interact with the final, visual layer of your website or app.
Here’s an in-depth look into how we approach each factor we discussed above.
Rainforest QA is the only tool on this list that lets you test the functionality and the visual layer of your app on every test step. To do this, we use virtual machines to simulate a user opening a browser and interacting with it just like a human would—by interacting with on-screen elements without touching the underlying code.
Rather than generating lines of Selenium code for you, Rainforest uses a proprietary test automation approach that makes it a true no-code option to let anyone—even non-technical team members—create, edit, and run any test scripts.
Even if you already have programming skills, writing and maintaining tests in Rainforest can save you a lot of time compared to the maintenance required in coded solutions like Selenium and Cypress.
To create a test step, you select from a dropdown menu of preset actions (such as “click”, “fill”, or “observe”). Then, you take a screenshot of the element you want to apply the action to by clicking and dragging a box over the element on the UI.
Once you’ve created each step of the functionality you want to test, you can preview the actions you’ve defined to verify that the test will do what you intended. When you’re ready to test, you launch it with the click of a button in the Rainforest platform, or a developer can kick it off via our API, CLI, GitHub Action, or CircleCI Orb.
If the test needs to be updated, anyone can look at the test script (see the left-hand column below), know exactly what’s happening, and make edits in the same way the step was created.
In addition to managing all of your automated testing in Rainforest, you can also access a team of crowd testers to handle test cases that are better suited for manual testing.
Any test written for the automation platform (i.e., by using our Visual Editor) can be converted into plain-English test steps for our manual testers to follow.
You can also choose to write test scripts for manual testing in plain-text rather than with the visual editor:
After you write a test script, you can submit it to the Rainforest community of QA specialists by simply clicking “Run Test.” Our on-demand testing teams provide the fastest test results of any manual testing services available today—returning results in 17 minutes from submission on average. Our specialists are available 24/7, even on holidays, so you get the fastest results possible.
Rainforest automatically records a video of every test it runs, whether the test passes or fails. With these videos, you can see the actual point of failure and everything leading up to it. When a test fails, you can compare it to a successful test run to quickly see if anything in the UI changed between runs.
If it’s a real bug that needs to be fixed, Rainforest QA offers a Jira integration, so you can automatically create a ticket for the development team that includes:
If it was a manual test that failed, the test results in Rainforest will also show mouse activity and the time spent on each step.
As we mentioned earlier, minimizing false failures is one of the biggest challenges for any automated regression testing tool.
Because Rainforest tests the visual layer of the application, its automated test scripts are less susceptible to breaking due to minor code changes that don’t affect the UI.
Regression testing tools that use Selenium (or Cypress or any other code-based testing framework) search for element ID tags in the underlying code of the application to find elements and perform actions on them. If the name of these ID tags changes even slightly, the test will often fail because it can’t find the element—even if the element is still present and the UI didn’t change at all.
Since Rainforest doesn’t touch the underlying code of the website or app, Rainforest tests will only fail if there’s change to the UI.
Of course, a Rainforest test could fail as a result of a minor UI change, such as a button slightly changing shape. But we have a built-in feature to mitigate these failures.
To help make sure a test can still pass even if there was a minor change in the UI, you can use text-matching rather than pixel-matching to find some elements. Text matching examines the text content of an element rather than the appearance.
For example, the buttons below both say “Buy Now” even though the colors and shapes are different.
If text matching is enabled, the test will pass with either version of the button. If the text locator is not enabled, it will only pass if your original screenshot matches the version being tested.
Another common scenario that causes false failures is slow load times in the testing environment. If a test tries to locate an element before the page has fully loaded, the test may fail. To help prevent this type of failure, Rainforest QA lets you:
Rainforest QA is an all-in-one platform, which means you’ll have access to everything you need to create, maintain, execute, and manage as many automated tests as you need. You get five hours of test automation for free every month, and it’s only $5/hr after that.
To try our crowdsourced QA specialists, our Professional plan comes with a free 14-day trial of our manual testing service. Then it’s just $25/hr.
You won’t be locked into a contract with our Professional plan. Instead, you only pay for what you use. Specifically, you only pay for the time spent running tests. It doesn’t cost you anything to write, maintain, manage, or even preview.
You can easily scale up or down as needed to stay within your budget while meeting your testing needs.
If you think Rainforest could be the right fit for your team, sign up for free today.
Selenium is one of the oldest platforms for automating browsers, and Selenium IDE is the record and playback feature. Many other tools are built on Selenium, so how Selenium handles each factor will be relevant for those tools as well.
Since Selenium only automates desktop browsers, Appium modifies Selenium to test browsers on mobile devices. In general, Appium has the same limitations as Selenium.
Watir (Web Application Testing In Ruby) is an open-source platform built on Ruby (one of the libraries connected to the Selenium WebDriver) but with a more user-friendly user interface than Selenium. Like Appium, Watir runs into many of the same shortcomings as Selenium.
IBM Rational Functional Tester is a record-and-playback tool that runs on Windows and Linux operating systems. If you’re already using one of IBM’s many other business services, it may be useful to keep your services in one place.
Ranorex Studio uses a mix of recorded actions and drag-and-drop actions from a preset list or repository for test creation.
Sahi Pro is another code-based tool designed to make coding more accessible to QA beginners with less coding experience.
QA Madness is a test automation and manual testing service meaning they take over some or all of the testing for you.
PerformanceLab specializes in performance testing, however, they also take popular tools and programming languages—namely TestComplete, Selenium, Soap UI, Appium, RFT, QTP/UFT, SAP TAO, and UIAutomator—and develop tools to make them easier to use.
They also let you manage all these tools and languages from one platform (e.g., you could use TestComplete for cross-browser testing, and Appium for mobile testing).
Katalon Studio uses the Selenium WebDriver to write tests but was designed to support collaboration between developers and non-developers.
Rainforest QA’s codeless automated testing platform lets you build regression tests that automatically check the visual components and the functionality of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams practicing continuous testing by regularly running 500+ automated software tests.
Get started with Rainforest QA for free. You can run five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
Rainforest’s CTO and co-founder, Russell Smith, and CIO, Derek Choy, recently discussed test automation limitations technical leaders need to know about before biting the bullet. In this post we cover the 3 biggest limitations to consider.
We provide practical guidance on how to start automation testing from scratch and how to choose a test automation tool.
Today, we’ll discuss the pros and cons of using testing automation as part of your QA strategy, and how to gain the most from its benefits while minimizing its downsides.
The right codeless test automation tools make it easy for anyone to speed up their QA process without trading quality.