Regression testing—when done well—gives software teams the confidence that their entire application works properly after a code change. But doing regression testing manually is time-consuming, costly, and difficult to scale.
As their applications grow in complexity, many teams end up having to throw more and more resources into regression testing—hiring more QA specialists and waiting longer for them to complete testing with each release cycle.
Automating your regression test suite can help you scale up testing without adding more headcount because you can complete tests in a shorter amount of time and run multiple tests simultaneously.
But choosing the right regression testing tool is tricky.
In this post, we’ll share what we’ve found to be the most important factors to consider when choosing a regression testing tool. Then, we’ll look at how 11 tools address these factors, starting with our tool, Rainforest QA.
Regression Testing Tools
- Rainforest QA
- Selenium
- Appium
- Watir
- IBM Rational Functional Tester
- TestComplete
- Ranorex Studio
- Sahi Pro
- QA Madness
- PerformanceLab
- Katalon Studio
Ready to automate your regression testing? Talk to us about setting up a Rainforest plan that fits your needs.
Factors to Consider When Choosing the Best Regression Testing Tool for Your Team
Software teams use regression testing to make sure any new code changes (e.g., a new feature or design change) haven’t adversely affected other existing functionalities in the application. When you do regression testing manually, you (or a team of testers) simply open the application in a browser and perform a set of actions to make sure everything works properly and looks right. Each test typically follows a script and is designed to test a specific feature or user path.
When you switch to automation, you ideally want to find a tool that helps you mimic this process at scale, and that won’t take more time and effort to set up and maintain than it saves in test execution time.
Here are six factors to consider to determine if a tool will actually save your team time—and help you catch more bugs:
Factor #1: Does the tool test the functionality and the visual layer of your application?
When a person performs a regression test, they interact with the actual UI—the visual representation of the code. This allows them to perform true end-to-end testing because they can check both the functionality (do the buttons work?) and the appearance of the app (do the buttons have the right colors, fonts, shapes, positioning, etc.?).
However, most regression testing tools don’t interact with the visual layer of the UI. When testing functionality, these automated regression testing tools primarily interact with the DOM, or the underlying code of the application, while assuming that the visual layer (what the user sees) is rendering properly.
We’ve written a lot more about this in our article about UI testing tools, but the important thing is there are a variety of situations where an automated test that only interacts with the underlying code will miss a visual bug that would be glaringly obvious to a user (or a human tester).
Since regression testing is often the last layer of testing before release, it’s particularly important that your visual regression testing tool can test both what the end-user will actually see and the functionality of the app for every test step.
Factor #2: Does the tool require programming skills to use?
Traditionally, you had to have programming skills to automate regression testing because the only tool available was Selenium, a framework for writing coded test scripts. But now there are a variety of no-code and low-code tools that let you create Selenium test scripts without writing the code directly. Most of these tools let you record yourself interacting with an app and then translate your actions into test scripts.
Unfortunately, these record-and-playback tools are really limited in the kind of tests they can create. Most teams end up needing to have a programmer who’s familiar with Selenium to create more sophisticated tests.
Luckily, there’s a third option: A no-code automation tool that doesn’t use Selenium at all. Skip to the Rainforest QA review if you want to learn more about that right away.
Factor #3: Can you manage manual and automated tests from the same platform?
Automating your regression test suite is a great way to save time and money, however, not all test cases can be automated. Some test scripts will always require human interpretation and some features may be in an evolving state that would cause an automated test to fail nearly every time it’s run. In these cases, manual testing is the better option.
Having one platform to manage automated and manual tests can save a lot of time (as well as money, in most cases).
Factor #4: Does it make it easy to identify test failures?
When an automated regression test fails, it doesn’t always mean the software has a bug. Sometimes, it just means the test itself is out-of-date. These are called false failures. But figuring out why a test failed can be one of the most time-consuming tasks of automated functional testing. When using a code-based tool, teams often have to sort through many lines of code to find the error. This is antithetical to the kind of agile regression testing that allows teams to ship code fast and frequently.
There are a few different ways tools try to make it easier to debug failures, but we’ve found the most useful option is to watch a video replay of the UI during the test.
Factor #5: Does the tool have built-in features to minimize false failures?
While it’s very helpful to have a tool that makes it easier to figure out why a test failed, it’s even better to have fewer false failures in the first place. Mitigating false failures is one of the biggest challenges of test automation, and each tool has different features to try to prevent tests from breaking.
Some tools are more susceptible to breaking due to any change—no matter how small—in the underlying code, and others will be somewhat immune to minor code changes but more sensitive to changes in the UI.
Each organization has different needs, but we think it’s usually better to use a system that reliably catches changes in the UI, even if that means putting up with a few false failures because of intentional design changes. We’ll explain why in detail below.
Factor #6: Does the tool make it easy to adjust the amount of testing as your day-to-day needs change?
Just about every tool provides a way to write and edit (or “maintain”) test scripts. But you still need a way to execute those tests.
For that, you need infrastructure to run those tests on. Many tools require that you engage a third party’s testing grid (i.e., set of devices upon which you can run tests) if you want to run your tests on anything other than devices that you manage locally.
Many testing platforms also require that you adopt a separate solution if you want a way to manage your test suites. (Which you definitely do want.)
It’s much easier to get started if you’re using an all-in-one tool that includes all of these features, but note that some all-in-one testing platforms charge you separately for each feature.
Once you’ve determined that the testing platform includes the necessary features, you’ll want to consider whether you’ll be locked into a contract. Many tools require monthly or yearly contracts that allow for a set amount of testing each month.
For example, a package might provide 500 tests per month. If you only use 250 tests one month, you’ll still pay for 500 tests. If you need 510 tests during a different month, you’ll either have to opt in for a higher package or skip some tests.
If you want a flexible pricing structure that can respond to weekly, daily, or even hourly changes in how much testing you need, look for a tool with usage-based pricing.
11 Automated Regression Testing Tools
Next, we’ll show how each of the following regression testing tools handles the factors mentioned above.
Rainforest QA
Rainforest QA is a no-code automated UI testing tool for creating regression tests that mimic how users interact with the final, visual layer of your website or app.
Here’s an in-depth look into how we approach each factor we discussed above.
Factor #1: Test the functionality and visual layer simultaneously
Rainforest QA is the only tool on this list that lets you test the functionality and the visual layer of your app on every test step. To do this, we use virtual machines to simulate a user opening a browser and interacting with it just like a human would—by interacting with on-screen elements without touching the underlying code.
Factor #2: Anyone Can Create and Edit Regression Tests
Rather than generating lines of Selenium code for you, Rainforest uses a proprietary test automation approach that makes it a true no-code option to let anyone—even non-technical team members—create, edit, and run any test scripts.
Even if you already have programming skills, writing and maintaining tests in Rainforest can save you a lot of time compared to the maintenance required in coded solutions like Selenium and Cypress.
Learn how Flux saved their engineers time and caught more bugs switching from Cypress to Rainforest.
To create a test step, you select from a dropdown menu of preset actions (such as “click”, “fill”, or “observe”). Then, you take a screenshot of the element you want to apply the action to by clicking and dragging a box over the element on the UI.
Once you’ve created each step of the functionality you want to test, you can preview the actions you’ve defined to verify that the test will do what you intended. When you’re ready to test, you launch it with the click of a button in the Rainforest platform, or a developer can kick it off via our API, CLI, GitHub Action, or CircleCI Orb.
If the test needs to be updated, anyone can look at the test script (see the left-hand column below), know exactly what’s happening, and make edits in the same way the step was created.
Factor #3: Manage Manual and Automated Testing from One Platform
In addition to managing all of your automated testing in Rainforest, you can also access a team of crowd testers to handle test cases that are better suited for manual testing.
Any test written for the automation platform (i.e., by using our Visual Editor) can be converted into plain-English test steps for our manual testers to follow.
You can also choose to write test scripts for manual testing in plain-text rather than with the visual editor:
After you write a test script, you can submit it to the Rainforest community of QA specialists by simply clicking “Run Test.” Our on-demand testing teams provide the fastest test results of any manual testing services available today—returning results in 17 minutes from submission on average. Our specialists are available 24/7, even on holidays, so you get the fastest results possible.
Factor #4: Video Replays to Make Identifying Test Failures a Breeze
Rainforest automatically records a video of every test it runs, whether the test passes or fails. With these videos, you can see the actual point of failure and everything leading up to it. When a test fails, you can compare it to a successful test run to quickly see if anything in the UI changed between runs.
If it’s a real bug that needs to be fixed, Rainforest QA offers a Jira integration, so you can automatically create a ticket for the development team that includes:
- Steps from the failed test
- A screenshot from the failed test step
- HTTP logs
- A link back to the test results in Rainforest to view video replays and browser logs
If it was a manual test that failed, the test results in Rainforest will also show mouse activity and the time spent on each step.
Factor #5: Pixel-matching, Text-matching, and Automatic Retries to Minimize False Failures
As we mentioned earlier, minimizing false failures is one of the biggest challenges for any automated regression testing tool.
Because Rainforest tests the visual layer of the application, its automated test scripts are less susceptible to breaking due to minor code changes that don’t affect the UI.
Regression testing tools that use Selenium (or Cypress or any other code-based testing framework) search for element ID tags in the underlying code of the application to find elements and perform actions on them. If the name of these ID tags changes even slightly, the test will often fail because it can’t find the element—even if the element is still present and the UI didn’t change at all.
Since Rainforest doesn’t touch the underlying code of the website or app, Rainforest tests will only fail if there’s change to the UI.
Of course, a Rainforest test could fail as a result of a minor UI change, such as a button slightly changing shape. But we have a built-in feature to mitigate these failures.
To help make sure a test can still pass even if there was a minor change in the UI, you can use text-matching rather than pixel-matching to find some elements. Text matching examines the text content of an element rather than the appearance.
For example, the buttons below both say “Buy Now” even though the colors and shapes are different.
If text matching is enabled, the test will pass with either version of the button. If the text locator is not enabled, it will only pass if your original screenshot matches the version being tested.
Another common scenario that causes false failures is slow load times in the testing environment. If a test tries to locate an element before the page has fully loaded, the test may fail. To help prevent this type of failure, Rainforest QA lets you:
- Customize wait times between test steps. By default, Rainforest QA puts a two-second delay between test actions to allow an app or web page to finish loading. If the test is unable to locate the element, it will wait an additional two minutes before marking that step as failed. These wait times can be changed in your settings, and you can also add additional wait times to individual test steps.
- Turn on automatic retries. In Rainforest, you can adjust settings so that each failed test is rerun until it passes, or up to the number of retries you define.
Factor #6: All-in-one Solution with Usage-based Pricing
Rainforest QA is an all-in-one platform, which means you’ll have access to everything you need to create, maintain, execute, and manage as many automated tests as you need.
To try our crowdsourced QA specialists, our Startup plan comes with a free 14-day trial of our manual testing service.
You won’t be locked into a contract with our Startup plan. Instead, you only pay for what you use. Specifically, you only pay for the time spent running tests. It doesn’t cost you anything to write, maintain, manage, or even preview.
You can easily scale up or down as needed to stay within your budget while meeting your testing needs.
If you think Rainforest could be the right fit for your team, talk to us about setting up an account.
Selenium/Selenium IDE
Selenium is one of the oldest platforms for automating browsers, and Selenium IDE is the record and playback feature. Many other tools are built on Selenium, so how Selenium handles each factor will be relevant for those tools as well.
- Factor #1: Does the tool test the functionality and the visual layer of your application? Selenium only tests the underlying code of an application—not the visual layer.
- Factor #2: Does the tool require programming skills? All test steps in Selenium are written in code, so you will need programming skills. Although Selenium IDE lets non-technical team members create tests, eventually you’ll need programming skills to maintain tests, create complex testing scenarios, or to understand why tests failed.
- Factor #3: Can you manage manual and automated tests from the same platform?
Selenium is a test automation platform only. - Factor #4: Does the tool have built in features to minimize tests failing due to broken tests?
Selenium doesn’t offer any built-in features to minimize test failures, however, there are ways to minimize test failures yourself by adding lines of code or following strict label management protocols. - Factor #5: Does it make it easy to identify test failures?
You can configure Selenium to take a snapshot of the underlying code at the point of break, but you don’t get video replays. - Factor #6: Does the tool make it easy to increase testing as your software grows and to scale up or down at a moment’s notice to meet daily goals?
Selenium is open-source, so you won’t have to worry about a rigid pricing structure. However, you will eventually need other tools in order to run multiple tests simultaneously, organize your test cases, or improve your workflow. Additionally, you may need to hire additional software engineers as your testing needs outgrow the capacity of your team.
Appium
Since Selenium only automates desktop browsers, Appium modifies Selenium to test browsers on mobile devices. In general, Appium has the same limitations as Selenium.
- Factor#1: Appium only tests the underlying code of an application—not the visual layer.
- Factor #2: All test steps in Appium are written in code, so you will need programming skills.
- Factor #3: Appium is a test automation platform only.
- Factor #4: Appium doesn’t offer any built-in features to minimize test failures.
- Factor #5: You can configure Appium to take a snapshot of the underlying code at the point of break to help identify why a test failed.
- Factor #6: Appium is open source and only provides for writing, editing, and running individual tests.
Watir
Watir (Web Application Testing In Ruby) is an open-source platform built on Ruby (one of the libraries connected to the Selenium WebDriver) but with a more user-friendly user interface than Selenium. Like Appium, Watir runs into many of the same shortcomings as Selenium.
- Factor #1: Watir only tests the functionality of the app by testing the underlying code.
- Factor #2: While lines of code in Selenium have to be written very specifically for complex test cases, Watir provides more flexibility. For example, if your network is slow, you would need to write additional lines of code in Selenium to allow for load time. Watir automatically waits for the page to load. That being said, it still requires programming skills.
- Factor #3: Watir only supports test automation (not manual testing).
- Factor #4 and #5: Watir is very flexible, so you will be able to integrate with nearly any tool to help with failure identification and mitigating broken tests—but there aren’t any built in.
- Factor #6: Watir is open source, however it only supports web testing—it doesn’t support mobile apps, API testing, or cross platform testing. So you may need additional tools if your application expands beyond web testing.
IBM Rational Functional Tester
IBM Rational Functional Tester is a record-and-playback tool that runs on Windows and Linux operating systems. If you’re already using one of IBM’s many other business services, it may be useful to keep your services in one place.
- Factor #1: IBM only tests the functionality of the software, not the visual layer.
- Factor #2: Although IBM offers a record-and-playback feature, it was built to help more technically advanced QA teams write code faster. Only someone with coding experience will be able to use IBM.
- Factor #3: Although IBM offers many different technical tools for software companies, they don’t provide any manual testing services.
- Factor #4: IBM has built-in features that let you take snapshots of the underlying code of your application at any given moment. These snapshots can be used to verify the state of your application before or after an action.
- Factor #5: IBM doesn’t mention any tools for mitigating test failures on their website.
- Factor #6: Some teams find it difficult to scale their testing with IBM because they’re locked into a contract and would have to hire more QA engineers in order to increase testing.
TestComplete
TestComplete by SmartBear offers record-and-playback and object recognition features for writing test scripts in JavaScript, Python, and VBScript. They support testing on most desktop, web, and mobile platforms (including both iOS and Android OS).
- Factor #1: TestComplete only tests the functionality of your app, not the UI.
- Factor #2: TestComplete lets non-technical team members start the test process, but software engineers will eventually need to take over to insert additional lines of code or interpret test results.
- Factor #3: In addition to automated testing, TestComplete lets you write manual test scripts and run them yourself on the platform. However, they don’t provide any assistance with running the tests.
- Factor #4: TestComplete doesn’t have any obvious tools built-in for understanding why a test failed, however, they do integrate with BugSnag which can help you know whether to focus on building new tests or bug fixes.
- Factor #5: TestComplete doesn’t mention any tools for mitigating test failures on their website.
- Factor #6: TestComplete is an all-in-one solution for automated testing. Desktop testing, mobile testing, and web testing are all priced separately with year long contracts.
Ranorex Studio
Ranorex Studio uses a mix of recorded actions and drag-and-drop actions from a preset list or repository for test creation.
- Factor #1: Ranorex Studio tests the functionality of the software and the visual layer via screenshots which can be taken at any point during the test to verify elements of web apps.
- Factor #2: Ranorex tests are built on code (even the screenshots get triggered by lines of code), so you will need coding experience.
- Factor #3: Ranorex only supports test automation.
- Factor #4: Ranorex lets you choose to record a video of a test run, but you’ll have to choose this setting manually and it only saves the most recent test videos.
- Factor #5: Ranorex offers a variety of settings that let you choose how to compare the screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.
- Factor #6: Rather than running tests on a virtual machine like Rainforest QA, Ranorex tests are executed on real devices—either your own local devices or on remote Ranorex devices. So, you’ll need to add additional devices in addition to extra QA engineers as your software grows. However, unlike most tools, Ranorex offers four different tiers of perpetual licenses so the pricing is more flexible than most.
Sahi Pro
Sahi Pro is another code-based tool designed to make coding more accessible to QA beginners with less coding experience.
- Factor #1: Sahi Pro only tests the functionality without any features for capturing or testing the UI during the test.
- Factor #2: Sahi Pro offers record-and-playback and object identification to generate code. You will need some coding experience.
- Factor #3: Sahi Pro is only used for automation, not manual testing.
- Factor #4: Sahi Pro offers a suite analysis feature designed to help pinpoint the lines of code that failed. It does this by comparing successful test runs to the failed test run.
- Factor #5: Sahi Pro provides their own code wrapper to help prevent test brittleness, however, the tests will still be susceptible to changes in the underlying code.
- Factor #6: Sahi Pro is an all-in-one solution with yearly contracts. Additionally, SAP, web, mobile, and desktop testing are all priced separately.
QA Madness
QA Madness is a test automation and manual testing service meaning they take over some or all of the testing for you.
- Factor #1: QA Madness automation only tests the functionality of your app. You can test the UI using their manual testing service, however.
- Factor #2: If you are satisfied with handing over all aspects of your testing to their team (you may want to read why we don’t recommend outsourcing ownership of QA), you won’t need any coding experience. If you prefer to have a hand in your QA, you will need a specialized software development team.
- Factor #3: QA Madness offers both manual and automated testing.
- Factor #4: The QA Madness team includes test failures in their test reports, so you will have some idea as to why they failed.
- Factor #5: You won’t have to worry about any of the tests failing due to broken tests because their team handles it for you. If you’re writing and running some tests yourself, they can do a full testing audit to find weaknesses in your testing and make suggestions on how to make tests less brittle.
- Factor #6: With QA Madness, you won’t have to add any internal team members, but you’ll have to contact them to know how their pricing structure works and if it’s scalable.
PerformanceLab
PerformanceLab specializes in performance testing, however, they also take popular tools and programming languages—namely TestComplete, Selenium, Soap UI, Appium, RFT, QTP/UFT, SAP TAO, and UIAutomator—and develop tools to make them easier to use.
They also let you manage all these tools and languages from one platform (e.g., you could use TestComplete for cross-browser testing, and Appium for mobile testing).
- Factor #1: Most of the tools PerformanceLab works with only test the functionality, but they may be able to work with you if you would like to add on a tool that tests the UI.
- Factor #2: You will need coding skills.
- Factor #3: PerformanceLab offers both a manual testing service and test automation.
- Factor #4 and #5: PerformanceLab doesn’t mention any built-in tools for understanding test failures or mitigating test brittleness.
- Factor #6: PerformanceLab is an all-in-one test automation platform, but you’ll have to contact them for pricing information.
Katalon
Katalon Studio uses the Selenium WebDriver to write tests but was designed to support collaboration between developers and non-developers.
- Factor #1: Katalon Studio doesn’t test the visual layer of the application.
- Factor #2: You will eventually need coding skills to write complex test scenarios or to understand test failures.
- Factor #3: Katalon Studio only offers test automation—not manual testing.
- Factor #4: Katalon Studio provides video replays of failed tests and snapshots of the underlying code at the point of failure to help you identify why the test failed.
- Factor #5: To minimize tests failing without the presence of an actual bug, you can write in additional lines of code or use Katalon’s built in self-healing feature that updates the test to use the closest locator it can find.
- Factor #6: Katalon Studio is an all-in-one test automation solution with yearly contracts.
Improve Your Test Coverage by Automating Your Regression Test Suite with Rainforest QA
Rainforest QA’s codeless automated testing platform lets you build regression tests that automatically check the visual components and the functionality of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams practicing continuous testing by regularly running 500+ automated software tests.
Talk to us about setting up a Rainforest plan that fits your needs.
