In software testing, regression tests are a series of tests run right before releasing a new feature or update to make sure the critical functionalities of your software are still working.
Regression testing is done to make sure a new build didn’t introduce bugs into other, existing features of your software. However, if you don’t make smart decisions about what you’re testing and how you’re testing, regression testing can end up creating more work for your team without leading to significant improvements in software quality.
In this post, we cover the basics of regression testing, how to get started, how to overcome the most common challenges, and additional best practices for improving software quality without wasting time and resources.
Ready to automate your software regression testing? Sign up for Rainforest QA—you can run up to five hours of no-code regression tests for free every month. It’s only $5/hour after that.
Regression testing is typically the last step in the software release process and focuses on testing the main user paths of the entire application rather than the new code specifically.
Regression testing can be performed manually or with an automation platform. While manual testing doesn’t require any special tools or knowledge, it’s slow, time-consuming, and prone to human error. Most teams (especially teams using the agile methodology) eventually need to automate their regression suite in order to keep up with release deadlines without sacrificing software quality.
Who can write, run, and maintain your regression test suite will depend on the type of regression testing tool you choose to use.
Regression testing tools typically fall into two categories: coding frameworks and no-code tools. Coding frameworks, like Selenium, provide a way for developers to automate actions in a browser by writing test scripts using a programming language. Most no-code (or ‘codeless’) testing tools generate coded test scripts for you by recording your interactions with the application or by a library of pre-written actions. However, most are pretty limited in the types of test cases that can be created without having to write additional code. Eventually, most teams need someone with coding experience to create the necessary test cases.
Rainforest QA is the only true no-code tool that lets anyone write complex test cases without writing any code.
To write or edit a Rainforest test, select an action such as ‘click’ or ‘fill.’ Then, take a screenshot of the element you want to apply the action to by clicking and dragging your mouse over the element. Rainforest will use this screenshot to locate, interact with, and verify the element in test runs.
The written steps appear to the left of your screen and the right panel of your screen is where you preview the steps. You can preview test steps as you write or view the entire test once all steps have been written out.
Then, when you’re ready for test execution, you can kick off a complete regression suite or select specific tests from within the Rainforest platform. Developers can also start a test run from within your CI/CD pipeline via our API, CLI, CircleCI Orb, or GitHub Action.
Many teams try to create a test case for every user path in an effort to ensure the entire application is bug-free. However, for most teams, complete test coverage takes too much time and causes delays in the release process.
The good news is that even a small regression suite can significantly improve the quality of your software. However, not all test cases are created equal. If you aren’t testing the right things, more testing just means more work.
To determine which test cases to start with, we like to use the analogy of plowing city streets after a snowstorm. First, you focus on clearing the highly traveled main roads. Then, as time permits before the next snowfall, you clear side streets. In large cities, some streets may never get plowed before the next snow storm.
Likewise, we suggest focusing your testing efforts on the user paths of your application that get used by the most users and cover critical functionalities (such as a login or checkout process). From there, you can add more tests as the software grows or as your quality goals change.
With any type of software testing, one of the main challenges is allocating enough time to complete the entire testing process while keeping up with release schedules. This is particularly true when it comes to regression testing since regression testing typically happens at the end of the software development lifecycle when there’s extra pressure to release quickly.
In this section, we cover the three main areas that slow down regression testing and show how Rainforest QA was designed to mitigate these challenges.
Most functional testing tools search for element locators in the underlying code (a.k.a., the DOM) of a web application to perform test steps. If the element locator is present, the test will move on to the next test step and eventually return a passing test. However, if the element locator is changed even slightly, the test will usually fail because it doesn’t think the element exists.
In most cases, a minor error in the element locator doesn’t affect the functionality of the element or how it appears on the user interface (UI). This means tests may fail even though nothing about the element has changed for the end-user.
If the element is used throughout multiple regression test cases (like a homepage button, for example), a minor change in the underlying code could cause nearly every test in your regression suite to fail. In this case, it will be difficult (or impossible) to tell if there are any real bugs in your software. This means you have to re-run the entire regression suite after fixing the error or skip rerunning the tests and risk shipping bugs to production.
To avoid this, we recommend using a tool like Rainforest QA that tests the UI directly rather than the underlying code. Just like a real user that never touches the DOM, Rainforest tests only respond to changes on the UI. As long as the UI stays the same, Rainforest tests can handle a wide variety of back-end code changes without breaking.
Note: Smoke tests can also help you avoid having to re-run entire regression suites because of one broken element. You can read more about automated smoke testing in this article.
Although choosing the right tool can eliminate some of the false failures that come with code changes, there’s another culprit that can cause false failures: outdated tests. If a test doesn’t get updated to accommodate new functionalities or to reflect new changes to existing functionalities, the test will most likely break because it’s looking for an old version of the software that no longer exists.
That’s why test maintenance is a necessary part of any automated software testing. If tests aren’t up to date, many teams skip running the test altogether or ignore failed tests because they assume it was caused by obsolete test cases and not a real bug.
However, test maintenance can be very time-consuming when using a tool that requires you to make corrections by editing the code directly. With Rainforest QA, anyone can view the test script, understand exactly what’s going to happen, and make the necessary edits without writing any code.
For example, if the ‘Try for free’ button in the screenshot below was changed to ‘Sign up for free,’ anyone can easily see which test step uses that button and take a new screenshot.
This is much easier (and faster) than analyzing and re-writing tons of code. And, you don’t have to rely entirely on developers’ time to keep up with test maintenance.
Understanding why a test failed is the final area that most teams struggle to find the time to complete. If you have a lot of failed tests to sort through, it can be difficult to attend to all of them before the release deadline. This means you either release late or release with potential bugs. Often, the severity of the bugs and the user flows they impact will be the determining factors in that decision.
Rainforest QA makes it really easy to understand test results and categorize failures. Here’s how it works.
Rainforest QA automatically records a video of every test run whether it passes or fails. These video recordings are stored indefinitely and can be used to figure out what changed between test runs.
If you aren’t able to immediately see what caused the failure, you can click on the ‘investigate action’ button for any test step. This gives you more information about how the result was determined, such as the difference between the original screenshot and the closest screenshot found.
You can also download HTTP and browser logs, which are often the key to identifying glitches in the testing environment.
Finally, you can categorize each test failure so the development team knows which bug fixes to prioritize.
Many teams believe that some testing is better than no testing when it comes to the quality of your software. While this is sometimes true, testing the wrong things in the wrong way will cost you a lot of time and resources without seeing much improvement in the quality of your software.
In this section, we cover a few best practices that will help you optimize your testing from the start and ensure you don’t waste time or resources.
You get five free hours of web application testing every month. Then, it’s only $5/hour for the time spent running your tests — not for the time spent or tools used to write, edit, or otherwise manage your test suite. All of that remains free.
You also get a free 14-day trial of our premium features which includes access to our on-demand manual testers (which are only $25/hour after the trial). Whether or not you choose to take advantage of the premium features, you won’t be locked into a contract with a set amount of testing.
Finally, Rainforest QA provides everything you need to write, run, and manage any size of regression suite including:
All of this is included with your account—you only pay for time spent running tests.
Rainforest QA is the only true no-code regression testing tool that mimics real user behavior on the UI. It helps you save time on maintaining tests and triaging test results for faster release times. Rainforest gives you everything you need for effective regression testing while never charging you for more than you need.
If you’d like to explore other options, check out one of our other buying guides, which can help you find the best automated testing tool for your testing priorities and budget:
Learn how to differentiate between various QA testing tools, how to decide which ones you need, and 30+ options to consider.
In this post, we'll cover the most common types of web application testing, which ones to prioritize, and tools to use.
Learn how to write a software test plan. Get started with our practical guide and free template.
Many quality assurance metrics evaluate QA activities rather than results. These are the 5 most helpful QA metrics.