Automated visual regression testing tools check for style issues and problems with the visual layer of an app or website.
The benefit of these tools is that they can catch issues on the visual layer (the user interface of the website or app, which customers see) that may get missed by test scripts that only interact with the underlying code (the DOM).
Common visual regressions include things like overlapping modules, hidden buttons, elements that render off-screen, brand styles reverting to old versions, and a host of minor or major CSS issues that can get missed by both human QA testers and automation scripts.
For example, here’s a screenshot of Google’s homepage with an error. The “I’m Feeling Lucky” button is missing. Now, if the button had truly been removed from the page, it wouldn’t be present in the underlying code of the page, and most automated regression testing tools would catch the bug.
However, if you scroll to the right within the browser, you’ll find that the button is, in fact, present, but it’s rendering off-screen because someone (in this hypothetical situation) accidentally used the wrong setting in some CSS—a not-uncommon occurrence.
Most regression testing tools that only interact with the underlying code of the page without a visual validation step would let this bug escape to production.
While a human tester would immediately notice the visual bug in the example above, take a look at the two screenshots below. Does the difference between the two immediately jump out at you?
When you're comparing the images side by side, you might notice that the space between the buttons is too big (by 10 pixels) in the second image. But for most human users (or testers), this kind of subtle change would be easy to miss. If you don’t promptly correct minor visual regressions like this, similar errors in the future can compound until you’re looking at a screen with an inconsistent layout.
Human testers also tend to overlook minor brand inconsistencies. If an error in development caused the app to revert to an old stylesheet, the buttons could appear with outdated brand colors or styles, eroding your brand (and in some cases, even damaging your company’s trademark rights).
The one way to consistently catch visual bugs like these without slowing down your testing process is to use a visual regression testing tool that seamlessly fits into your automated testing workflow.
Over the last few years, there has been an explosion in development of visual testing tools, and the differences between them aren’t always obvious.
In this post, we’re going to cover what to look for in a visual regression testing tool and share what we believe are the five best tools available today.
Sign up for Rainforest QA—you can run up to five hours of no-code, automated visual regression tests for free, every month. It’s only $5/hour after that.
If you’re going to add a visual regression testing tool to your mix, there are a few things to look for to decide if the tool will meet your needs.
Automated testing tools can be divided into two categories:
Visual regression testing tools also fall into those two categories. If you don’t have programmers on your QA team, you’ll want to look for a no-code tool.
A visual regression testing tool needs to reliably trigger a failure whenever it finds visual regressions on the front end that affect the user experience, while ignoring visual changes that aren’t bugs.
It sounds simple, but it’s actually quite hard to pull off. Most tools, especially the open-source options, simply compare screenshots of the entire UI and will flag every single change, making them too sensitive for most users.
Some visual testing tools are standalone testing solutions, meaning you can create, run, and edit tests with them, but they only do visual validation, so they’re meant to be used in addition to some other solution for regression testing. Other tools are just plug-ins designed to add visual validation to your existing automation framework. They usually do this by triggering a screenshot step in the middle of an existing automated test script.
In contrast, our tool, Rainforest, is an all-in-one solution for automating both standard and visual regression testing. (But more on that later.)
There are some free open-source tools that you’ll never have to pay for, but open-source tools don’t come with customer support teams, typically require programming skills to use, and are usually more limited in what they can do.
On the commercial side, many tools structure their pricing into tiers that can force you to upgrade to a much more expensive tier when you max out the starter plan. And some tools lock you into long-term contracts. An ideal tool will let you pay for only as much testing as you need, and scale up or down freely as your needs change.
Now we’ll share our five favorite visual testing tools, starting with our tool, Rainforest QA.
Rainforest QA is a no-code automated UI testing tool with a drag-and-drop editor that lets you create automated tests that mimic how users interact with the final, visual layer of your website or app.
Here’s a deep dive into the features that Rainforest customers love.
Rainforest’s automation navigates through the app on the visual layer just like a user would—looking for buttons, clicking on links, putting text into forms, etc.
It validates the visual content by looking for precise screenshots of on-page elements (like a sign-up button), while also validating the overall functionality of the page by completing typical user actions.
Because it’s no-code, anyone, including non-developers, can set up, edit, and run tests in Rainforest.
To create a test, you choose from a list of available actions in a drop-down menu in the test editor (click, fill, press key, etc.). After you choose the first action, you click-and-drag with the mouse to create a small screenshot to indicate where on the page the action will happen (on a button, on a hyperlink, in a form, etc.).
You can add as many actions as you need to complete the test. You can even test things like sending and checking emails, activating new user accounts, and downloading files.
One of the action options you can choose in Rainforest is “Observe.” The “Observe” action will verify that an element is both visible and looks like the screenshot you took when you created the test. This is important for catching visual bugs.
If the color has changed on a button, or another element is overlapping the element you want to check, the test will fail. But notably, the Observe action doesn’t observe the entire UI—it only checks the box of pixels that you defined in the test. That means that there can be minor changes to other sections of the page, and the test will pass as long as all the elements you included in the test were present.
But sometimes, Rainforest’s exact pixel-matching can still be too sensitive for your needs. Let’s say you updated your stylesheets to reflect a rebrand, but you forgot to update your test suite. Since all of the buttons now have different styling, the visual changes would trigger failures across your test suite.
To provide more flexibility in situations like this, Rainforest offers text matching to allow tests to pass even if there have been minor changes to visual elements. You can toggle text matching on or off for any test step.
With text matching turned on, Rainforest will search for the text of the button or element in your screenshot if it can’t find an exact pixel match of the screenshotted element. If it finds the matching text anywhere on the screen, the test will still be able to proceed. Even so, you’ll be notified in the test results page that there was a visual change (see the Element Mismatch warning in the screenshot below), letting you know that you need to update the screenshots.
And if you didn’t intend to change the styling of your buttons, you’ll have a chance to investigate the change before pushing code to production.
Rainforest QA is an all-in-one test automation solution, which means it has built-in tools to allow users to create, run, and manage as many automated tests as they need.
With Rainforest, you don’t need to pay for additional services like a testing grid, test case management tools, or team collaboration tools just to manage your automated testing.
Rainforest QA includes:
Rainforest automatically records a video of every test it runs (whether the test passes or fails). When a test fails, you can review the video to see the actual point of failure and everything leading up to (and following) it.
With most automated software testing tools, you’d have to sort through lines of code in order to identify why a test failed. With Rainforest, anyone can look at test results, know exactly what happened in the test, and quickly understand why the test failed.
Rainforest charges users based on how many minutes of testing they do each month.
The Professional plan, which gives you access to all but the most advanced testing features, lets you run up to five hours of test automation every month—at no cost. It’s only $5/hour after that, so you don’t have to worry about moving into a more expensive tier as your testing program grows. In this plan, you’re billed monthly based on your usage, and there are no long-term commitments. (For organizations doing a lot of testing and looking for even more features and support, we offer Enterprise plans.)
See for yourself why Rainforest QA is the best visual regression testing tool. Click here to get started with Rainforest QA for free.
While we think Rainforest is the best overall visual testing tool, you might consider some of the following tools if you’re absolutely committed to using Selenium (but you might want to read why many organizations are looking for Selenium alternatives); if you’re exclusively testing mobile apps; if you want to use an open-source tool; or if you only want to validate the visual rendering of a single web page.
How It Works: Working within your favorite test runner (Applitools supports more than 40 testing frameworks and languages, so it’s compatible with almost any development environment), you can insert visual checkpoints with just one line of code (generated for you by Applitools) into any step in an automation test script.
Like many other visual regression tools, Applitools takes a screenshot of the entire UI when you write the test, and sets it as the baseline. The next time you run the test, it will take a new screenshot and compare it to the baseline, using an AI-powered image comparison algorithm to filter out the noise.
What We Like: The AI aspect of Applitools helps it overcome the obvious false positives/noise problems of tools that just take a screenshot of the entire UI and alert you to ANY changes. It has four layers of sensitivity, called match levels:
If those four match levels don’t work for you, you can also carve out sections of the page that you know change too often for the AI to handle (dynamic content, moving content, features in beta, etc.). However, we think it’s more efficient to specify what you DO want to evaluate (as you would with Rainforest’s visual editor), rather than having to carve out the stuff you DON’T. If you carve out too much of the page, your test effectively has a bunch of holes where bugs can slip through.
What We Don’t Like: Unfortunately, Applitools is extremely limited in functionality if you don’t have programming skills. It’s not an accessible tool for non-technical QA team members. You can use Applitools with the Selenium IDE record-and-replay plugin to add visual checkpoints into tests, but these tests are extremely limited (you can’t do database testing, you don’t get detailed test reports, and it can’t handle conditional statements).
Pricing: The Free Forever plan offers one user and 100 visual checkpoints per month. If you want to have multiple users and CI/CD integration, you’ll need to sign a 1-year contract, and they don’t list pricing on their website, so you’ll have to inquire with the sales team.
How It Works: Kobiton focuses on mobile testing and real-device testing, but when it comes to visual testing, it’s really trying to help you maintain a consistent visual output across all devices and form factors. It offers scriptless manual testing: you can record a manual test and it converts it into an Appium script (Selenium scripts are also available). You’ll get automatic notifications of visual anomalies.
What We Like: Kobiton is one of the only options that is built for mobile first, and mobile is hard to do well. Also, they go beyond just pointing out differences to making suggestions for improvements based on their analysis of the Top 50 apps in the world. Additionally, users say the screenshot function makes gathering screenshots for different screen sizes quick and easy, which helps them get apps up on app stores faster.
What We Don’t Like: The only option Kobiton has for filtering noise and handling dynamic content is a setting to tell the AI to “navigate to the best matching element,” but there’s no way to know how consistently it will be able to identify the best match.
Pricing: Kobiton’s pricing is based on minutes of test run-time, like Rainforest. However, the cheapest plan starts at $500 a year for 500 minutes/month.
How It Works: PhantomCSS is built on top of PhantomJS/CasperJS. It can define the steps of the test script so you can navigate around your app, and then it takes screenshots of each step and uses ResembleJS to compare the images to a baseline. You’ll get notified of every difference. It has no tools for minimizing false positives.
What We Like: PhantomCSS has one of the most active and healthy communities on GitHub for any open-source visual regression testing tool. That’s very important for an open-source because that community is your only source for support when you run into trouble using it.
How It works: Percy is a tool owned by Browserstack. If you’re already using Browserstack for cross-browser testing or any of its other automation tools for functional testing, it integrates smoothly with your existing unit and integration tests. When a visual test is kicked off, Percy renders UI snapshots across different browsers and responsive widths. Through pixel-by-pixel baseline comparisons, Percy detects and highlights relevant visual changes that have appeared in each page and state of your UI.
What We Like: Percy is a really fast way to test how a single web page looks on all of the browsers and device combinations. And it can test how it looks as the width of the screen changes, to find any widths where the responsive design is broken (overlapping elements, huge white spaces, etc.). Users mention that the snapshot stabilization feature can minimize false positives from the tiny differences between the ways different browsers render an app.
What We Don’t Like: Pixel-by-pixel image comparison is too sensitive for functional testing of web applications where you’re navigating through multiple screens, inputting information, evaluating results, etc. The noise from all of the false positives will quickly make it unusable for testing more than one page at a time.
Pricing: Percy has a free tier which provides 5,000 screenshots/month, unlimited team members, and unlimited projects. The Professional Plan starts at $149/month and includes unlimited users, cross-browser support, automatic browser upgrades, responsive visual testing, source code integrations, and 1 year build history.
With a visual regression testing tool, you can be confident your app is free of visual bugs that disrupt the user experience or erode your brand image.
Rainforest QA’s no-code automated testing platform lets you build regression tests that automatically check the visual components of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams regularly running 500+ automated software tests.
Get started with Rainforest QA for free. You can run up to five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
We provide a detailed review with recommendations of the top 10 software testing tools available today.
In this post, we define what 'smoke testing software' means, provide specific examples of the types of tests to include, and discuss a few best practices.
This guide covers the advantages and disadvantages of the most common types of automated software testing tools.
These 10 Testim.io alternatives for functional software testing can make test writing and maintenance easier.