Automated visual regression testing tools check for style issues and problems with the visual layer of an app or website.
The benefit of these tools is that they can catch issues on the visual layer (the user interface of the website or app, which customers see) that may get missed by test scripts that only interact with the underlying code (the DOM).
Common visual regressions include things like overlapping modules, hidden buttons, elements that render off-screen, brand styles reverting to old versions, and a host of minor or major CSS issues that can get missed by both human QA testers and automation scripts.
For example, here’s a screenshot of Google’s homepage with an error. The “I’m Feeling Lucky” button is missing. Now, if the button had truly been removed from the page, it wouldn’t be present in the underlying code of the page, and most automated regression testing tools would catch the bug.
However, if you scroll to the right within the browser, you’ll find that the button is, in fact, present, but it’s rendering off-screen because someone (in this hypothetical situation) accidentally used the wrong setting in some CSS—a not-uncommon occurrence.
Most regression testing tools that only interact with the underlying code of the page without a visual validation step would let this bug escape to production.
While a human tester would immediately notice the visual bug in the example above, take a look at the two screenshots below. Does the difference between the two immediately jump out at you?
When you’re comparing the images side by side, you might notice that the space between the buttons is too big (by 10 pixels) in the second image. But for most human users (or testers), this kind of subtle change would be easy to miss. If you don’t promptly correct minor visual regressions like this, similar errors in the future can compound until you’re looking at a screen with an inconsistent layout.
Human testers also tend to overlook minor brand inconsistencies. If an error in development caused the app to revert to an old stylesheet, the buttons could appear with outdated brand colors or styles, eroding your brand (and in some cases, even damaging your company’s trademark rights).
The one way to consistently catch visual bugs like these without slowing down your testing process is to use a visual regression testing tool that seamlessly fits into your automated testing workflow.
Over the last few years, there has been an explosion in development of visual testing tools, and the differences between them aren’t always obvious.
In this post, we’re going to cover what to look for in a visual regression testing tool and share what we believe are the five best tools available today.
- Visual Regression Testing Tools: What to Look for
- Best Overall Visual Regression Tool for Web Apps (and No-Code QA Teams): Rainforest QA
- Best Visual Regression Testing Tool for Selenium Power Users: Applitools
- Best Mobile Visual Regression Testing Tool: Kobiton
- Best Open Source Visual Validation Tool: PhantomCSS
- Best Visual Diff Tool for Marketing Sites, Blogs, and News Sites: Percy
Talk to us about how you can quickly and easily create automated visual regression tests with no code with Rainforest.
Visual Regression Testing Tools: What to Look for
We’re going to assume that you’re already doing some software regression testing, whether you’re clicking through your app manually or using an automated testing tool.
If you’re going to add a visual regression testing tool to your mix, there are a few things to look for to decide if the tool will meet your needs.
Do you need programming skills to use it?
Automated testing tools can be divided into two categories:
- No-code tools that let anyone create tests with either a drag-and-drop interface or a record-and-playback plugin, and
- Testing frameworks to help developers write coded test scripts.
Visual regression testing tools also fall into those two categories. If you don’t have programmers on your QA team, you’ll want to look for a no-code tool.
How does it filter out all of the false positives that come from tiny rendering differences?
A visual regression testing tool needs to reliably trigger a failure whenever it finds visual regressions on the front end that affect the user experience, while ignoring visual changes that aren’t bugs.
It sounds simple, but it’s actually quite hard to pull off. Most tools, especially the open-source options, simply compare screenshots of the entire UI and will flag every single change, making them too sensitive for most users. Debugging all of these “false positive” will frustrate any team aiming for agile regression testing that allows small, frequent releases.
Is it an all-in-one testing solution, or is it an add-on to another automation tool?
Some visual testing tools are standalone testing solutions, meaning you can create, run, and edit tests with them, but they only do visual validation, so they’re meant to be used in addition to some other solution for regression testing. Other tools are just plug-ins designed to add visual validation to your existing automation framework. They usually do this by triggering a screenshot step in the middle of an existing automated test script.
In contrast, our tool, Rainforest, is an all-in-one solution to automate regression testing, both visual and otherwise. (But more on that later.)
Is it open-source or commercial? If commercial, is the pricing flexible and scalable?
There are some free open-source tools that you’ll never have to pay for, but open-source tools don’t come with customer support teams, typically require programming skills to use, and are usually more limited in what they can do.
On the commercial side, many tools structure their pricing into tiers that can force you to upgrade to a much more expensive tier when you max out the starter plan. And some tools lock you into long-term contracts. An ideal tool will let you pay for only as much testing as you need, and scale up or down freely as your needs change.
Now we’ll share our five favorite visual testing tools, starting with our tool, Rainforest QA.
Best Overall Visual Regression Tool for Web Apps (and No-Code QA Teams): Rainforest QA
Rainforest QA is a no-code automated UI testing tool with a drag-and-drop editor that lets you create automated tests that mimic how users interact with the final, visual layer of your website or app.
Here’s a deep dive into the features that Rainforest customers love.
No-Code Automation That Anyone Can Use with Built-in Visual Validation
Rainforest’s automation navigates through the app on the visual layer just like a user would—looking for buttons, clicking on links, putting text into forms, etc.
It validates the visual content by looking for precise screenshots of on-page elements (like a sign-up button), while also validating the overall functionality of the page by completing typical user actions.
Because it’s no-code, anyone, including non-developers, can set up, edit, and run tests in Rainforest.
To create a test, you choose from a list of available actions in a drop-down menu in the test editor (click, fill, press key, etc.). After you choose the first action, you click-and-drag with the mouse to create a small screenshot to indicate where on the page the action will happen (on a button, on a hyperlink, in a form, etc.).
You can add as many actions as you need to complete the test. You can even test things like sending and checking emails, activating new user accounts, and downloading files.
Precise Screenshots & Text Matching to Filter Out Noise
One of the action options you can choose in Rainforest is “Observe.” The “Observe” action will verify that an element is both visible and looks like the screenshot you took when you created the test. This is important for catching visual bugs.
If the color has changed on a button, or another element is overlapping the element you want to check, the test will fail. But notably, the Observe action doesn’t observe the entire UI—it only checks the box of pixels that you defined in the test. That means that there can be minor changes to other sections of the page, and the test will pass as long as all the elements you included in the test were present.
But sometimes, Rainforest’s exact pixel-matching can still be too sensitive for your needs. Let’s say you updated your stylesheets to reflect a rebrand, but you forgot to update your test suite. Since all of the buttons now have different styling, the visual changes would trigger failures across your test suite.
To provide more flexibility in situations like this, Rainforest offers text matching to allow tests to pass even if there have been minor changes to visual elements. You can toggle text matching on or off for any test step.
With text matching turned on, Rainforest will search for the text of the button or element in your screenshot if it can’t find an exact pixel match of the screenshotted element. If it finds the matching text anywhere on the screen, the test will still be able to proceed. Even so, you’ll be notified in the test results page that there was a visual change (see the Element Mismatch warning in the screenshot below), letting you know that you need to update the screenshots.
And if you didn’t intend to change the styling of your buttons, you’ll have a chance to investigate the change before pushing code to production.
Built-in Tools for Running Tests on Different Browsers and Reporting Bugs
Rainforest QA is an all-in-one test automation solution, which means it has built-in tools to allow users to create, run, and manage as many automated tests as they need.
With Rainforest, you don’t need to pay for additional services like a testing grid, test case management tools, or team collaboration tools just to manage your automated testing.
Rainforest QA includes:
- A built-in “test grid” of virtual machines featuring more than 40 web browsers (including current and old versions of Chrome, Firefox, Internet Explorer, etc.) and platforms across desktop and mobile.
- Test case organization options allow you to sort tests into different run groups, feature sets, and priority levels.
- Integrations with Slack, Microsoft Teams, and email, so your team can get real-time notifications of test failures and other updates.
- A JIRA integration so you can automatically create tickets for your developers for test failures. Each ticket includes steps from the failed test, a screenshot from the failed test step, HTTP logs, and a link back to more information in Rainforest, including a video reproduction of the failed test.
- Developer tooling including an API, a CLI, and integrations for your CI workflows in CircleCI and GitHub.
- A worldwide community of human QA testers available on-demand, 24×7 to run tests that can’t be automated, or that need subjective feedback.
Video Replays of Every Test for Easy Failure Categorization
Rainforest automatically records a video of every test it runs (whether the test passes or fails). When a test fails, you can review the video to see the actual point of failure and everything leading up to (and following) it.
With most automated software testing tools, you’d have to sort through lines of code in order to identify why a test failed. With Rainforest, anyone can look at test results, know exactly what happened in the test, and quickly understand why the test failed.
Rainforest charges users based on how many minutes of testing they do each month.
See for yourself why Rainforest QA is the best visual regression testing tool. Talk to us about setting up a Rainforest plan that fits your needs.
Other Visual Regression Testing Tools
While we think Rainforest is the best overall visual testing tool, you might consider some of the following tools if you’re absolutely committed to using Selenium (but you might want to read why many organizations are looking for Selenium alternatives); if you’re exclusively testing mobile apps; if you want to use an open-source tool; or if you only want to validate the visual rendering of a single web page.
Best Visual Regression Testing Tool for Selenium and Cypress Power Users: Applitools
How It Works: Working within your favorite test runner (Applitools supports more than 40 testing frameworks and languages, so it’s compatible with almost any development environment), you can insert visual checkpoints with just one line of code (generated for you by Applitools) into any step in an automation test script.
Like many other visual regression tools, Applitools takes a screenshot of the entire UI when you write the test, and sets it as the baseline. The next time you run the test, it will take a new screenshot and compare it to the baseline, using an AI-powered image comparison algorithm to filter out the noise.
What We Like: The AI aspect of Applitools helps it overcome the obvious false positives/noise problems of tools that just take a screenshot of the entire UI and alert you to ANY changes. It has four layers of sensitivity, called match levels:
- Exact is a pixel-to-pixel comparison, which they don’t recommend because it’s likely to cause false positive results.
- Strict compares everything including content (text), fonts, layout, colors, and position of each of the elements. This is the level they recommend for regression tests within one browser or operating system because the algorithm knows to ignore rendering changes that are not visible to the human eye.
- Content is similar to strict, but it ignores colors.
- Layout compares the layouts (i.e. structure) of the baseline and actual images. It makes sure that elements are properly aligned and positioned relative to each other, but it ignores the actual content and color of those elements. It’s great for checking the UI of apps with user-generated content or other types of dynamic content that would change between each test run.
If those four match levels don’t work for you, you can also carve out sections of the page that you know change too often for the AI to handle (dynamic content, moving content, features in beta, etc.). However, we think it’s more efficient to specify what you DO want to evaluate (as you would with Rainforest’s visual editor), rather than having to carve out the stuff you DON’T. If you carve out too much of the page, your test effectively has a bunch of holes where bugs can slip through.
What We Don’t Like: Unfortunately, Applitools is extremely limited in functionality if you don’t have programming skills. It’s not an accessible tool for non-technical QA team members. You can use Applitools with the Selenium IDE record-and-replay plugin to add visual checkpoints into tests, but these tests are extremely limited (you can’t do database testing, you don’t get detailed test reports, and it can’t handle conditional statements).
Pricing: The Free Forever plan offers one user and 100 visual checkpoints per month. If you want to have multiple users and CI/CD integration, you’ll need to sign a 1-year contract, and they don’t list pricing on their website, so you’ll have to inquire with the sales team.
Best Mobile Visual Regression Testing Tool: Kobiton
How It Works: Kobiton focuses on mobile testing and real-device testing, but when it comes to visual testing, it’s really trying to help you maintain a consistent visual output across all devices and form factors. It offers scriptless manual testing: you can record a manual test and it converts it into an Appium script (Selenium scripts are also available). You’ll get automatic notifications of visual anomalies.
What We Like: Kobiton is one of the only options that is built for mobile first, and mobile is hard to do well. Also, they go beyond just pointing out differences to making suggestions for improvements based on their analysis of the Top 50 apps in the world. Additionally, users say the screenshot function makes gathering screenshots for different screen sizes quick and easy, which helps them get apps up on app stores faster.
What We Don’t Like: The only option Kobiton has for filtering noise and handling dynamic content is a setting to tell the AI to “navigate to the best matching element,” but there’s no way to know how consistently it will be able to identify the best match.
Pricing: Kobiton’s pricing is based on minutes of test run-time, like Rainforest. However, the cheapest plan starts at $500 a year for 500 minutes/month.
Best Open-Source Visual Validation Tool: PhantomCSS
How It Works: PhantomCSS is built on top of PhantomJS/CasperJS. It can define the steps of the test script so you can navigate around your app, and then it takes screenshots of each step and uses ResembleJS to compare the images to a baseline. You’ll get notified of every difference. It has no tools for minimizing false positives.
What We Like: PhantomCSS has one of the most active and healthy communities on GitHub for any open-source visual regression testing tool. That’s very important for an open-source because that community is your only source for support when you run into trouble using it.
What We Don’t Like: Like all open-source visual regression testing tools, there’s too much noise for practical usage because you’ll get notified of every difference between your baseline image and the screenshot of each test. And I mean literally every difference. The only way to handle content that you know will have minor changes frequently is to use Javascript to replace content within the elements you’re testing with a set of representative dummy content—so ultimately you’re not testing what the user would see.
Pricing: Free
Best Visual Diff Tool for Marketing Sites, Blogs, and News Sites: Percy
How It works: Percy is a tool owned by Browserstack. If you’re already using Browserstack for cross-browser testing or any of its other automation tools for functional testing, it integrates smoothly with your existing unit and integration tests. When a visual test is kicked off, Percy renders UI snapshots across different browsers and responsive widths. Through pixel-by-pixel baseline comparisons, Percy detects and highlights relevant visual changes that have appeared in each page and state of your UI.
What We Like: Percy is a really fast way to test how a single web page looks on all of the browsers and device combinations. And it can test how it looks as the width of the screen changes, to find any widths where the responsive design is broken (overlapping elements, huge white spaces, etc.). Users mention that the snapshot stabilization feature can minimize false positives from the tiny differences between the ways different browsers render an app.
What We Don’t Like: Pixel-by-pixel image comparison is too sensitive for functional testing of web applications where you’re navigating through multiple screens, inputting information, evaluating results, etc. The noise from all of the false positives will quickly make it unusable for testing more than one page at a time.
Pricing: Percy has a free tier which provides 5,000 screenshots/month, unlimited team members, and unlimited projects. The Startup plan starts at $149/month and includes unlimited users, cross-browser support, automatic browser upgrades, responsive visual testing, source code integrations, and 1 year build history.
Improve Your Test Coverage with Visual Regression Testing from Rainforest QA
With a visual regression testing tool, you can be confident your app is free of visual bugs that disrupt the user experience or erode your brand image.
Rainforest QA’s no-code automated testing platform lets you build regression tests that automatically check the visual components of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams regularly running 500+ automated software tests.
Talk to us about setting up a Rainforest plan that fits your needs.
