The goal of user interface (UI) testing is to verify that:
However, most software testing tools either test the appearance of the application or the functionality—few tools do both well.
To test functionality, most automated software testing tools primarily interact with the underlying code of an application, while assuming that the visual layer (what the user sees) is rendering properly.
These test scripts might include steps like “click on the ‘Sign Up’ button” and “verify that the confirmation message is visible.” While that might sound like the test is verifying the appearance of the Sign Up button, the test script is actually just looking for the presence of the code that represents the Sign Up button. As long as the code for the element exists and the visibility is set to visible, the test will pass.
In reality, there are a variety of situations where the code for an element, like a Sign Up button, can be present and have the correct visibility setting, but the appearance of the element will have some error in the UI. It could be rendering off-screen or beneath another element because of an accidental change to the positioning settings in the CSS.
Because of this, many teams take one of the following approaches to testing the appearance of an application:
Either way, you’ll end up needing two tools or methods to complete one job. In contrast, Rainforest QA, our no-code automated testing tool, tests both the functionality and appearance of an application simultaneously.
In this post, we’ll show you how we designed Rainforest to overcome the biggest challenges of UI testing, and we’ll also compare it to 13 other UI testing tools.
Sign up for Rainforest QA—you can run up to five hours of no-code visual regression tests for free, every month. It’s only $5/hour after that.
Rainforest is a no-code automated UI testing tool with a drag-and-drop editor that lets you create automated tests that mimic how users interact with the final, visual layer of your website or app. You can test the functionality of the application and validate the appearance at the same time.
Here’s a detailed look into how that works:
Automated tests created in Rainforest navigate through the app on the visual layer just like a user would—looking for buttons, clicking on links, putting text into forms, etc.
Rainforest validates the visual content by looking for precise screenshots of on-page elements (like a sign-up button), while also validating the overall functionality of the page by completing typical user actions.
Instead of checking whether elements are present in the underlying code, Rainforest tests use pixel matching to complete each test step. To create a test or edit a step in an existing test, you would choose from a drop-down menu of actions (i.e. click, fill, etc.) and take small screenshots of the element you want to apply the action to.
To start the test, you just hit “Run Test” and you’ll automatically have access to our network of virtual machines to run your test on, allowing you to run hundreds of tests simultaneously.
This makes it a true no-code testing solution that lets anyone—including non-developers—create, edit, and run tests.
In addition to letting anyone test both the visual layer and the functionality of your app, Rainforest QA comes equipped with:
Many tools that use screenshots to verify visual elements compare screenshots of the entire page or of large sections of the page. With Rainforest tests, you take the screenshots yourself (rather than letting the tool take the screenshot for you) so they can be as large or as small as you choose. Then, the test only checks the box of pixels that you defined in the test.
That means that there can be minor changes to other sections of the page, and the test will pass as long as all the elements you included in the screenshot were present. If you take a small screenshot that only includes the element (and not any of the surrounding area), the test will pass even if the location of the element is changed.
For example, in the image below, the screenshot of the ‘Start for free’ button is limited to only what is contained in the button (i.e., text and color).
If your screenshot includes some of the background and surrounding elements (as seen in the next image), the test will be sensitive to the position of the element relative to the background and other elements included in the screenshot.
If the test does fail, Rainforest highlights the failed test step and makes suggestions for a new screenshot based on the closest match found.
This example shows a fairly obvious difference (someone updated the copy on this page to say “Talk to Sales” instead of “Talk to a human”), however, there can be cases where the difference isn’t immediately obvious (one letter is serifed while the others are san-serif, for example).
In these cases, Rainforest helps you compare the differences between the two screenshots by layering them one on top of the other.
If you want to update the screenshot, all it takes is a click of your mouse.
You may run into situations where you want to make sure the test will pass even if there are some visual changes. Let’s say you updated the styling (font, color, shapes, etc.) of your app to reflect a rebrand, but you haven’t updated your test suite yet. Since all of the buttons now have different styling, the visual changes will trigger failures across your test suite.
To provide more flexibility in situations like this, Rainforest offers text matching to allow tests to pass even if there have been minor changes to visual elements. You can toggle text matching on or off for any test step.
With text matching turned on, Rainforest will use optical character recognition (OCR) to search for text in the element in your screenshot if it can’t find an exact match of the screenshotted element. If it finds the matching text anywhere on the screen, the test step will pass. Even so, you’ll be notified in the test results page that there was a visual change (see the Element Mismatch warning in the screenshot below), letting you know that you need to update the screenshots.
And if you didn’t intend to change the styling of your buttons, you’ll have a chance to investigate the change before pushing code to production.
Being able to see exactly how a test played out is often crucial to finding and fixing visual bugs. Often teams need to see test steps that led up to or follow the failed test step in order to understand why the test failed. Additionally, it can often be helpful to compare a failed test run to a successful test run to see where the change occurred.
This is why Rainforest automatically records a video of every test it runs (regardless of whether the test passes or fails). When a test fails, you can review the video to see the actual point of failure and everything leading up to (and following) it.
With many automated software testing tools, you might have to sort through lines of code in order to identify why a test failed. With Rainforest, anyone can look at test results and video recordings, know exactly what happened in the test, and quickly understand why the test failed.
Our free-forever Professional plan makes software test automation accessible to anyone. This plan has everything you need for test automation—parallel testing on virtual machines and unlimited team members—without any hidden costs or signed contracts.
You get up to 5 hours of free testing every month and it's only $5/hr after that. This means you can easily scale your testing as your company grows.
We’ll now discuss some UI testing tools that take a traditional code-based approach to verifying the functionality of elements in the UI, but use screenshots to verify the appearance of the app.
A few common test automation tools that use screenshots for visual validation are:
When using the screenshot option, it automatically takes a screenshot of the whole screen. If this doesn’t fit your need, you can either “cut out” areas that the test should ignore or simply retake a smaller screenshot. Additionally, you can set how strict of a match the test requires to pass.
Each part of the test (other than the actual screenshot) is turned into code.
Applitools is a plugin that lets you work with nearly any test runner (Applitools supports more than 40 application testing frameworks and languages) to insert visual checkpoints with just one line of code (generated for you by Applitools) into any step in an automation test script.
Applitools takes a screenshot of the entire UI and lets you choose between four levels of sensitivity: exact, strict, content, or layout. If none of these levels work for you, you can also manually ‘carve out’ areas of the UI the test should ignore. Overall, Applitools relies more heavily on programming skills than other visual testing tools.
Ranorex is another record-and-playback tool for desktop, mobile, or web applications. You can use the basic record-and-playback feature to create tests entirely written in code, or you can take screenshots to verify visual elements.
Ranorex offers a variety of settings that let you choose how to compare the screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.
PhantomCSS is an open source tool built on top of PhantomJS/CasperJS. It can define the steps of the test script so you can navigate around your app, and then it takes screenshots of each step and uses ResembleJS to compare the images to a baseline.
Unlike other visual tools, PhantomCSS doesn’t offer any features to mitigate sensitivity—you will be notified of any inconsistencies.
The following code-based tools are considered UI testing tools, but they offer very few features to test the appearance of an application, so many teams supplement their testing efforts with manual testing or a screenshot plugin from the previous list.
The main advantage of these tools is that you can perform cross-browser testing. Since each browser renders the visual layer of a website with slight differences, it’s difficult to create one test that can validate the visual UI across multiple browsers. But if you’re only checking the functionality of the underlying code, it is possible to write one test and run it through multiple browsers. A few common code-based UI testing tools are:
With a good UI test tool, you can be confident your app is free of visual bugs while also testing the usability of your application. Rainforest QA’s no-code automated testing platform lets you build regression tests that automatically check the visual components of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated GUI testing or QA-mature teams practicing continuous integration and regularly running 500+ automated software tests.
Get started with Rainforest QA for free. You can run up to five hours of codeless automated UI tests for free, every month. It’s only $5/hour after that.
Rainforest’s CTO and co-founder, Russell Smith, and CIO, Derek Choy, recently discussed test automation limitations technical leaders need to know about before biting the bullet. In this post we cover the 3 biggest limitations to consider.
We provide practical guidance on how to start automation testing from scratch and how to choose a test automation tool.
Today, we’ll discuss the pros and cons of using testing automation as part of your QA strategy, and how to gain the most from its benefits while minimizing its downsides.
The right codeless test automation tools make it easy for anyone to speed up their QA process without trading quality.