The goal of user interface (UI) testing is to verify that:
- All visible elements in a software product work properly and
- All of those elements appear with the correct colors, fonts, shapes, and positioning.
However, most software testing tools either test the appearance of the application or the functionality—few tools do both well.
To test functionality, most automated software testing tools primarily interact with the underlying code of an application, while assuming that the visual layer (what the user sees) is rendering properly.
These test scripts might include steps like “click on the ‘Sign Up’ button” and “verify that the confirmation message is visible.” While that might sound like the test is verifying the appearance of the Sign Up button, the test script is actually just looking for the presence of the code that represents the Sign Up button. As long as the code for the element exists and the visibility is set to visible, the test will pass.
In reality, there are a variety of situations where the code for an element, like a Sign Up button, can be present and have the correct visibility setting, but the appearance of the element will have some error in the UI. It could be rendering off-screen or beneath another element because of an accidental change to the positioning settings in the CSS.
Because of this, many teams take one of the following approaches to testing the appearance of an application:
- They use human testers to manually interact with various user paths looking for visual bugs (which is time-consuming and prone to human error), or
- They use additional plugins to insert screenshot checkpoints that compare the whole page or sections of the page to baseline images (but this method catches every single visual change, not just visual bugs, making it hard to find the signal in the noise).
Either way, you’ll end up needing two tools or methods to complete one job. In contrast, Rainforest QA, our no-code automated testing tool, tests both the functionality and appearance of an application simultaneously.
In this post, we’ll show you how we designed Rainforest to overcome the biggest challenges of UI testing, and we’ll also compare it to 13 other UI testing tools.
- How Rainforest QA tests the appearance and functionality of the UI without coded test scripts
- Code-based UI Testing Tools with Screenshot Plugins for Visual Validation
- Code-based UI Testing Tools with No Visual Validation
Talk to us about setting up a Rainforest account and see why Rainforest is the easiest way to to run automated UI tests without having to code.
How Rainforest QA Tests the Appearance and Functionality of the UI without Coded Test Scripts
Rainforest is a no-code automated UI testing tool with a drag-and-drop editor that lets you create automated tests that mimic how users interact with the final, visual layer of your website or app. You can test the functionality of the application and validate the appearance at the same time.
Here’s a detailed look into how that works:
All-in-one UI Testing Tool
Automated tests created in Rainforest navigate through the app on the visual layer just like a user would—looking for buttons, clicking on links, putting text into forms, etc.
Rainforest validates the visual content by looking for precise screenshots of on-page elements (like a sign-up button), while also validating the overall functionality of the page by completing typical user actions.
Instead of checking whether elements are present in the underlying code, Rainforest tests use pixel matching to complete each test step. To create a test or edit a step in an existing test, you would choose from a drop-down menu of actions (i.e. click, fill, etc.) and take small screenshots of the element you want to apply the action to.
To start the test, you just hit “Run Test” and you’ll automatically have access to our network of virtual machines to run your test on, allowing you to run hundreds of tests simultaneously.
This makes it a true no-code testing solution that lets anyone—including non-developers—create, edit, and run tests.
In addition to letting anyone test both the visual layer and the functionality of your app, Rainforest QA comes equipped with:
- Access to multiple operating systems and browsers, including current and older versions of Chrome, Firefox, Internet Explorer, etc.
- Jira integrations so you can automatically create tickets for tests that need debugging. Tickets include the failed test steps, a screenshot from the failed step, and a link back to the failed test in Rainforest.
- A CLI and integrations with CI tools (including Circle CI and GitHub) to incorporate Rainforest tests into any CI/CD pipeline.
- An API to integrate your testing process into any software stack.
- Built-in test data like randomized email addresses, credit card numbers, names, etc.
- Integrations with Slack, Microsoft Teams, and email, so your team can get real-time notifications of test failures and other updates.
- A worldwide community of human QA testers available on-demand, 24×7 for manual testing. Many teams find this to be useful for tests that can’t be automated or that need subjective feedback.
Results You Can Trust with Specific Screenshots, Smart Suggestions, and Text-matching
Many tools that use screenshots to verify visual elements compare screenshots of the entire page or of large sections of the page. With Rainforest tests, you take the screenshots yourself (rather than letting the tool take the screenshot for you) so they can be as large or as small as you choose. Then, the test only checks the box of pixels that you defined in the test.
That means that there can be minor changes to other sections of the page, and the test will pass as long as all the elements you included in the screenshot were present. If you take a small screenshot that only includes the element (and not any of the surrounding area), the test will pass even if the location of the element is changed.
For example, in the image below, the screenshot of the ‘Start for free’ button is limited to only what is contained in the button (i.e., text and color).
If your screenshot includes some of the background and surrounding elements (as seen in the next image), the test will be sensitive to the position of the element relative to the background and other elements included in the screenshot.
If the test does fail, Rainforest highlights the failed test step and makes suggestions for a new screenshot based on the closest match found.
This example shows a fairly obvious difference (someone updated the copy on this page to say “Talk to Sales” instead of “Talk to a human”), however, there can be cases where the difference isn’t immediately obvious (one letter is serifed while the others are san-serif, for example).
In these cases, Rainforest helps you compare the differences between the two screenshots by layering them one on top of the other.
If you want to update the screenshot, all it takes is a click of your mouse.
You may run into situations where you want to make sure the test will pass even if there are some visual changes. Let’s say you updated the styling (font, color, shapes, etc.) of your app to reflect a rebrand, but you haven’t updated your test suite yet. Since all of the buttons now have different styling, the visual changes will trigger failures across your test suite.
To provide more flexibility in situations like this, Rainforest offers text matching to allow tests to pass even if there have been minor changes to visual elements. You can toggle text matching on or off for any test step.
With text matching turned on, Rainforest will use optical character recognition (OCR) to search for text in the element in your screenshot if it can’t find an exact match of the screenshotted element. If it finds the matching text anywhere on the screen, the test step will pass. Even so, you’ll be notified in the test results page that there was a visual change (see the Element Mismatch warning in the screenshot below), letting you know that you need to update the screenshots.
And if you didn’t intend to change the styling of your buttons, you’ll have a chance to investigate the change before pushing code to production.
Easy Failure Categorization with Video Replays
Being able to see exactly how a test played out is often crucial to finding and fixing visual bugs. Often teams need to see test steps that led up to or follow the failed test step in order to understand why the test failed. Additionally, it can often be helpful to compare a failed test run to a successful test run to see where the change occurred.
This is why Rainforest automatically records a video of every test it runs (regardless of whether the test passes or fails). When a test fails, you can review the video to see the actual point of failure and everything leading up to (and following) it.
With many automated software testing tools, you might have to sort through lines of code in order to identify why a test failed. With Rainforest, anyone can look at test results and video recordings, know exactly what happened in the test, and quickly understand why the test failed.
Talk to us about setting up a Rainforest account.
Code-based Tools with Screenshot Plugins for Visual Validation
We’ll now discuss some UI testing tools that take a traditional code-based approach to verifying the functionality of elements in the UI, but use screenshots to verify the appearance of the app.
A few common test automation tools that use screenshots for visual validation are:
Squish
Squish is a record-and-playback tool that supports JavaScript, Perl, Python, and Ruby with screenshot checkpoints. This means it records your movements in code as you manually click through a user path. When you playback your recording, the tool just executes the code it generated. Then, at any point, you can verify an element using a screenshot or lines of code.
When using the screenshot option, it automatically takes a screenshot of the whole screen. If this doesn’t fit your need, you can either “cut out” areas that the test should ignore or simply retake a smaller screenshot. Additionally, you can set how strict of a match the test requires to pass.
Each part of the test (other than the actual screenshot) is turned into code.
Applitools
Applitools is a plugin that lets you work with nearly any test runner (Applitools supports more than 40 application testing frameworks and languages) to insert visual checkpoints with just one line of code (generated for you by Applitools) into any step in an automation test script.
Applitools takes a screenshot of the entire UI and lets you choose between four levels of sensitivity: exact, strict, content, or layout. If none of these levels work for you, you can also manually ‘carve out’ areas of the UI the test should ignore. Overall, Applitools relies more heavily on programming skills than other visual testing tools.
Ranorex Studio
Ranorex is another record-and-playback tool for desktop, mobile, or web applications. You can use the basic record-and-playback feature to create tests entirely written in code, or you can take screenshots to verify visual elements.
Ranorex offers a variety of settings that let you choose how to compare the screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.
PhantomCSS
PhantomCSS is an open source tool built on top of PhantomJS/CasperJS. It can define the steps of the test script so you can navigate around your app, and then it takes screenshots of each step and uses ResembleJS to compare the images to a baseline.
Unlike other visual tools, PhantomCSS doesn’t offer any features to mitigate sensitivity—you will be notified of any inconsistencies.
Code-based Tools with No Visual Validation
The following code-based tools are considered UI testing tools, but they offer very few features to test the appearance of an application, so many teams supplement their testing efforts with manual testing or a screenshot plugin from the previous list.
The main advantage of these tools is that you can perform cross-browser testing. Since each browser renders the visual layer of a website with slight differences, it’s difficult to create one test that can validate the visual UI across multiple browsers. But if you’re only checking the functionality of the underlying code, it is possible to write one test and run it through multiple browsers. A few common code-based UI testing tools are:
- Selenium IDE: Selenium is one of the oldest testing frameworks available. Selenium IDE is the open source record-and-playback version of Selenium.
- Appium: Appium is an open source tool built off of Selenium specifically for Android and iOS mobile apps.
- Cypress: Cypress is an open source tool made specifically for developers writing tests in JavaScript.
- Katalon Studio: Katalon Studio is a popular record-and-playback tool built on the Selenium WebDriver.
- AutoIt: AutoIt is a free scripting language for automating interactions with the GUI.
- MicroFocus Unified Functional Testing (UFT): UFT is a record-and-playback tool that only supports the VBScript testing language.
- TestComplete: TestComplete lets you write test cases yourself using an object repository or using their record-and-playback feature in JavaScript, Python, VBScript.
- Eclipse: Eclipse is an open source functional testing tool based on Java for testing SWT, Eclipse and GEF-based applications.
- TestProject: TestProject is a free record-and-playback tool for web, mobile, and API testing.
Get Started with True UI Testing with Rainforest QA
With a good UI test tool, you can be confident your app is free of visual bugs while also testing the usability of your application. Rainforest QA’s no-code automated testing platform lets you build regression tests that automatically check the visual components of your app. It’s a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with automated GUI testing or QA-mature teams practicing continuous integration and regularly running 500+ automated software tests.
Talk to us about setting up a Rainforest plan that fits your needs.
