The goal of automating UI testing is to speed up the software release process and help developers catch front-end bugs sooner and fix them faster.
The potential benefits of automated UI testing for web applications are huge:
- No one has to spend time mindlessly clicking through the app.
- Many tests can be run at once, which is faster and more scalable than manual testing.
- Test results come back in just minutes.
- Developers can find out about bugs that are only present in the UI (but don’t show up in unit testing or integration testing) much sooner, allowing them to find the root cause faster because they have less code to dig through and it’s fresher in their minds.
Most software companies do UI testing by asking developers or QA engineers to write test scripts to test the front-end of the application within a web browser. The most popular automation frameworks are Selenium and Cypress. But conventional approaches to UI test automation have a few drawbacks:
- You’re writing code to test code, so the possibility of human error in the test scripts is always present.
- The automation scripts interact with the underlying code of your application, not the visual layer, which means they can miss a lot of bugs that the vast majority of your users would immediately notice.
- It requires programming skills, which means your developers have to spend some of their time writing and maintaining UI tests rather than building new features, or you have to hire dedicated (and often expensive) QA engineers.
In recent years, codeless test automation tools that automatically generate test scripts have become available so that non-technical employees can write tests too. While this helps to a degree, the software still just generates code—which doesn’t actually solve any of the problems mentioned above.
Rainforest QA is an automated UI testing tool that solves all three of these issues with a no-code interface that creates tests to mimic how real human users navigate through your app. (Talk to us about setting up a Rainforest plan that fits your needs.)
In this post, we’ll dig into why testing the underlying code isn’t reliable and explain why testing the visual layer of your app is the only way to have complete confidence in the accuracy of your UI testing results.
Why Testing the Underlying Code Doesn’t Work
When customers interact with your app, they’re not interacting with the underlying HTML code (a.k.a. the DOM) directly—they’re interacting with a visual representation of the underlying code. This visual representation, not the underlying code, is what’s known as the user interface (or graphical user interface).
Any software that tests the underlying code is testing a proxy for the UI, but it’s not actually running UI tests—even if it claims to. Only software that tests the visual layer can truly perform automated UI testing.
Even No-Code Test Automation Tools Require Software Engineers
Whether you’re using a testing framework like Cypress to make test writing easier or a codeless tool that generates Selenium tests for you, you will almost always run into a situation where you need a team member with programming and scripting skills to understand test failures and fix broken tests.
One of the biggest challenges companies face when trying to build and maintain QA is the cost of employing QA engineers. If a UI automation testing tool could allow anyone to create and maintain tests, you wouldn’t need as many software engineers, and the engineers you do have could focus more on developing rather than on testing.
Understanding What Caused a Test to Fail Can Take Hours
When you need to run lots of tests quickly (functional testing or regression testing, for example), using automated testing is much faster than running manual tests, but finding out why a test failed is often a headache. Most automated testing tools will take a screenshot of the UI or a snapshot of the underlying code at the point of a break, but this doesn’t tell you why a break happened. If the break happened because of a bug earlier in the test, it can take hours to recreate test scenarios and sort through code to figure out why the test failed.
Let’s say you’re testing an e-commerce site. You go through the steps of adding an item to a cart, then you go to checkout, and there’s nothing in the cart, which causes the test to fail. A screenshot of the point where the test failed will just show you an empty cart, but you won’t know whether it’s empty because the item never got added or if the app just failed to display the item in the cart.
Tests Are Brittle to Minor Changes in Underlying Code That Don’t Affect the UI
All code-based automation tools look for element locators in the underlying code to verify if an element is visible. For example, let’s say you want to verify that a message saying ‘signup was successful’ appears at the end of a signup process. Code-based tests will search through the underlying code for the element locator of the pop-up message. If the locator is present, the test will pass.
There are two problems with this validation approach:
- The slightest change in the element locator can break your test even if the end results look identical to the user. Maybe you renamed the locator during an update or accidentally missed a letter in the locator when creating the test. Either way, the test would fail because it couldn’t find an exact match even though the element would look and act the same way for the user.
- Just because the locator is found doesn’t necessarily mean the user can see the element. There could be another element on top of the element we’re trying to test for, or a bug in the positioning logic could render the element off screen. In both cases, the test would pass because the locator was found in the underlying code, but a human wouldn’t actually be able to see the element.
While you can write additional code to help avoid these pitfalls, in the end, code-based tests simulate a user experience and make assumptions about UI elements, rather than actually doing what the user will do and testing what the user will see.
Why Testing the Visual Layer Using Rainforest QA Actually Works
Rainforest is the only automated UI testing solution for web applications that interacts with your application just like an end user would.
Rainforest Automation is a True End-to-End Testing Solution
Instead of searching for locators in the underlying code, Rainforest tests use pixel-matching to verify that an element is actually visible. Our virtual machines simulate a browser and interact with it just like a human would—by interacting with on-screen elements without touching the underlying code.
The benefit of testing the visual layer with pixel-matching is obvious when you think about testing for things like confirmation messages. To assert the presence of a confirmation message via a DOM-based automation testing tool, you would need to create separate assertions to test: Is the element visible? Does it say what I want it to say? Is it the correct color? Is it the correct size? And so on.
With Rainforest tests, all of this gets automatically tested with pixel-matching, and the tester only has to take one screenshot.
This testing process is much closer to the actual user workflow, meaning it’s actually testing the application from one end to the other. I would argue that DOM-based testing isn’t actually end-to-end testing because it short-circuits all the steps involved with visually interacting with elements in a web browser.
Anyone Can Write and Maintain Rainforest Tests
With most code-based platforms, setting up an automated UI test suite is costly due to the amount of time it takes and the level of expertise required, which is reflected by a QA engineer’s (high) salary.
Rainforest QA changes that by giving anyone the ability to create tests and contribute to quality assurance—no technical background required.
To create (or edit) any test step in Rainforest QA, simply select a preset action (like “click”, “select”, or “type”) and then click-and-drag to select the element you want to apply the action to.
Once you’ve created each action of the functionality you want to test, you can play back the actions you created to verify that the test will do what you intended.
Then, when you’re ready to test, you launch it with the click of a button in the Rainforest platform, or a developer can kick it off via our API or CLI. Our automation service will run your tests on virtual machines in our cloud. You can also run multiple tests at once on our extensive network of cloud-based virtual machines.
For a more detailed look into how to create a full test, check out this 4-minute video.
Not only can anyone write test cases, but anyone can maintain tests and identify test failures—all without any programming experience.
Rainforest QA has a built-in suggested fix feature that makes it quick and easy to repair broken tests. The suggested fix feature is triggered anytime a screenshot can’t be found, even if the test passed.
The element that can’t be found is highlighted and you can quickly take a new screenshot to fix the bug.
Embedded tests are another feature that can make writing and maintaining tests easier.
Embedded tests allow you to quickly embed one test in another. This means you can create just one test for signing up, for example, and embed that test in every other test that requires a login step.
Here’s an example of how we embed a ‘Rainforest Signup Flow’ test into another test (it includes every action from clicking the ‘Try for Free’ button through verifying that a ‘Successful’ message will appear):
If any steps in the signup flow need to be updated (due to a product change, for example), we can update the test in just one place and it’ll automatically be updated in every single test that has the “Rainforest Signup Flow” test embedded in it.
Identifying Test Failures is Easy
Determining why certain tests failed is often one of the most time-consuming aspects of automated UI testing—particularly when using software that tests the underlying code.
Rainforest QA makes debugging easy by recording a video of every test (even ones that pass). Within each video recording, you can see what action failed and why because the action gets highlighted in red and a brief message appears stating the reason it failed.
This lets anyone read and understand test results. If the test failed because the test itself was broken (e.g. due to a product change), anyone can jump into the visual test editor to fix the test steps. If it was a failure caused by an actual bug, you can add tickets to Jira with one click or download the relevant video recording and HTTP logs for your engineering team.
By giving anyone the ability to create tests, maintain tests, and identify test failures, your organization will have the flexibility to decide who should own QA at any point during the development lifecycle. For example, software developers could be in charge of writing tests as they build a new feature and then hand those tests off to a non-technical QA team for maintenance.
Test Don’t Break with Minor Code Changes That Don’t Affect the UI
A minor change to the underlying code, such as renaming an element locator (which doesn’t actually impact the user experience), won’t break a Rainforest test.
A Rainforest test would fail only if the actual visual appearance of the button changed. For example, if the button was accidentally hidden due to a bug in a recent code push, the test would (and should) fail.
For instances where the appearance of the button has changed but is still functional, Rainforest QA offers optional text matching (which you can turn on or off at any test step).
Text matching examines the text content of an element rather than the appearance. For example, the buttons below both say “Buy Now” even though the colors and shapes are different.
If text matching is enabled, the test will pass with either version of the button. If the text locator is not enabled, it will only pass if your original screenshot matches the version being tested.
This is helpful if you’re only interested in testing the content of the button rather than the design.
Test Actions Outside of the Browser
A Rainforest test is essentially a set of instructions for a virtual machine to perform. This means you can design tests to leave the browser or interact with web browsers, desktop apps, and the UI of the operating system—whether it’s Windows or MacOS.
You can even see how your web application interacts with browsers with ad-blockers. This is just one more way that Rainforest tests are more consistent with the actual user experience than other code-based tests.
For example, here’s a video with a detailed walkthrough of creating a test that saves a file to the desktop from one location, then uploads the file to Google Drive. This is a really useful feature for validating downloads or uploads of files to or from your app.
Integrate with CI/CD Pipeline Tools
Test execution can quickly become a bottleneck unless it’s integrated into your CI/CD pipeline. Rainforest QA offers plugins to integrate with various tools, including CircleCI, so that developers can kick off a suite of Rainforest tests at the same time as their unit tests and integration tests. You can also download our CLI and run Rainforest tests in any development environment.
For custom workflows and more complex test data analysis, our open source API makes it possible to integrate Rainforest into any software stack. Implementing a fully-automated CI/CD pipeline makes it much easier to achieve your team’s test coverage goals without slowing down development in an agile environment.
Get Started with True Automated UI Testing
Rainforest automation enables your team to move faster, without breaking things, all while providing superior automated UI testing capabilities and decreasing the burden of maintenance. It’s easy to get started with automated GUI testing and build as you go. You don’t need to learn a new programming language, you don’t need to buy additional services from other vendors to run or manage your tests, and anyone on your team can create, run, maintain, debug, and interpret tests.
It’s a fast and scalable all-in-one solution that’s appropriate for teams that are just dipping their toes into automated testing, as well as QA-mature teams running 500+ software tests on a regular basis.
Talk to us about setting up a Rainforest plan that fits your needs.