Automated software testing tools are used to automate interactions with software so tests can be run without a human having to perform each action. This lets one person run multiple tests at once and eliminates most of the human error associated with manual testing.
Although there are many types of software testing, functional testing is the most common and the one most teams start with. That’s why we’ll be focusing on functional testing in this article.
Functional testing deals with making sure the various features of your software work properly and appear correctly for end users.
The traditional approach to automated functional testing is to use a coding framework that automates interactions within a browser. While this approach can be faster than manual software testing, it has several limitations:
In this tutorial, we’ll demonstrate how Rainforest QA—our no-code automated software testing tool—solves these limitations. Then, we’ll review six other software testing tools and discuss how they compare.
Sign up for Rainforest QA—you can run up to five hours of no-code automated software testing for free, every month. It’s only $5/hour after that.
Rainforest QA is a new approach to testing that anyone can do.
Rainforest QA tests mimic real users interacting with an application by using pixel-matching to interact with the visual layer of your software.
The Rainforest visual test editor is divided into two sections: The left-hand side has a drop-down menu where you’ll choose from a list of actions such as ‘click’, ‘fill’, ‘observe’, etc. The right-hand side shows the entire user interface of the virtual machine where the test will be run—including the application you’re testing, the browser you’re using, and the operating system the machine is running.
For most test actions, you’ll use your mouse to click-and-drag a box over an element on the right-hand side to apply an action to it. This captures a screenshot of the element—which is used to find or verify the element in future test runs.
To speed up the test writing process, you can embed one test in another. Embedded tests let you create a common sequence of steps (e.g., your website’s checkout process) and insert that sequence into any test that needs it. Then, whenever you make edits in the sequences of steps, the edits will be applied in every other test that uses that sequence.
You can preview the test steps you’ve written at any point while writing or editing a test. This helps you ensure the test will do what you intend it to do. Once you’re ready to run a test or a set of tests, you can start the test run from within the Rainforest platform with the click of a button. You can also kick off a test run via our API, CLI, CircleCI Orb or GitHub Action.
Most test automation tools work by automating a single browser tab. During the test run, the browser itself is interacting with the application, simulating user actions to complete test cases. While this works for most test cases that happen entirely inside the browser window, it means you can’t create test steps that interact with browser settings, open a new tab, or leave the browser altogether to test something on the desktop.
Rainforest QA, on the other hand, automates operating systems on virtual machines. This allows you to test anything visible on your screen. This is great for software where critical test cases include things like installing a browser extension or downloading something to your desktop. You can also create tests to see how your application performs when different browser settings are turned on or off.
Rainforest also makes desktop application testing possible.
With any automated software testing, some amount of test maintenance is unavoidable. That’s because whenever you make changes (e.g., adding a new feature or fixing a bug) to the application under test, one of those changes might cause the test to become obsolete. This means you could get test failures because the test can’t find the element it’s looking for, even though there’s nothing wrong with the software. Planning for regular test maintenance helps ensure tests only fail when there’s a bug in the software.
While some test maintenance is inevitable, the type of tool you use to run your tests can affect what type of code changes will make your tests obsolete and whether you have to allocate developer resources to maintenance or if it can be handled by less technical team members.
We’ll cover these two factors separately.
When it comes to the types of code changes that require the test to be updated, you’ll want to choose a testing tool whose tests only break when there are changes to what a real user will see on the UI.
However, most test automation tools typically require the tests to be updated whenever there’s a change to the underlying code of the application (the DOM)—even if that change doesn’t affect the UI. This is because most software testing tools search for element locators in the DOM to interact with and identify elements. If the element locator is present, the test will pass. If that same locator gets changed by even one character (to correct a labeling mistake, for example), the test will fail even though a real user wouldn’t notice any difference.
In contrast, Rainforest tests ignore any changes in the DOM that don’t affect the visual layer and only break if there’s a change on the UI. If an element needs to be renamed on the back-end, the Rainforest test won’t be affected unless the element’s appearance was also changed. This can lessen the overall burden of test maintenance.
For times when you’re more concerned with the functionality of the software and less concerned about the exact appearance of each element, you can turn on text-matching for any Rainforest test step. Text-matching uses the content rather than the appearance (e.g., color, shape, etc.) of an element to find a match and complete test steps.
For example, both buttons in the screenshot below contain the words ‘buy now’, however, each one has a different shape and color. With text-matching turned on, either version of the button would return a successful result.
With most automated testing platforms—including low-code tools that allow non-coders to create tests—the most efficient way to fix a broken test is to edit the code directly, which requires programming skills.
But with Rainforest, anyone can look at the plain English test steps and understand what is being tested and easily edit individual steps. In the image below, if the screenshot of the Submit button in Step 5 (“Click left-mouse on Submit”) needed to be updated because the button’s appearance was changed, anyone could jump into the text and update the screenshot in a few seconds.
Even if a developer wants to update a test, they find that editing a Rainforest test is much faster and easier than parsing through hundreds of lines of code to find and fix tiny inconsistencies that cause the tests to fail.
One of the biggest, unexpected time sucks of automated software testing is understanding why a test failed. Sometimes you may need to compare a failed test run to a successful test run, view everything that happened leading up to the failed test step, or consider various aspects of the testing environment in order to figure out what went wrong and how to fix it.
Rainforest QA automatically records every test run whether it passes or fails and stores the recording indefinitely. You can view the recording right next to the test steps and see exactly how each step played out. You’ll also be given HTTP and browser logs in order to see if the failure was due to a change in the testing environment.
The ‘investigate action’ button lets you see additional details about the failure including the closest match found for both the appearance of the original screenshot and the text content (if text-matching was enabled).
Finally, Rainforest QA comes with a Jira integration to help with debugging. If you find a real bug in the software, you can automatically create a ticket for the development team that includes the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.
Rainforest QA is an all-in-one platform that gives you access to everything you need to create, execute, and manage an automated test suite including:
Unlike most tools that charge you extra for essential features like running multiple tests at once or lock you into a contract that limits your testing, we provide usage-based pricing.
You get the first five hours of test runs free of charge every month. Then, you simply pay $5 an hour for any time spent running tests after that. You won’t be charged for the time or tools used to preview, write, edit or otherwise manage your test suite.
Sign up with Rainforest QA and get started with automated software testing to improve your software quality.
Selenium is the original automation framework for testing web applications. Selenium IDE is their record-and-playback feature.
Record-and-playback tools record your screen as you move through a test case and use that recording to generate coded test scripts so that the test can be repeated with automation. Since you aren’t actually writing the code, record-and-playback tools are often called ‘scriptless’ or ‘codeless’ tools.
The main disadvantage of Selenium IDE and other record-and-playback tools is that test cases are limited to the exact actions taken in the original recording. For example, if the last step of your test is to view a ‘Sign Up Was Successful’ screen, the test will wait a designated amount of time for the page to load but it won’t be able to verify that the screen actually did load (unless a developer manually adds code to the test).
Additionally, most teams find that only someone who understands the code will be able to figure out why a test failed. So most teams still rely heavily on developers’ time to write and maintain tests. Plus, these tools are still using code to test code, so you’ll run into all of the disadvantages discussed above that come when interacting with the DOM instead of the visual layer of an application.
Many teams are drawn to Selenium IDE because it’s one of the only open-source record-and-playback tools available. However, you’ll still have to pay for a testing grid to run multiple tests at once, and as we mentioned, you’ll probably still need to factor in the cost of a developer to support your testing team.
Also worth noting: Selenium only offers support through a community chat or written documentation.
Finally, many other tools are built off of the Selenium framework and simply add features to it. For example, Appium is a platform that modified the Selenium framework so that developers could do end-to-end testing on mobile applications as well (for both iOS and Android OS).
For example, most record-and-playback tools won’t let you stop a recording to add code and then restart the recording—but Katalon does.
Other helpful features include built-in keywords, project templates, and test management via customizable tags.
However, you still need to allot developer time to software testing and use another tool if you want to test the visual layer of the UI.
Teams that have more experience with building software than with the software testing process may find all of TestComplete’s features overwhelming to figure out.
TestComplete supports several different types of testing including keyword-driven and data-driven testing, cross-platform testing, API testing, and some forms of performance testing. It also supports tests created in some unit testing frameworks (such as JUnit or TestNG).
Finally, TestComplete also offers an object recognition engine that allows you to test elements inside a PDF or chart for more complete test coverage.
Ranorex Studio comes the closest of all the tools on this list to being able to match Rainforest QA’s ability to test the visual layer of an application.
Ranorex lets you take screenshots of the UI at any test step to verify the UI. However, all other test steps use code and you have to manually add these visual verifications when you want them rather than having every test step automatically test both the functionality and the UI of the software. Additionally, many developers find it’s much easier and quicker to write any test case using Rainforest QA.
Although Ranorex tests are written in code, they offer a mix of recorded actions and drag-and-drop actions from a preset list or repository to help speed up test creation. You can record test runs to help understand results, but you have to choose this setting manually and only the most recent recordings will be saved.
Ranorex offers real device mobile testing in addition to web application testing.
Telerik Test Studio is a software testing solution that puts an emphasis on building your tests with preset actions (although they do offer a record-and-playback feature as well).
While Ranorex Studio allows you to perform GUI testing, Telerik Test Studio essentially removes the UI from the equation. They offer the option to run tests on a headless browser, meaning you won’t see the UI during test execution. Instead, the tests will run the code via a command-line interface or network communication. This can make tests run faster because you don’t have to wait for all the visuals to load—however, it’s not testing what the end user actually sees.
Micro Focus UFT (formerly QTP) is a versatile test automation framework that offers a variety of methods for creating a test to accommodate all levels of coding experience.
A few of these options include manually writing test scripts using a programming language, inserting actions from a library, object identification, and record-and-playback.
They only support Microsoft Windows, however, you can do desktop app testing, web app testing, and mobile app testing. You can also do cross-browser testing on Safari, Firefox, Chrome, IE and Edge.
Rainforest QA is a scalable, all-in-one automated software testing solution that’s suitable for small teams just getting started with software quality assurance or agile teams practicing continuous integration and regularly running a test suite of 500+ automated UI tests.
Get started with Rainforest QA for free. You get 5 hours of free testing every month. It’s only $5/hour after that—and you only pay for what you use.
We provide a detailed review with recommendations of the top 10 software testing tools available today.
In this post, we define what 'smoke testing software' means, provide specific examples of the types of tests to include, and discuss a few best practices.
These 10 Testim.io alternatives for functional software testing can make test writing and maintenance easier.
Learn how Rainforest QA makes desktop application testing easy with its no-code visual testing platform.