Functional testing is one of the final steps before a software application goes live, and it’s designed to ensure critical user paths are operating correctly.
It’s a kind of black box testing that doesn’t evaluate the underlying code of the application, but instead tests whether a specific input leads to an expected output in the user interface (UI).
However, most functional testing tools produce test scripts that interact with the underlying code of a web browser without directly testing the visual layer of an application. This means they have three major limitations:
- They require programming skills to use them effectively and often have a steep learning curve to master the nuances of the tool.
- They’re limited to testing actions inside of a single browser tab.
- They don’t let you verify how the application appears to an end user.
Rainforest QA is the only tool that allows you to test the functionality of your application and the appearance of the UI simultaneously without any programming skills. Rainforest tests can also operate across multiple browser tabs and even execute tasks outside the browser.
In this post, we’ll compare 17 functional testing tools, starting with Rainforest QA. If you’d like to learn more about the different types of functional testing and where it fits into the release process, check out our guide to automated functional testing.
Table of Contents
- Rainforest QA
- Selenium
- TestComplete by Smartbear
- SoapUI
- MicroFocus UFT
- IBM Rational Functional Tester
- Sauce Labs
- Ranorex Studio
- Tricentis Tosca
- Watir
- Sahi Pro
- Katalon Studio
- Appium
- Telerik Test Studio
- Rapise
- Testim
- Mabl
Talk to us about setting up a Rainforest QA plan that fits your needs.
Rainforest QA
Rainforest QA lets anyone create automated tests without writing code. Rather than interacting with the underlying code of the application, Rainforest tests mimic how users interact with the final, visual layer of the application.
We designed Rainforest to help any team increase test coverage without adding headcount. Here are just a few of the main capabilities that make Rainforest QA a great fit for functional testing:
Test Outputs as the User Will See Them
Most functional testing tools use element locators in the underlying code of the application to identify elements, such as buttons or form fields, and then perform actions on them. The main goal of these tests is to make sure the elements work properly.
If you also want to verify the appearance of elements on the screen, you can add additional test steps to check that an element has the right HTML and CSS settings related to its color, size, style, etc. As long as those settings are correct, the test will pass. However, this isn’t a reliable way to catch bugs in the UI.
Sometimes the locator is present—with the correct settings—but the element can still render incorrectly on the visual layer. For example, a bug in the CSS that defines the positioning of an element can make it render it off-screen. Or, a pop-up modal might cover it. In these situations, most functional testing tools would return a passing test but real users would quickly notice the bug.
Rainforest takes a completely different approach to make sure the tests are actually evaluating the UI as the user will experience it with every step of the test.
Rather than spending tons of time writing and maintaining tests that don’t fully test the real user experience, Rainforest QA lets anyone quickly create tests that automatically test the UI every step of the way. Our no-code visual editor lets you create tests that use pixel-matching to locate, interact with, and verify elements. This means the tests interact with the application just like a human would—directly interacting with the UI without touching the underlying code.
Anyone Can Create and Maintain Tests
With Rainforest, anyone can learn how to write or read a test within minutes regardless of their programming ability. Even if you do have programming skills, Rainforest makes it much easier and faster to write and maintain tests.
To write a test in Rainforest QA, simply choose from a dropdown menu of preset actions such as ‘click’, ‘fill’, ‘scroll’, etc. Then, click-and-drag your cursor around the element you want to apply the action to. This takes a screenshot which will be used to find or verify elements during a test run.
Looking at the left-hand column, anyone can read the steps and know exactly what’s going to happen. You can also play back each step or the test as a whole to ensure it will do what you intended it to do.
You can also embed one test into another test to speed up the test creation (and test maintenance) process. This allows you to write a common sequence of test steps just once and insert those steps into any test that needs it.
Then, any test step can be edited in the same way it was created. If you’re using embedded tests, you’ll only have to update the test in one place and it’ll automatically be updated in every other test that uses that set of steps.
Finally, you’ll automatically have access to our network of virtual machines to run your tests, allowing you to run hundreds of tests simultaneously. You can kick off tests from within the Rainforest platform or as part of your CI/CD pipeline via our API, CLI, CircleCI Orb, or GitHub Action.
Test Actions Outside of a Single Tab or Browser
Because most functional testing tools only automate individual browser tabs (rather than an entire virtual machine), test cases will be limited to testing actions within one browser tab. But many software teams have critical test cases that require leaving the tab or the browser to fully test the functionality of their application. A few common scenarios include:
- Downloading a desktop app: If your product is a desktop application, for example, it’s important that you have a test that covers whether or not customers will be able to successfully download, install, and open the app. This means you’ll need a test that can leave the browser and interact with the operating system of the device.
- Installing a browser extension: Most functional testing tools are also unable to interact with any browser settings, which means they aren’t able to verify that you could add a browser extension. If your entire product relies on users installing a browser extension, this is a critical test case.
- Testing interactions between two users: If your product includes any kind of messaging app, there are many scenarios where it’s helpful to be able to open two browsers and actually test the sending and receiving of messages between those two users.
Rainforest tests automate an entire virtual machine, so they can open or close tabs and browsers, interact with browser toolbars and dropdown menus, and even interact with desktop apps. For example, here’s a video walkthrough of creating a test that saves a file to the desktop from one location, then uploads the file to Google Drive.
Easily Identify the Reason for Every Test Failure
Figuring out why a test failed is often one of the most time-consuming parts of functional testing.
Rainforest QA makes it really easy to debug test failures because every test gets recorded (whether it passes or fails) and can be played back at any time. This lets you see every step of the test exactly how it would’ve appeared to a real user.
The test step will be highlighted in red along with a brief explanation of why it failed, as shown in the bottom left of the screenshot below. You can watch the replay by clicking the “Play” button in the video player.
You’ll also have access to HTTP and browser logs. If the test failed because of an actual bug that you want to fix, you can create a ticket in Jira to send to the development team that includes:
- The failed test steps
- A screenshot of the failed test step
- HTTP logs
- A link to the full test results and video recording in Rainforest
All-in-One Testing Solution
To scale up automated testing as your software grows, you’ll need a way to write and manage tests, run multiple tests simultaneously, and manage communication within the QA team.
Rainforest QA is an all-in-one solution equipped with:
- Easy test suite management including folders, tags, and filters.
- A network of virtual machines to run multiple tests at once.
- Access to multiple operating systems and browsers for testing, including current and older versions of Chrome, Firefox, Internet Explorer, etc.
- Built-in test data like randomized email addresses, credit card numbers, names, etc.
- Integrations with Slack, Microsoft Teams, and email, so your team can get real-time notifications of test failures and other updates.
Talk to us about setting up a Rainforest plan that fits your needs.
Selenium
Selenium was originally just a testing framework for automating interactions with browsers. Now, when developers refer to Selenium, it could mean the original testing framework or the WebDriver (which supports Java, Python, Ruby, C#, JavaScript, Perl, and PHP programming languages).
There’s also a record-and-playback feature called Selenium IDE which lets you create very basic functional tests without coding. All of these Selenium variations support cross-browser testing.
Although it’s still a widely-used testing tool—because it’s been around the longest—Selenium has several disadvantages that can make it difficult to scale up testing.
TestComplete by Smartbear
TestComplete by SmartBear is built on JavaScript, Python, and VBScript. It also supports tests created in some unit testing frameworks (such as JUnit or TestNG). They support testing on most desktop, web, and mobile platforms (including both iOS and Android OS). TestComplete lets you search for and insert code for specific test steps based on preset keywords.
TestComplete also offers an object recognition engine that allows you to test elements inside a PDF or chart.
SoapUI
SoapUI is also supported by SmartBear but it focuses on Web services testing. You can opt for the open-source version which allows you to create tests from a single device, or you can upgrade to a paid plan which provides additional integrations and back-end services.
Soap UI supports data-driven testing, performance testing, and load testing.
MicroFocus UFT
Micro Focus UFT offers different methods for creating a test. The most common are to manually write the code yourself or to use the record-and-playback option. Their support for functional testing and regression testing is somewhat limited because they can only test desktop and web applications (no mobile application testing abilities) and they only support the Microsoft OS.
However, Micro Focus does work with multiple browsers including Safari, Firefox, Chrome, IE, and Edge.
IBM Rational Functional Tester
IBM Rational Test is a record-and-playback platform that runs on Windows and Linux operating systems. With their platform, you can manually insert triggers at any point to take snapshots of the underlying code of your application. These snapshots can be used to verify the state of your application before or after an action.
If you’re already using one of IBM’s many other business services, it may be useful to keep your services in one place. However, reviews suggest that the IBM user interface feels like it was designed for technically advanced QA teams.
Sauce Labs
A lot of testing teams use the Sauce Labs testing grid to run multiple tests at the same time, however, Sauce Labs also provides a low-code way to generate Selenium, Appium, Espresso or Cypress test scripts. To use Sauce Labs’ test automation solution, you upload spreadsheets of detailed actions and datasets written in plain text and the testing platform generates the test scripts for you.
Sauce Labs supports testing on real devices for mobile, web, and API testing.
Ranorex Studio
Ranorex is a record-and-playback tool for desktop, mobile, or web applications. You can create tests using just code, or you can use screenshots to verify visual elements.
Ranorex offers a variety of settings that let you choose how to compare the screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.
Tricentis Tosca
Tricentis Tosca is an enterprise-focused end-to-end testing tool, built for teams practicing agile methodologies. They support continuous testing with integrations for various DevOps tools such as Jira, Jenkins, and Github. It’s especially helpful for organizing test data.
Reviewers say the test management features help their testers increase their productivity. Finally, Tricentis Tosca offers real-time risk assessment of business processes so you can decide what testing is the most important at any given moment.
Watir
Watir (Web Application Testing In Ruby) is an open source platform built with a more user-friendly interface than Selenium.
While test scripts in Selenium have to be written very specifically for complex test cases, Watir provides more flexibility. For example, if your network is slow, you would need to specify wait times for every action in Selenium to accommodate the load time. Watir automatically waits for the page to load. Watir only supports web testing—it doesn’t support mobile apps, API testing, or cross-platform testing.
Sahi Pro
Sahi Pro is a test automation tool designed to make coding more accessible to QA beginners with less coding experience. They offer a record-and-playback feature along with object identification. Additionally, the platform will compare failed test runs to successful test runs to help identify why the test failed.
Sahi Pro is an all-in-one solution with yearly contracts. They have a segmented pricing structure that prices web, mobile, and desktop testing separately.
Katalon Studio
Katalon Studio was designed to support collaboration between developers and non-developers and is built on the Selenium WebDriver. It also works with Groovy and Java. Unlike some record-and-playback tools, Katalon lets developers stop and restart a recording at any point during test creation or test execution to add or edit test steps or assertions. This makes it much easier to create complex test cases and fix broken tests.
Katalon also provides helpful tools such as built-in keywords and project templates to help create test cases faster. You can learn more about Katalon in this article that compares Katalon vs. Selenium vs. Cypress vs. Rainforest QA.
Appium
Appium modifies Selenium so you can test browsers on mobile devices and supports most mobile testing platforms including iOS, Android, and Windows apps.
Appium supports native, web, and hybrid applications and scripting languages using the Selenium WebDriver. Plus, it’s very customizable and can be integrated into any software development tool.
Telerik Test Studio
Telerik Test Studio uses a mix of selecting preset actions and recording actions. Telerik puts more of an emphasis on building your tests with preset actions, so the tests may take longer to create than other record-and-playback options.
They offer the option to run tests on a headless browser meaning you won’t see the UI as the test runs. Instead, the tests will run the code via a command-line interface or network communication. This can make tests run faster because you don’t have to wait for all the visuals to load, but it’s not testing what the end user actually sees.
Rapise
Rapise by Inflectra is a browser automation tool for mobile, desktop, web testing, and API testing. The tool records your screen and generates a spreadsheet of test steps which is where you can edit the test. This is intended to let non-coders read and edit tests. They also provide the option to write test code in JavaScript.
Testim
Testim.io is a tool that uses a Chrome extension to record test steps for basic tests. To create more complex tests, you’ll need to insert custom code via a built-in JavaScript editor. It has integrations with TestRail, GitHub, Applitools, BrowserStack, and Sauce Labs.
Testim.io’s paid plans only allow you to run between two and eight concurrent tests, and there’s an absolute limit to the number of tests you can run in a month.
Mabl
Mabl is a record-and-playback codeless test tool. You can record tests with a Chrome extension and then run them in a browser of your choosing including Firefox, Internet Explorer, and Safari. You also can store reusable code modules which are common test sequences that can be inserted into future tests.
They provide detailed reporting on passed and failed tests, JavaScript errors, broken links, and visual changes.
Get Started with Automated Functional Testing with Rainforest QA
With a functional testing tool that actually tests the UI, you can be confident your app is free of visual bugs while also testing the functionality of your application. Rainforest QA is a no-code automated testing platform that lets you build functional tests to automatically check the visual components of your app with every step. It’s a scalable, all-in-one solution that’s suitable for small teams just getting started with automated GUI testing or teams practicing continuous integration and regularly running a test suite of 500+ automated software tests.
Talk to us about setting up a Rainforest plan that fits your needs.
