Testim is an automated software testing tool that was developed with the goal of making test writing and maintenance easier by incorporating artificial intelligence (AI).
Many teams choose Testim for automated software testing for two reasons:
- Their record-and-playback feature makes it easier for non-technical team members to help create tests.
- Their AI-based “Smart Locators” make tests less susceptible to minor changes in the underlying code.
However, teams using Testim will eventually run into one of these problems:
- If “Smart locators” are activated, the tests can potentially miss visual bugs that real users would immediately notice. On the other hand, if “Smart locators” are deactivated, the tests can easily break from minor changes to the underlying code, which leads to more maintenance.
- It can be difficult to scale because their pricing structure limits the number of tests you can run simultaneously and the total number of tests you can run per month.
In this post, we compare 10 Testim alternatives — starting with our solution, Rainforest QA. We’ll discuss how well they address each of the issues mentioned above and share a few additional features each solution has to offer.
In this review, we’ll cover the following tools:
- Rainforest QA
- Selenium IDE
- Katalon Studio
- Ranorex Studio
- Tricentis Tosca
- UI Vision RPA
1. Rainforest QA: Scalable No-Code Testing for Any Team
Rainforest QA is a codeless test automation solution that mimics how real users interact with software on the visual layer.
Here’s how we address the three issues mentioned above:
Anyone Can Create Complex Test Cases — without Code
Record-and-playback tools like Testim tend to be best-suited for testing basic navigational elements of a webpage, like selecting menu options, following links, clicking on buttons, or entering type into forms. Often, it can be hard to tell what’s actually being tested with each step.
For example, a test written with a record-and-playback tool may look like it’s verifying that a page has loaded when in reality it’s simply waiting the preset amount of time before moving on to the next test step. In order for the test to verify that a page has actually loaded, a developer will typically need to add extra code manually.
Additionally, the test scripts these recorders create tend to be much larger than test scripts coded by hand, meaning they take longer to execute and they contain a lot of unnecessary code. This makes it very tedious to debug the test scripts — and again, it’s a job that can only be done by someone with programming skills.
All of this means you’ll often need a full-time quality assurance engineer on the team to help write and maintain tests.
Rainforest QA’s proprietary automation takes a different approach that allows anyone to write a test using our simple visual test writer instead of writing code or using a test recorder.
Here’s how it works:
To write or edit a test step, start by choosing from a present menu of actions (e.g. ‘click,’ ‘observe’ or ‘fill’). Then, with a click-and-drag of your mouse, take a screenshot of the element you want to apply the action to.
You can add things like additional wait times or the exact text you want typed into a text box, for example. Rainforest QA also provides a library of randomized data for things like names, email addresses, credit card numbers, etc. This saves testers a lot of time that would’ve been spent creating that data themselves.
Additionally, anyone can look at the test steps on the left-hand side of the screen and know exactly what’s being tested with each action — as shown in the image below.
To speed up the test writing process, you can use embedded tests.
Embedded tests allow you to create a frequently used sequence of test steps just once and embed that sequence into any test that needs it (e.g. a signup sequence).
When you’re ready to run a set of tests, you can kick them off from within the Rainforest platform or from within your CI/CD pipeline via our API, CLI, CircleCI Orb, or GitHub Action. Either way, you’ll be able to run hundreds of tests simultaneously using our network of virtual machines.
Spend Less Time on Maintenance by Testing What Real Users Actually See
Most code-based tools rely on element locators in the underlying code of an application to find and interact with elements to perform test steps. If the element locators change by even one character, your test could break — even if nothing’s changed on the UI.
To solve this problem, Testim offers something called ‘Smart Locators’. Smart Locators are designed to ignore minor changes and continue with the test even if an element’s locator has changed. This works pretty well for keeping the tests from breaking, however it introduces a different problem: If the change in the locator does reflect a change in the UI, the Smart Locators may make the test pass even if the change introduced a critical visual bug.
That’s why we think it’s better to use pixel-matching to test the UI directly, rather than testing the underlying code. This avoids the issue of tests breaking due to minor, behind-the-scenes code changes and ensures you catch visual bugs. (That’s also why Rainforest is a good visual regression testing option.)
For situations where you simply want to know that a real person will be able to find and use an element and you’re less concerned with specific visual characteristics, you can use text-matching.
Instead of looking for a perfect match of the screenshot of an element, text-matching looks for specific text on the screen.
For example, both buttons in the image below have different colors, shapes, and font sizes. However, both buttons have the words ‘Buy Now’. With text-matching turned on, the test would pass with either version of the button.
Scale Up at A Moment’s Notice
Testim offers a free tier of their service which includes 1,000 test runs per month. At this level, you can only run one test at a time, and you’re limited to one account per organization.
Once your testing efforts grow out of the free tier, you can opt-in for a contracted paid tier. Each tier sets different limits on how many tests you can run each month and how many of those tests can be run simultaneously.
Rainforest QA, on the other hand, makes software test automation accessible to anyone with a simple and scalable pricing structure.
There’s no limit on how many tests you can run each month or how many users you can add. You can also run hundreds of tests simultaneously. And you won’t be locked into a contract, which means you can easily scale testing up or down to fit project requirements.
Finally, our Startup plan provides everything you need to write, run, and manage a full suite of functional tests. That includes a vast network of virtual machines (that can be configured with almost any combination of the big four browsers and Mac or Windows operating systems) and integrations with communication tools like Slack and Microsoft Teams.
Bonus Feature: Quickly Triage Test Results with Video Replays
Understanding why a test failed is often one of the most time-consuming parts of testing.
Not having test history can also make it difficult to notice trends as to why tests are failing or breaking — which means you won’t be able to fix the problem.
With Rainforest QA, you won’t need to know a coding language to figure out why a test failed. Plus, you’ll always have access to all test results and test run history.
Here’s how it works:
Rainforest QA automatically records a video of every test whether it passes or fails. Then, any failed test step will be highlighted in red with a brief explanation of why it failed.
Under every test step, you’ll see an ‘investigate action’ button which will show you even more details on why the test step failed including the closest match found.
You’ll also be given HTTP logs and browser logs which can help you determine if the test failure was due to an inconsistency in the test environment.
Finally, once you’ve found a bug that needs to be addressed, you can use our Jira integration to automatically create a ticket to send to the development team. The Jira ticket will include the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.
Talk to us about setting up a Rainforest plan that fits your needs.
2. Selenium IDE
Selenium is the oldest solution for automating browsers to perform web testing. Selenium IDE is their record-and-playback tool. Teams that are already using Selenium may like Selenium IDE, however, it’ll still have the same drawbacks that most record-and-playback testing solutions have.
Many software testing tools (including several in this list) are built on Selenium. We’ve written in more detail about Selenium’s disadvantages, but Selenium’s downsides are very similar to every other tool — including Testim — that interacts with the underlying code of an application (or the DOM) instead of the visual layer of the user interface.
Mabl is a SaaS solution that integrates software testing into developers’ workflows. Their test authoring feature utilizes record-and-playback technology like so many others via a Chrome extension, and it has the same limitations as other record-and-playback tools. Although you can only record tests in Chrome, you can run tests in Firefox, Internet Explorer, or Safari. You can also store common test sequences as “reusable flows” and insert them into future tests, which saves time when creating tests.
4. Katalon Studio
Katalon Studio is a record-and-playback tool built on the Selenium WebDriver (it also works with Groovy and Java). Unlike some record-and-playback tools, Katalon lets developers stop and restart a recording at any point during test creation or test execution to add or edit test steps or assertions. This makes it much easier to create complex test cases and fix broken tests, however, you’ll still be dependent on software engineers for test authoring.
They offer an open-source version of their product, but eventually, most agile teams find they need to upgrade to a paid version to keep up with testing.
6. Ranorex Studio
Ranorex is a record-and-playback tool for desktop, web, or mobile apps. The record-and-playback feature will simply generate code for you like most codeless tools, but you can also add screenshot verifications to test steps.
You can choose between several different settings to compare screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.
Appium modifies Selenium so you can test browsers on mobile devices instead of just desktops. It supports most mobile testing platforms including iOS, Android, and Windows apps. Unlike many of the tools on this list that offer low-code options, Appium is entirely dependent on programming skills.
Appium supports native, web, and hybrid mobile applications. Plus, it’s very customizable, so it can be integrated into nearly any software development tool.
To help with debugging, Appium lets you configure Appium to take a snapshot of the underlying code at the point of break to help identify why a test failed.
Leapwork is a no-code visual test automation tool that utilizes a point-and-click generator (similar to a record-and-playback tool) and visual flowcharts. However, you will still need a testing grid (e.g. LambdaTest or BrowserStack) if you want to run multiple tests at once.
Leapwork is built with IT personnel in mind, so most users find the interface confusing unless they are very technically inclined.
They support desktop and web application testing.
Leapwork doesn’t provide pricing information on their website. Instead, you can reach out to their team to discuss a tailored contract.
9. Tricentis Tosca
Tricentis Tosca is an end-to-end testing tool built for teams practicing agile methodologies. They support continuous testing with integrations for various dev tools. Many users report that the test management features help increase their productivity.
Tricentis Tosca offers various features to help with cross-browser testing such as self-healing testing technology. Tricentis Tosca also offers real-time risk assessment of business processes so you can decide what testing is the most important at any given moment.
10. UI Vision
Like Selenium, UI Vision automates browsers for testing web applications. However, their robotic process automation (RPA) plugin also lets testers perform UI testing on desktop applications.
In addition to codeless UI testing, they offer web scraping and screen scraping. They offer an open-source tier and three one-time fee packages. Each tier has some limitations on how many drag-and-drop and preset mouse events you’ll have access to for writing tests. And, all but the highest tier only offer community support.
Choosing the Right Testim Alternative
Testim may not be the right test automation platform for every organization because it requires programming skills to use, needs extensive maintenance, and scales poorly.
Rainforest QA solves these problems by letting anyone manage testing and by reducing time spent on maintenance. It’s a scalable, all-in-one solution that’s suitable for small teams just getting started with automated UI testing or teams practicing continuous integration and regularly running a test suite of 500+ automated software tests.
Talk to us about setting up a Rainforest plan that fits your needs.