Testim is an automated software testing tool that was developed with the goal of making test writing and maintenance easier by incorporating artificial intelligence (AI).

Many teams choose Testim for automated software testing for two reasons:

  • Their record-and-playback feature makes it easier for non-technical team members to help create tests.
  • Their AI-based “Smart Locators” make tests less susceptible to minor changes in the underlying code.

However, teams using Testim will eventually run into one of these problems:

  • The record-and-playback feature can only support simple test cases (so, only someone who knows JavaScript can create complex use cases).
  • If “Smart locators” are activated, the tests can potentially miss visual bugs that real users would immediately notice. On the other hand, if “Smart locators” are deactivated, the tests can easily break from minor changes to the underlying code, which leads to more maintenance.
  • It can be difficult to scale because their pricing structure limits the number of tests you can run simultaneously and the total number of tests you can run per month.

In this post, we compare 10 Testim alternatives — starting with our solution, Rainforest QA. We’ll discuss how well they address each of the issues mentioned above and share a few additional features each solution has to offer. 

In this review, we’ll cover the following tools:

  1. Rainforest QA
  2. Selenium IDE
  3. Mabl
  4. Katalon Studio
  5. Cypress
  6. Ranorex Studio
  7. Appium
  8. Leapwork
  9. Tricentis Tosca
  10. UI Vision RPA

1. Rainforest QA: Scalable No-Code Testing for Any Team

Rainforest QA is a codeless test automation solution that mimics how real users interact with software on the visual layer. 

Here’s how we address the three issues mentioned above:

Anyone Can Create Complex Test Cases — without Code

Record-and-playback tools like Testim tend to be best-suited for testing basic navigational elements of a webpage, like selecting menu options, following links, clicking on buttons, or entering type into forms. Often, it can be hard to tell what’s actually being tested with each step. 

For example, a test written with a record-and-playback tool may look like it’s verifying that a page has loaded when in reality it’s simply waiting the preset amount of time before moving on to the next test step. In order for the test to verify that a page has actually loaded, a developer will typically need to add extra code manually.

Additionally, the test scripts these recorders create tend to be much larger than test scripts coded by hand, meaning they take longer to execute and they contain a lot of unnecessary code. This makes it very tedious to debug the test scripts — and again, it’s a job that can only be done by someone with programming skills. 

All of this means you’ll often need a full-time quality assurance engineer on the team to help write and maintain tests.

Rainforest QA’s proprietary automation takes a different approach that allows anyone to write a test using our simple visual test writer instead of writing code or using a test recorder. 

Here’s how it works: 

To write or edit a test step, start by choosing from a present menu of actions (e.g. ‘click,’ ‘observe’ or ‘fill’). Then, with a click-and-drag of your mouse, take a screenshot of the element you want to apply the action to. 

Add a Click Action in Rainforest QA: Try for Free Button

You can add things like additional wait times or the exact text you want typed into a text box, for example. Rainforest QA also provides a library of randomized data for things like names, email addresses, credit card numbers, etc. This saves testers a lot of time that would’ve been spent creating that data themselves. 

Additionally, anyone can look at the test steps on the left-hand side of the screen and know exactly what’s being tested with each action — as shown in the image below.

Rainforest Signup Flow example

To speed up the test writing process, you can use embedded tests

Embedded tests allow you to create a frequently used sequence of test steps just once and embed that sequence into any test that needs it (e.g. a signup sequence)

Rainforest Signup Flow: Embedded Test example

When you’re ready to run a set of tests, you can kick them off from within the Rainforest platform or from within your CI/CD pipeline via our API, CLI, CircleCI Orb, or GitHub Action. Either way, you’ll be able to run hundreds of tests simultaneously using our network of virtual machines.

Spend Less Time on Maintenance by Testing What Real Users Actually See

Most code-based tools rely on element locators in the underlying code of an application to find and interact with elements to perform test steps. If the element locators change by even one character, your test could break — even if nothing’s changed on the UI.

To solve this problem, Testim offers something called ‘Smart Locators’. Smart Locators are designed to ignore minor changes and continue with the test even if an element’s locator has changed. This works pretty well for keeping the tests from breaking, however it introduces a different problem: If the change in the locator does reflect a change in the UI, the Smart Locators may make the test pass even if the change introduced a critical visual bug. 

That’s why we think it’s better to use pixel-matching to test the UI directly, rather than testing the underlying code. This avoids the issue of tests breaking due to minor, behind-the-scenes code changes and ensures you catch visual bugs. (That’s also why Rainforest is a good visual regression testing option.)

For situations where you simply want to know that a real person will be able to find and use an element and you’re less concerned with specific visual characteristics, you can use text-matching.

Allow Text Matching: Try for Free Button

Instead of looking for a perfect match of the screenshot of an element, text-matching looks for specific text on the screen. 

For example, both buttons in the image below have different colors, shapes, and font sizes. However, both buttons have the words ‘Buy Now’. With text-matching turned on, the test would pass with either version of the button.

Initial vs Updated "Buy Now" Button example

Scale Up at A Moment’s Notice

Testim offers a free tier of their service which includes 1,000 test runs per month. At this level, you can only run one test at a time, and you’re limited to one account per organization. 

Once your testing efforts grow out of the free tier, you can opt-in for a contracted paid tier. Each tier sets different limits on how many tests you can run each month and how many of those tests can be run simultaneously.

With Rainforest, there’s no limit on how many tests you can run each month or how many users you can add. You can also run hundreds of tests simultaneously. And you won’t be locked into a contract, which means you can easily scale testing up or down to fit project requirements. 

Finally, our plans provide everything you need to write, run, and manage a full suite of functional tests. That includes a vast network of virtual machines (that can be configured with almost any combination of the big four browsers and macOS or Windows operating systems) and integrations with communication tools like Slack and Microsoft Teams. 

Bonus Feature: Quickly Triage Test Results with Video Replays

Understanding why a test failed is often one of the most time-consuming parts of testing.

Most teams using Testim find that only someone who knows JavaScript will be able to triage test results. And, Testim puts limits on how long you have access to test results and test history. For example, their free tier only lets you keep test results for seven days and test history for one month. This can be a problem because often it’s necessary to compare the current test run to previous test runs to understand what happened, especially with flaky tests

Not having test history can also make it difficult to notice trends as to why tests are failing or breaking — which means you won’t be able to fix the problem. 

With Rainforest QA, you won’t need to know a coding language to figure out why a test failed. Plus, you’ll always have access to all test results and test run history. 

Here’s how it works:

Rainforest QA automatically records a video of every test whether it passes or fails. Then, any failed test step will be highlighted in red with a brief explanation of why it failed. 

Rainforest Signup Flow: Element Mismatch

Under every test step, you’ll see an ‘investigate action’ button which will show you even more details on why the test step failed including the closest match found.

Book a Flexible Trip: Investigate Action

You’ll also be given HTTP logs and browser logs which can help you determine if the test failure was due to an inconsistency in the test environment.

Airbnb: Download HTTP Logs, Browser Logs, Video

Finally, once you’ve found a bug that needs to be addressed, you can use our Jira integration to automatically create a ticket to send to the development team. The Jira ticket will include the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.

Talk to us about setting up a Rainforest plan that fits your needs.

2. Selenium IDE

Selenium IDE homepage: Open source record and playback test automation for the web

Selenium is the oldest solution for automating browsers to perform web testing. Selenium IDE is their record-and-playback tool. Teams that are already using Selenium may like Selenium IDE, however, it’ll still have the same drawbacks that most record-and-playback testing solutions have.  

Selenium also offers a WebDriver that supports multiple programming languages (including Java, Python, Ruby, C#, JavaScript, Perl, and PHP) and is compatible with Selenium IDE. 

Many software testing tools (including several in this list) are built on Selenium. We’ve written in more detail about Selenium’s disadvantages, but Selenium’s downsides are very similar to every other tool — including Testim — that interacts with the underlying code of an application (or the DOM) instead of the visual layer of the user interface. 

3. Mabl

Mabl homepage: Intelligent Test Automation for Agile Teams

Mabl is a SaaS solution that integrates software testing into developers’ workflows. Their test authoring feature utilizes record-and-playback technology like so many others via a Chrome extension, and it has the same limitations as other record-and-playback tools. Although you can only record tests in Chrome, you can run tests in Firefox, Internet Explorer, or Safari. You can also store common test sequences as “reusable flows” and insert them into future tests, which saves time when creating tests. 

Unlike a lot of tools, Mabl offers detailed information surrounding passed and failed tests, JavaScript errors, broken links, and visual changes. 

4. Katalon Studio

Katalon Studio homepage: An all-in-one test automation solution

Katalon Studio is a record-and-playback tool built on the Selenium WebDriver (it also works with Groovy and Java). Unlike some record-and-playback tools, Katalon lets developers stop and restart a recording at any point during test creation or test execution to add or edit test steps or assertions. This makes it much easier to create complex test cases and fix broken tests, however, you’ll still be dependent on software engineers for test authoring.

You can learn more about Katalon in our article that compares Katalon vs. Selenium vs. Cypress vs. Rainforest QA.  

5. Cypress

Cypress homepage: The web has evolved. Finally, testing has too.

Cypress.io is a test automation solution built to make test authoring in JavaScript easier for developers. They offer preset actions to minimize the amount of code that needs to be written out manually. With the Cypress software installed onto your hard drive, you can view actions in the Test Runner browser on one side of the screen and the corresponding lines of code on the other side of the screen in real-time.  

They offer an open-source version of their product, but eventually, most agile teams find they need to upgrade to a paid version to keep up with testing.

6. Ranorex Studio

Ranorex Studio homepage: Functional UI Test Automation

Ranorex is a record-and-playback tool for desktop, web, or mobile apps. The record-and-playback feature will simply generate code for you like most codeless tools, but you can also add screenshot verifications to test steps. 

You can choose between several different settings to compare screenshots, such as pixel by pixel or by overall shapes. You can also choose to have the test look for the screenshot in one specific area of the screen or in any area of the screen.

7. Appium

Appium homepage: Appium is an open source test automation framework for use with native, hybrid and mobile web apps.

Appium modifies Selenium so you can test browsers on mobile devices instead of just desktops. It supports most mobile testing platforms including iOS, Android, and Windows apps. Unlike many of the tools on this list that offer low-code options, Appium is entirely dependent on programming skills. 

Appium supports native, web, and hybrid mobile applications. Plus, it’s very customizable, so it can be integrated into nearly any software development tool.

To help with debugging, Appium lets you configure Appium to take a snapshot of the underlying code at the point of break to help identify why a test failed.

8. Leapwork

Leapwork homepage: Test automation that's accessible and easy to maintain.

Leapwork is a no-code visual test automation tool that utilizes a point-and-click generator (similar to a record-and-playback tool) and visual flowcharts. However, you will still need a testing grid (e.g. LambdaTest or BrowserStack) if you want to run multiple tests at once. 

Leapwork is built with IT personnel in mind, so most users find the interface confusing unless they are very technically inclined.

They support desktop and web application testing. 

Leapwork doesn’t provide pricing information on their website. Instead, you can reach out to their team to discuss a tailored contract. 

9. Tricentis Tosca

Tricentis Tosca homepage: Intelligent test automation

Tricentis Tosca is an end-to-end testing tool built for teams practicing agile methodologies. They support continuous testing with integrations for various dev tools. Many users report that the test management features help increase their productivity. 

Tricentis Tosca offers various features to help with cross-browser testing such as self-healing testing technology. Tricentis Tosca also offers real-time risk assessment of business processes so you can decide what testing is the most important at any given moment.

10. UI Vision

UI Vision: Visual RPA Desktop Automation Made Easy

Like Selenium, UI Vision automates browsers for testing web applications. However, their robotic process automation (RPA) plugin also lets testers perform UI testing on desktop applications. 

In addition to codeless UI testing, they offer web scraping and screen scraping. They offer an open-source tier and three one-time fee packages. Each tier has some limitations on how many drag-and-drop and preset mouse events you’ll have access to for writing tests. And, all but the highest tier only offer community support. 

Choosing the Right Testim Alternative

Testim may not be the right test automation platform for every organization because it requires programming skills to use, needs extensive maintenance, and scales poorly.

Rainforest QA solves these problems by letting anyone manage testing and by reducing time spent on maintenance. It’s a scalable, all-in-one solution that’s suitable for small teams just getting started with automated UI testing or teams practicing continuous integration and regularly running a test suite of 500+ automated software tests.

Talk to us about setting up a Rainforest plan that fits your needs.