While Selenium IDE has been popular in the automated testing world, most teams using Selenium run into these problems:
- Only software developers can run the tests. Selenium is an open-source test automation framework with its own scripting language, so a Selenium engineer has to write and maintain the tests.
- It’s costly to maintain. Even minor, behind-the-scenes code changes can break Selenium tests, and every broken test needs developer time to fix.
- It means paying for additional services. An organization using Selenium has to pay for tools like Sauce Labs or BrowserStack to efficiently scale up testing.
There are lots of different test automation tools—from easier ways to generate Selenium code to no-code SaaS (software as a service) options—all trying to solve these problems.
In this article, we’ll talk about key differences between the following tools that represent the three main approaches to solving Selenium shortcomings:
- Rainforest QA: A no-code testing tool for testing the visual layer of an application with user-friendly features that help save time.
- Cypress.io: Another open-source testing framework similar to Selenium but written in JavaScript and with significant upgrades that help integrate automation testing into the development workflow. Unlike Selenium, Cypress doesn’t support popular programming languages like Java, Python, Ruby, C#, PHP, etc.
- Katalon Studio: One of the dozens of no-code interfaces that generate Selenium scripts and also allow for developers to make manual edits.
All three of these test automation tools for web apps offer built-in solutions for test management and parallel testing (i.e. running multiple tests at once), so you don’t have to pay for additional services like a testing grid. To help you understand how each approach works in practice, we’ll discuss them in terms of:
- Testing the visual layer vs. the underlying code
- Writing test scripts
- Maintaining tests
- Finding the root cause of test failures
- Testing actions outside of a browser
- Cross-browser testing abilities and compatible web browsers
- Pricing
Testing the Visual Layer vs. the Underlying Code
Rainforest: A No-Code Visual UI Testing Solution That Uses Pixel-Matching
Rainforest QA is the only testing software that tests the visual layer of your app rather than the underlying code. We do this by pixel-matching elements of your software apps. This means the tests search for matching arrangements of pixels rather than code-based locators to complete various actions. (In the next section we go over how to create a test.)
Tests that use code-based locators or XPath locators may indicate that an element is visible without verifying that it is actually visible. Pixel-matching is more reliable because it’s testing what the user will see rather than what the computer thinks they should be seeing.
In this article, we go into more detail about web testing scenarios where the code is present but the element isn’t visible for the user (e.g. when a popup hides the element being tested from the user).
Cypress.io: An Open-Source Testing Framework for Testing the Underlying Code Using JavaScript
Cypress tests are built on top of Mocha and Chai (JavaScript testing frameworks) and test the functionality of the underlying code of a web app. If you already use JavaScript frameworks, you’ll be familiar with the syntax and can get started using Cypress quickly, but you’ll still be testing the underlying code instead of the visual layer of your app.
Another aspect to consider is the complexity of modern JavaScript. Because modern JavaScript is generated by a computer (rather than hand-written in simple lines of code), there are more layers and wrappers around each element that make it difficult to identify where one element ends and another begins. You can add special properties during setup to help you find these code-based locators, but that takes extra time during test creation and maintenance.
Katalon: A No-Code Tool That Tests the Underlying Code with Selenium Scripts
Katalon is a no-code tool that uses Selenium scripts to test the underlying code of an app. To create a test, you execute the steps of the user path you want to test, and Katalon records the code associated with your movements. While this provides a way for individuals without technical experience to get involved, you’ll need coding experience to identify the reason why tests fail and to actually fix them.
Who Can Write Tests and Who Owns QA
Rainforest QA: Anyone Can Write and Maintain Tests
Rainforest QA lets anyone create (and maintain) tests in a matter of minutes whether they have a technical background or not. This means anyone from the product team can jump in and help as needed.
To create (or edit) a test in Rainforest QA, you first select a preset action (e.g. ‘click’, ‘fill’, or ‘observe’). Then, in the virtual machine previewing your live app, click-and-drag to select the element you want to apply the action to.
Watch this 2-minute video to see how easy it is to create your first test.
Because anyone can write and understand Rainforest tests, anyone on your team who cares about or has a vested interest in quality can own QA.
For example, if a product manager is being held responsible for the outcome of a particular feature, they can help with QA before release and make informed decisions about the trade-off between speed and test coverage.
Cypress.io: Usable by Front-end Developers (or Anyone Who Knows JavaScript)
Cypress test scripts can be written by anyone who knows JavaScript (which includes most front-end developers). If you don’t know JavaScript, you’ll have a steep learning curve.
Once you install the Cypress software onto your hard drive, the Test Runner displays the actions in the browser on one side of the screen and the corresponding lines of code on the other side of the screen in real-time.
Cypress also provides preset functions—ex. Cy.wait (amount of time you want the test to ‘wait for’ an element to load)—and tools (such as Clocks, Stubs, and Spies) that help the developer control test functions and speed up the test writing process.
Since you need to know JavaScript to write and maintain Cypress tests, only software engineers or QA engineers can do QA with Cypress. Even if they’re responsible for QA, most developers are held more accountable to release speed than overall product quality. This can lead them to prioritize speed over quality, which can eventually lead to quality issues that could harm the company.
Katalon: Anyone Can Create a Test (But Developers Have to Fix Complex Bugs)
Katalon tests can be created by anyone and automation engineers can step in any time to manually manipulate the code.
Katalon provides helpful tools such as built-in keywords and project templates to help create test cases faster. While using Katalon takes some of the pressure off of developers by letting anyone help create the tests, software engineers are the only ones who can update tests. This often leads to a bottleneck effect in your release pipeline.
Maintaining Automation Tests
Rainforest QA: Minor Changes in Underlying Code Won’t Break Tests
Rainforest QA tests are less susceptible to breaking from minor code changes that don’t affect the UI.
Let’s say you want to rename a frequently used code-based locator to match updates in your product. This kind of change could cause tests created with Selenium or Cypress to fail because they use code-based locators to find objects during the tests. But as long as the element is still visible, a Rainforest test would be able to proceed.
Rainforest tests only break if there’s a change to the UI, due to a real bug that will affect the user experience.
Although Rainforest tests can break because of UI changes, our text matching feature is one way to help control test brittleness. Text matching examines the content of an element rather than the appearance.
For example, the buttons below both say “Buy Now” even though the colors and shapes are different.
If text matching is enabled, the test will pass with either version of the button. If the text locator is not enabled, it will only pass if your original screenshot matches the version being tested.
This is helpful if you’re only interested in testing the content of the button rather than the design.
Cypress.io: Easy to Fix Tests If You Know JavaScript
Cypress tests are relatively easy to fix if you know JavaScript, but they can be flaky in response to changes in the underlying code.
Cypress does offer tutorials and lists of best practices to help you write less brittle tests, but these are mostly just specific ways to wrap locators in additional text.
Katalon: More Time-Consuming than Other Solutions
Katalon tests are more time-consuming to fix than other options. A software engineer can stop a test at any point, make a few changes to fix a bug, and then resume the test, but you have to know Selenium to do this.
In most cases, if someone without any programming skills wants to fix a test, they’d have to re-record entire sections or entire tests.
Finding the Root Cause of Test Failures
Rainforest QA: Understand Every Test Failure within Minutes
Rainforest QA records a video of every test, whether it passes or fails, so you can play back the video to see why a failure happened.
You can configure other automated testing software to take a screenshot of the UI or a snapshot of the underlying code at the point of a failure, but this doesn’t tell you why a failure happened.
Let’s say you want to test a signup process and the ‘signup was successful’ message doesn’t appear at the end of your test, causing the test to fail. A screenshot of the UI for this last step won’t tell you if there’s a problem with the signup information going through or if there’s a problem with the message itself.
Reproducing and triaging bugs is faster and simpler using video recordings because you can view any failure in the context of the whole rather than as an isolated event.
Rainforest QA also has a suggested fix feature to help you quickly find the root cause of a test failure. The suggested fix feature is triggered any time a screenshot of an element can’t be found. With just a couple of clicks, you can take a new screenshot and fix most broken test steps.
Note: For more best practices to reduce test automation maintenance, read this article.
Cypress.io: Sort through Lines of Code to Understand Test Failures
If you’re running tests using Cypress’s API, it automatically takes screenshots of any test failure and includes messages with suggestions for what went wrong and how to fix it.
If the tests are run from the Command Line Interface (CLI), Cypress can record videos of the lines of code during test execution but you have to set this up manually every time. In addition, the videos get deleted each time a new run starts.
As in most automated testing software, the failed step will be highlighted in red. Because modern JavaScript can be complicated to read, understanding a test failure and fixing the test can only be done by someone who has working knowledge of JavaScript.
Katalon: Only Software Engineers Will Understand Test Failures
Katalon also has a record and playback feature. Like Cypress, you can see the actions in a browser on one side of the screen and the underlying code for those actions on the other side of the screen.
If you have coding experience, this is a nice feature. If you don’t have any coding experience, you won’t know how to read or fix the tests.
Testing Actions Outside of a Web Browser
Rainforest QA: Verify Downloads, Test Web Browser Interactions, and More
A Rainforest test is essentially a set of instructions for a virtual machine to perform. This means it can interact with web browsers, desktop apps, and the UI of the operating system.
You can test actions such as opening and closing web browsers or desktop apps or see how your web application interacts with browsers with ad-blockers.
Here’s a video with a detailed walkthrough of creating a test that saves a file to the desktop from one location, then uploads the file to Google Drive.
Cypress.io and Katalon: Only Test Elements within a Browser Tab
Cypress and Katalon tests are essentially a set of instructions to be performed within a browser page (Katalon uses the Selenium Webdriver and Cypress runs tests directly inside the browser). This means they can’t leave the browser in any way to open new tabs or verify downloads. It also means there’s limited support for iFrames.
Cypress has some workarounds that allow you to verify assertions about downloaded or uploaded files, but it can’t truly test a web app’s interaction with a desktop app.
Cross-Browser Testing and Compatible Browsers
Rainforest QA: Supports New and Old Versions of Major Browsers
Rainforest QA gives you access to current and older versions of any of the four major browsers (i.e. Safari, Microsoft Edge, Chrome, and Firefox).
Because Rainforest QA tests on the visual layer and each browser renders slightly different visuals, Rainforest QA’s automation service doesn’t offer cross-browser support. Most often, the types of bugs that cause a test to pass in one browser but fail in another will cause glaringly obvious bugs (such as a completely white screen).
We recommend you choose a primary browser to run the majority of your tests in and also run a few basic tests in the other browsers. This way, you’ll still catch most of the important bugs in every browser. Alternatively, our community of crowd testers can run your test scripts across any combination of browsers.
Cypress.io: Supports Newer Versions of Chrome, Microsoft Edge, and Firefox
Cypress offers cross-browser support on Chrome (64+ and Electron), Microsoft Edge (79+), and Firefox (86+), however, currently they don’t offer testing abilities on Safari.
Katalon: Supports Newer Versions of Major Browsers
Katalon offers cross-browser testing on Chrome (58+), Firefox (56+), Safari (9+), Microsoft Edge, Linux, and Internet Explorer (9+).
Pricing
Rainforest QA
Rainforest offers annual plans that start at a tenth of the cost of hiring a QA engineer in the U.S. You can create and maintain unlimited tests and invite as many team members as you need. You can spend as much time as you want creating and maintaining tests.
Test cases run in parallel, giving you extra-fast results, and you can run these tests on the latest version of all major desktop browsers.
Cypress.io
Free
The free version of Cypress allows for multiple tests to be run at once, test debugging through screenshots and manual editing, API integration, analytical reporting, and community support. You can have 3 users and get up to 500 test results per month.
Team
The ‘Team’ plan includes everything in the free version plus flake detection features, Jira integration, and direct support through email. This plan supports up to 10 users and 10,000 test results per month for a cost of $75 per month.
Business
In addition to all ‘Team’ features, the ‘Business’ plan offers organization tools such as SSO and the ability to run failed specs first and cancel runs after a failure. For $300 per month, this plan supports up to 40 users and 10,000 test results per month.
Enterprise
Cypress’ ‘Enterprise’ plan offers full customization in pricing, number of users, and legal terms. They also offer personalized support.
Katalon
Katalon Studio
Katalon Studio is Katalon’s free version of their software intended for individual use. They offer web, desktop, mobile, and API testing. At this level, the testing capabilities are pretty basic. You can run the test and know whether it failed or not, but you won’t have access to screenshots, recordings, or suggested fix features.
Katalon Studio Enterprise
With the Katalon Studio Enterprise plan, you have access to suggested fix features, screenshots of test breaks, data-driven testing, report history, private plugins, and help desk support.
The Katalon pricing structure is built on licenses billed either monthly or yearly. A monthly license can only be for one device (what they call a ‘node-locked’ license), whereas a yearly license can be for one device or for 3 devices per session (what they call a ‘Floating’ license).
Depending on what you choose, the Katalon Studio Enterprise plan costs $76 per month per license, $839 per year per license, or $1,689 per year per license.
Katalon Runtime Engine
Rather than a separate package, Katalon Runtime Engine is an add-on for either studio plan mentioned above. It supports any execution environment including console mode or CLI. You can schedule tests and perform batch operations. The Katalon Runtime Engine is $54 per month per license, $599 per year per license, or $1,319 per year per license with a minimum purchase of 50 licenses at a time.
The Best Option for End-to-End Testing
With Rainforest QA, anyone can automate and maintain end-to-end tests without learning a new programming language. It helps QA teams keep up with agile development teams and teams doing continuous integration and continuous delivery.
It’s a scalable, all-in-one test automation solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams regularly running 500+ software tests.
Talk to us about setting up a Rainforest plan that fits your needs.