Selenium is the oldest and most well-recognized automated testing tool for web apps, so a lot of software teams try it out when they first experiment with automated testing. But most teams quickly run into three Selenium disadvantages:
- It doesn’t truly test the user interface (UI)—which is what your users actually interact with.
- Writing and maintaining test scripts is very time-consuming and requires specialized coding skills. It’s not friendly to beginners.
- Identifying why a test failed can also be time-consuming and complicated.
Because of these challenges, many software companies that use Selenium find testing to be a huge drain on time, effort, and resources. That’s why we designed Rainforest QA—to solve the limitations of Selenium and make QA easy to scale and maintain.
In this post, we’ll discuss each of the disadvantages of Selenium in detail and show how Rainforest QA solves those problems.
Talk to us about setting up a Rainforest plan that fits your testing needs.
Disadvantage #1: Selenium Only Tests the Underlying Code, Not the Visual Layer
Selenium tests search for element locators in the underlying code of an application in order to interact with web elements to perform test steps. Usually, these steps involve interacting with the element to move the test forward (e.g., clicking the submit button) or verifying the presence of an element. You can also add test steps that verify various visual settings of the element, such as size, position, or color. If the element locator is present and the settings are correct, the test should pass.
However, this approach to test automation has two inherent downsides:
- Sometimes a Selenium test will pass even though there’s a bug on the UI that a real user would immediately notice. This happens because the locator is present (with the correct settings) in the underlying code but a visual bug is causing the element to be hidden or rendered incorrectly on the UI. For example, a pop up could be covering the element you’re testing for or the element could be rendered off screen. This makes Selenium unreliable for catching visual regressions or visual bugs.
- Sometimes Selenium tests will fail even though nothing has been changed in the UI. This is because even the slightest change in an element locator can break a Selenium test. For example, let’s say your team renamed the locator during an update but didn’t update the test, or accidentally missed a letter in the locator when creating the test. Either way, a Selenium test would fail because it couldn’t find an exact match—even though the element would look and act the same way on the visual layer.
While there are workarounds to help ensure you catch visual bugs (more on this in a later section), in the end, Selenium tests make assumptions about elements on the UI rather than testing what real users will actually see.
Solution: Rainforest Tests Interact with the Visual Layer to Validate the True User Experience
Rainforest QA tests interact with the visual layer of your application. Instead of searching for element locators in the underlying code of an application, Rainforest tests search for elements (like buttons or form fields) based on a matching combination of pixels in the UI.
We provide a detailed tutorial about how to write a Rainforest test in a later section.
This is the closest to how a real human would interact with the application because the test only uses what’s available visually and doesn’t touch the underlying code. This also means Rainforest tests won’t break because of minor changes to element locators that don’t affect the UI.
Disadvantage #2: Writing and Maintaining Selenium Tests Takes a Long Time and Only Developers Can Do It
Even though Selenium supports many programming languages, developers using Selenium usually find that writing and maintaining test scripts is very time-consuming. Not only do they have to learn a new testing framework, but each individual test script requires a lot of set up and coding. To write Selenium test scripts:
- You’ll have to write code for each aspect of each element you want to test. For example, if you want to verify that a button is visible, is located next to a block of text and has certain visual attributes (shape, color, size, etc.), each of those aspects will likely need to be verified individually.
- You’ll most likely need to add custom attributes to each element of the application you want to test. Modern website and application builders generate code that can be very complex which makes it hard to pinpoint exactly which piece of code represents an element. Because of this, many teams add custom attributes that make elements easier to find. This is fairly easy to add as you build a new feature, but adding it to software that’s already been developed can be a massive chore.
- You’ll probably need to cobble together a bunch of plug-ins to make failure categorization easier and to make tests more resilient to changes in the underlying code. Selenium doesn’t offer any built-in features to help with these issues, but since it’s an open-source tool, there are a lot of plug-ins available to address specific shortcomings. By engaging with the larger testing community, you can find plug-ins and learn techniques that other teams have used to make their tests less sensitive to changes in the web application and to help categorize test failures, but if your plug-in stops working, there’s no support team available to help you troubleshoot the problem. And most of these workarounds ultimately require adding even more code to each test.
There’s a Selenium WebDriver that supports multiple programming languages (including Java, Python, Ruby, C#, JavaScript, Perl, and PHP) and a record-and-playback tool (called Selenium IDE) which can make it easier to write test scripts. In the end, however, you’ll still end up writing or generating test scripts built on code. Whenever a test fails or needs to be updated, you’ll still have a ton of code to sort through.
Solution: Rainforest’s No-Code Editor Lets Anyone Create and Maintain Tests in Minutes
Rainforest QA is a true no-code platform that lets anyone create (and maintain) tests in a matter of minutes whether they have a technical background or not.
To create (or maintain) a test step in Rainforest, you choose from a dropdown menu of preset actions—such as ‘click’, ‘fill’, or ‘scroll. Then, you take a screenshot of the element you want to apply the action to by clicking and dragging the mouse over the element.
To verify the visibility, color, shape, etc. of an element, you only need the one screenshot.
Because Rainforest tests interact with the UI directly, you won’t have to add any custom attributes to your application before writing tests.
Additionally, Rainforest QA has built-in features to help make tests more resilient to changes in the application. Not only are Rainforest tests immune to minor changes in the underlying code of the application that don’t affect the UI, but you can also choose to use text matching for any test step. With text matching turned on, Rainforest will search for the text content of an element rather than just the appearance of the element.
For example, both of the buttons below say “Buy Now” even though their appearances (i.e., color and shape) are different.
If text matching is enabled, the test will pass with either version of the button. If text matching is not enabled, it will only pass if the original screenshot matches the current version being tested. This gives you the flexibility to decide for each test step whether you want to test the entire design of the element or just the content.
Disadvantage #3: Debugging Test Failures in Selenium Takes Too Long
After a test run, you’ll want to see which tests passed and which ones failed. Selenium doesn’t have any reporting capabilities for test results. Instead, you’ll have to click into each test individually to see if it passed or failed, or add on other test management tools, such as TestNG, JUnit, or Maven.
Once you know which tests have failed, you’ll need to figure out why they failed. Selenium doesn’t offer any built-in features to help you identify why a test failed, which means it could take hours to find the cause of the failure.
You can add additional code so that a snapshot of the underlying code is automatically taken at the point of failure. However, in many situations, tests fail because of a bug in a previous part of the test. In these situations, one snapshot at the point of break won’t be enough to tell you why the failure occurred.
For example, let’s say you want to verify that a ‘Download was successful’ message appears at the end of your test. A snapshot of the underlying code could tell you that the message didn’t appear but it can’t tell you if that’s because the download itself was unsuccessful or if an error with the confirmation message is the issue.
With Selenium, you’ll most likely end up sorting through tons of code to find the small errors that caused the failure. This is often too slow and labor intensive for teams that need to release quickly and frequently.
Solution: Rainforest’s Video Replays Let Anyone Quickly Understand Test Failures
Rainforest QA automatically creates test reports for each test run that clearly show you which tests passed, which ones failed, and the run history for each test.
Rainforest also automatically records a video of every test whether it passes or fails. This lets you view the failed test step and every step leading up to it. You can also compare a failed test to a successful test, which can be really helpful when the reason for a failure isn’t immediately obvious.
Failed test steps get highlighted in red along with a brief explanation of why the step failed and an ‘investigate action’ button.
The ‘investigate action’ button lets you see more detail about the failure including the closest match found for both the appearance of the original screenshot and the text content (if text matching was enabled).
Then, if there’s a real bug, anyone can automatically send a ticket to Jira for the development team to fix the bug. The ticket will include the failed test step, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.
3 More Rainforest Features That Make It Easy to Achieve Better Test Coverage with Less Time and Effort
1. Test Actions Outside the Browser
Selenium automates web browsers, which means each test is limited to testing within a single browser tab. For many teams, this means they won’t be able to create test cases for some of the critical functions of their app. For example:
- Downloading a desktop application: If your product is an application that needs to be downloaded, you’ll need a test that can interact with the operating system and homescreen of the device in order to test whether or not customers will be able to successfully download, install, and open the app. Note: Teams using Selenium would have to use another tool, like UFT (formerly QTP) to complete these test cases.
- Installing a browser extension: If your product relies on users installing a browser extension, you’ll need a test that can interact with browser settings.
- Testing interactions between two users: If your product includes any kind of messaging app, there are many scenarios where you may need to open two browsers within one test and actually test the sending and receiving of messages between those two users.
Rather than just automating browsers, Rainforest QA automates entire virtual machines running operating systems including Windows, MacOS, iOS, and Android. Rainforest tests can interact with anything you see on a screen, which means you’ll be able to perform any of the operations described above. Here’s an example video of how to create a test that saves a file to the desktop from one location then uploads the file to Google Drive.
2. Manage Manual and Automated Testing from the Same Platform
Automation is the fastest, most effective way to run software testing in most situations. But there are some tests that are better suited for manual testing. A test is better suited for manual testing if any of the steps require subjective interpretation or are subject to frequent change (as is the case with many test cases that cover brand-new features).
Most teams using Rainforest QA find that automated testing can be handled so efficiently that they have no problem handling a few manual tests in-house. But, Rainforest QA also offers the fastest manual testing available via our community of QA experts.
Any test written in the Rainforest visual editor can also be sent to the Rainforest community for manual testing, where results will be returned in 17 minutes on average.
3. Easily Scale Up Testing without Adding Other Tools
While the Selenium automation tool is open-source, it only provides a way for you to write and run tests on your own devices. If you want to run multiple tests simultaneously, manage a large number of tests, or simplify communication with the QA team, you’ll eventually need to add additional (paid) tools and services.
Rainforest QA is equipped with everything you’ll need to run and manage a full suite of automated and manual tests, including:
- Easy test suite management.
- A network of virtual machines to support parallel testing (eliminating the need for services like Selenium Grid, Sauce Labs, or BrowserStack).
- Access to multiple operating systems and browsers, including current and older versions of Safari, Chrome, Firefox, Internet Explorer, etc.
- Built-in test data like randomized email addresses, credit card numbers, names, etc.
- An API to integrate your testing process into any software stack.
- Easy integration into CI/CD pipelines with our CLI, GitHub Action, or CircleCI Orb.
- Integrations with Slack, Microsoft Teams, and email, so your team can get real-time notifications of test failures and other updates.
- Unlimited team members.
Talk to us about setting up a Rainforest plan that fits your testing needs.
