15+ automated testing tools for web applications in 2023

If you’re interested in automating the testing of your web application, there are three categories of tools to understand:

• Open-source frameworks like Selenium and Cypress
• Low-code tools
• No-code tools

The best automated testing solution for your web application will depend largely on the resources you have and the tradeoffs you’re willing to make.

In this piece, I’m going to help you understand the strengths and shortcomings that come with each of these types of testing tools, as well as who they’re each best-suited for.

Open source frameworks for automated testing of web apps

Before we get into the benefits and challenges of these tools, here’s a handful of the most popular open source frameworks for end-to-end automated testing of web apps: 

Selenium

Selenium supports all major browsers, runs on all major operating systems (Windows, MacOS, and Linux), and supports test creation via any one of a number of programming languages including JavaScript, Groovy, Java, C#, PHP, Python, Perl, and Ruby. It’s quite complex to configure and use relative to other frameworks, so it’s good for teams who can QA engineers and don’t necessarily need to move fast.

Cypress

Cypress is popular among some front-end developers since its tests are written in JavaScript, its tests execute very fast (especially relative to Selenium), and there’s built-in support for other compelling features like stubbing APIs and simulating network conditions. 

But it comes with notable limitations. For example, it only works with Firefox and Chromium browsers (no compatibility with Safari or Internet Explorer) and tests can’t run across multiple browsers or tabs. You can’t execute tests in parallel without paying for Cypress Cloud, the premium version of Cypress.

Appium

Appium is derived from Selenium Webdriver (the core component of Selenium) and specializes in native mobile app and web app tests for iOS and Android devices. Similar to Selenium, it supports multiple programming languages.

Playwright

Playwright is a Microsoft-backed project that’s quickly grown in popularity since its release in 2020. It can do several things Cypress can’t, including do cross-browser test execution across all major browsers, perform parallel testing without a paid upgrade, and support multiple languages (JavaScript, Java, Python, and .NET).

The benefits of open source automated testing frameworks

Power and flexibility

If you’re a developer or QA engineer who’s familiar with an open source framework and one of its supported scripting languages, you can create automated scripts for any number of test cases, including complex ones. 

First, the inherent versatility of code makes code-based test scripting flexible and powerful.

Plus, many third-party developers have created plugins (open source and otherwise) to extend the functionality of these frameworks.

In addition to end-to-end testing, you can also use frameworks like Selenium and Cypress for unit testing.

Free support 

The frameworks I’ve mentioned have large and active communities where you can get troubleshooting help at no cost, but of course the consistency and reliability of responses will vary.

The challenges of open source testing frameworks 

Technical barriers

What all of the open source software testing frameworks have in common is that they require coding skills (and knowledge of a testing framework) to use. Only personnel who can interpret code and understand the way the tests are structured can write or update tests.

These personnel are usually either QA engineers or front-end developers who split their time between shipping code and maintaining the test suite. So if you can’t afford to hire additional specialized QA headcount or to take developer time away from shipping code, these tools aren’t going to be a good fit for you.

Limited cost-effectiveness

Technically, open source frameworks are free to use, as long as your configuration doesn’t require premium add-ons. 

For example, Selenium is basically just a framework for automating browsers. If you want to get insightful test results, test management tools, visual regression testing, and other “table stakes” testing features, you’ll need to integrate third party plugins, some of which will require payment. If you want to run tests in parallel, you’ll either need to provision (real or virtual) machines, or pay for a testing grid from BrowserStack, Sauce Labs, or similar.

Plus, these tools require expensive technical headcount to operate them: 

Expensive, specialized headcount

QA engineers and developers are expensive. In fact, the expense of these roles is usually the biggest financial cost of working with an open source framework when you’re testing a web app.

Even if you don’t hire QA engineers and instead use your existing developers to manage your test suite, there’s the opportunity cost of having them create and maintain end-to-end tests instead of shipping code. 

Trading between bottlenecks and product quality risks

Working with test scripts written in code is a double-edged sword: they’re versatile, but they take longer to create and update than test scripts created with low or no code. 

Every time a purposeful or unintended change in your web app causes your automated tests to fail, someone has to dig around in your Selenium code (for example) to find the affected strings. Then, that someone has to update the code in the affected tests based on the expected behavior of the app, which they may or may not be familiar with. And this all assumes anyone with the right expertise is available when you need them.

The usual result is either (1) code doesn’t get shipped as the software development team waits for tests to get updated to a passing state, or (2) the team gets tired of being blocked and ships code without updating the automated test suite. 

The test suite becomes less reliable, making the software development team even less inclined to invest in its maintenance, creating a vicious cycle where the automated tests ultimately fail to do their job in protecting the quality of the product.

No visual regression testing, by default

Open source automated testing frameworks can miss bugs that make your web app unusable.

That’s because, by default, these tools focus on testing the functionality of your web app while ignoring its visual layout and appearance. So, for example, a test meant to validate the presence and functionality of a button could pass even if the button was obscured by a popup or accidentally colored to blend in with the background.

The only way to add automated visual regression testing to these tools is via a 3rd-party (paid) plugin like Applitools.

Who are open source testing tools for?

Open source tools are a good fit for teams who can afford the expensive technical headcount required to manage these complex tools and their coded test suites. You’ll need to hire dedicated QA engineers and/or to take the time of your front-end developers away from shipping code.

These tools are also a good fit for large, complex web apps that require the versatility of automated test scripts written in code.

Between the expense of headcount and the difficulty of creating and maintaining tests quickly, they’re typically not a good fit for resource-constrained startups that prioritize shipping fast. 

Low-code tools for automated testing of web apps

Most low-code test automation tools are “record and playback” tools: they record your actions as you interact with your web app and then automatically transform those actions into automated test steps. It’s an easy way for anyone to quickly create tests without using any code.

The majority of record-and-playback tools can record basic tests with no coding, but require coding skills to create complex test steps. For instance:

• Selenium IDE 
• TestComplete
• Testim
• mabl
• Katalon Studio
• Rapise
• Ghost Inspector
• Telerik Test Studio

Some record-and-playback tools can create more-complex tests than others using just the recording feature. But, generally speaking, you’ll need technical skills to create test steps that cover software testing scenarios like validating the contents of a download or confirming an email was received.

Most of these tools support JavaScript for creating test steps, but there are exceptions. Katalon supports Groovy and JAVA, for example, and TestComplete supports JavaScript, Python, VBScript, Jscript, DelphiScript, C#, and C+.

Once you’ve recorded your tests, these tools differ in terms of how much you have to use code to update and maintain your automated tests.

For example, at least one tool, Squish, requires coding skills to interpret and update any recorded tests. Once you record test actions with Squish, it creates a test in one of its supported scripting languages: Python, Perl, Ruby, JavaScript, or Tcl.

With other recording tools, you could conceivably avoid using coding skills to create and maintain your test suite. But there’s a catch that no one really talks about:

The majority of record-and-playback tools force you to interact with code, even if you don’t need to use coding skills, per se. This slows down the test automation process, whether you’ve got technical skills, or not.

Most people would identify a web application element like a button or headline by its visual appearance (“the big, green login button”) or content (“the headline that says, ‘generate test scripts without code’”). 

But these testing tools identify and interact with elements in your web app via behind-the-scenes “locators” in your web app’s front-end code (the “DOM”). These locators could include CSS classes, Xpaths, or custom element IDs, for instance.

You’ll typically encounter locators once a test is recorded and you want to review and/or edit the recorded test steps.

In some cases, you’ll even have to translate locator code when you’re recording a test.

Even record-and-playback tools that don’t allow any coded test scripting at all — like Screenster and DataDog Continuous Testing — require you to interact with locators.

The benefits of low-code test automation

Faster test creation

Coding automated tests by hand in an open source framework can be slow and/or difficult. Low-code tools make it easier to create automated tests more quickly with a recorder, while still offering the versatility of coding to cover more complex test cases. (In fact, tools like Katalon are basically a front-end for open source frameworks like Selenium and Appium.)

More than just web application testing

Some low-code test automation tools have been built to offer more than just end-to-end, functional testing of web applications. 

• TestComplete and Katalon can both also test Windows desktop applications, and native mobile applications. 
• mabl offers performance testing.
• Katalon, mabl, and Rapise do API testing.
• TestComplete supports behavior-driven development (BDD) scenarios via Gherkin syntax. (Other tools are compatible with Cucumber.) 

The challenges of low-code test automation

Locators slow things down

Having to dig around in the DOM to figure out which locator belongs to which element in your web app adds (pretty annoying) overhead to test creation and maintenance.

Unnecessarily complex

Many of these tools have complicated user interfaces and workflows that require training and familiarity to use with any success. Their tests are written with proprietary commands and vernacular that feel like they might as well be written in code. (Testim is probably the best of the bunch in terms of ease of use.)

Companies often have just one or two people on the team who have developed the skills to use these tools. Having a limited number of specialists can create bottlenecks in the release process when demands for test creation and maintenance are high.

No visual regression testing, by default

Like open source tools, low-code testing tools focus on functional testing, not visual regression testing. 

Most of these tools (except for Katalon) feature unsophisticated, exact-match algorithms that compare baseline screenshots of your app to screenshots captured during tests. Unless the screenshots match pixel-for-pixel, the tool will throw an alert or fail the relevant test. So even minor and expected updates to your app could cause disruptive, false-positive test failures that need to be debugged. 

Other tools like Testim require that you integrate a third-party tool like Applitools for any sort of visual testing.

Regardless, you’ll need to configure additional steps for every verification of a screen or app element you want to perform in the testing process.

Who are low-code testing tools for?

Low-code tools are for teams who want to move more quickly than they can with open source tools, but who have the technical headcount available to maintain tests and/or create test coverage for complex cases. 

Since low-code testing tools are quite complex (despite requiring less coding skill than an open source tool), they typically require specialized headcount, who can bottleneck the release process. So they’re not a good fit for resource-constrained startups, or for teams interested in sharing responsibility for managing the test suite across different roles. 

No-code tools for automated testing of web apps

There are very few automated testing tools that can truly claim to be no-code. Most testing tools — even some that claim to be “codeless” — require interacting with code to one extent or another.

For example, Virtuoso highlights that you can write its tests in natural language. But you still have to deal with locators when maintaining its tests, which isn’t very “natural.” 

On the other hand, TestRigor is a genuine no-code, natural language solution. You can write tests in plain English. 

But it’s not like ChatGPT — you can’t just write whatever you want and expect TestRigor’s automation to understand. You need to learn its 40+ different test actions (click, type, scroll) and how to use them in correctly-formatted expressions that TestRigor can interpret.

For example, if you want a test to click on an element anywhere other than the center of the element (the default), then you need to specify an offset, which isn’t intuitive:

Click on “Delete” with offset “20,10”

As you can see, “no-code” doesn’t necessarily mean “easy to use.”

Plus, you’ll never have to look at — let alone decipher — locator code when you’re using Rainforest. You can easily interpret and update test steps because they’re all in plain English. 

Rainforest doesn’t have to use locators because it identifies elements in your web app not by their underlying code, but by their visual appearance, just like a real user would. And Rainforest interacts with those buttons, links, form fields and other elements via the visual layer of your application, just like a real end user would.

That means, unlike the vast majority of other automated testing tools, Rainforest does both functional testing and visual testing by default in every test. (Intelligent, AI-powered image-matching algorithms ignore minor visual changes in your app that a human tester wouldn’t notice or care about, so you’re not constantly debugging false-positive test failures.)

This UI testing approach means Rainforest is uniquely able to test anything that appears on-screen — not just within the browser window. So your Rainforest tests can, for example: download and verify the contents of files, edit browser extension settings, and do just about anything else a real user could on a Windows 10 machine. 

For example, here’s a recording of a test that downloads and then installs Brave Browser:

Finally, unlike open source tools and many low-code tools, Rainforest includes everything you need to get started with and scale automated testing of web applications. There’s no need to integrate any plugins or pay for any 3rd-party services. It includes:

• A proprietary no-code automation framework for creating and maintaining end-to-end tests.
• Cloud-based Windows 10 virtual machines (VMs) running Chrome browsers. (Our Unlimited plan includes multiple operating systems and browsers.)
• Test results that include everything you need for test debugging: video recordings, plain-English test steps, HTTP logs, and browser logs.
• Integrations with email, Slack, and MS Teams for notifications and with JIRA so you can auto-generate tickets for bugs.
• Test scheduling plus an API, CLI, and CI integrations for continuous testing.

The benefits of Rainforest’s no-code test automation 

Rainforest is more intuitive than other test automation solutions, so it’s easy for anyone to get started automating a test suite right away without any training, regardless of their coding skills. 

And you won’t get slowed down dealing with any element locators — all tests are written in plain English so they’re easy to interpret and maintain.

Since Rainforest uniquely does both functional and visual testing by default, you’ll have the confidence that your automated UI tests are protecting the end-user experience of your web app, not just its behind-the-scenes functionality. It tests what your end users will see, not what a computer sees. 

Rainforest is an all-in-one testing platform, so you don’t need to worry about provisioning any additional machines or services to get everything you need to create and run an automated test suite.

The challenges of no-code test automation

No-code test automation tools like Rainforest are generally flexible enough to cover the majority of test cases. However, in some cases, a technical person might prefer the versatility of code available via an open source framework to address edge cases or particularly complex test cases.

While some low-code software testing tools can also test mobile applications and Windows desktop applications, Rainforest has been optimized specifically for testing web applications. So if you’re interested in a solution for testing mobile apps and not just a web testing tool, Rainforest might not be a good fit for you.

Who is Rainforest for?

Rainforest is ideal for startup teams ready to transition away from manual testing. They want to move faster, but don’t want to (or can’t) hire expensive QA headcount to manage complex testing tools. 

Rainforest is also perfect for teams who don’t want to put the full burden of managing end-to-end test automation on developers, because the devs need to stay focused on shipping code. These teams want anyone to be able to contribute to test maintenance without training or hand-holding so web app updates can keep getting shipped, fast.

Talk to us about setting up a Rainforest account.