If you and your team are ready to transition from the slog of manual testing to faster automated tests, codeless test automation tools might hold a lot of appeal.
Chances are, you don’t have and don’t want to hire (expensive) QA engineers to wrangle complex, open source testing solutions like Selenium. You want your front-end developers focused on shipping code, not getting mired in test suite maintenance in Cypress.
So you’re hoping the relative simplicity and flexibility of a codeless test automation tool will help your software development team — including non-technical members — quickly ramp up automated test coverage. Since there’s not any code involved, it should be pretty straightforward for your testers to figure out, right?
Unless you choose the right tool, it won’t be as easy as you think. Choosing the wrong tool could be a drag on your release process.
In this piece, I’m going to explain three big frustrations startup teams have when they adopt (most) codeless test automation tools. I’ll use specific examples from some of the most popular codeless test automation tools, including:
- Rainforest QA
- Squish
- Ghost Inspector
- mabl
- Katalon Studio
- TestComplete
- TestRigor
- Ranorex Studio
- Testim
- Perfecto
We’ve designed our no-code test automation platform, Rainforest QA, to address the challenges software teams encounter in other codeless tools. Talk to us about setting up an account.
“No code” often means “some code”
If you’ve already hired a technical QA team, a complex, full-code test automation solution like Selenium can make sense when you’re ready to automate your regression test suite. But for many other teams, no-code is the only workable path to test automation, since it presumably allows more of your existing headcount — technical or not — to contribute to test coverage.
The problem is: with many “codeless” test automation tools, you can avoid code when you’re creating a test, but when you inevitably have to update your tests, you’ll have to interact with code.
To be clear, I’m not talking about having the option to use code like JavaScript to customize test scripts, as some low-code tools offer. This feature can be pretty appealing to some teams.
I’m talking about not having the choice to avoid code when you’re performing one of the most frequent workflows of test automation: maintaining your tests. It feels like a half-truth when these tools present themselves as “codeless.”
In the best-case scenario, dealing with bits of code in test steps instead of plain English will slow you down, regardless of whether you’ve got coding skills or not.
In the worst cases, some tools completely exclude non-technical people from a critical part of the testing workflow.
Take record-and-playback tools, for example. They can record your steps as you interact with your app and automatically transform those steps into a test script. It’s generally a no-code way to create tests.
The first problem with some of these tools — like Selenium IDE — is that you can only use them to create very basic tests. You’ll need coding skills to create test steps involving even minor complexity.
A number of the record-and-playback tools, however, allow you to record tests for moderately-complex test scenarios without using any code. But what happens when you need to update a test?
Squish
Squish represents the worst case: its recording functionality only creates test scripts in one of its supported programming languages (JavaScript, Python, Perl, Ruby, and Tcl). Only a tester with the right programming knowledge can edit — or even understand — a test in Squish once it’s been created.
Ghost Inspector
Ghost Inspector is somewhat better than Squish in that the actions in each step (e.g., click, keypress, assign) are in plain English. But the app elements that correspond to those actions are only described by CSS classes or Xpath.
Imagine having to update this test. Hopefully it’s easy to interpret most of your CSS classes — otherwise you’ll have to constantly dig around in the front-end code (or “DOM”) of your app to understand what app element each CSS class or Xpath represents. Working with plain-English test steps is going to be faster and require less mental overhead.
mabl
Cross-browser tests recorded with mabl are indeed (mostly) in plain English. But the mabl test recorder (“mabl trainer”) will generally ask for additional user input to confirm the best way to identify app elements you’ve interacted with.
Unless you’re familiar with the CSS of your app and know the CSS will remain stable, you probably won’t know how to respond with any confidence. And without being able to correctly identify elements in your app, mabl tests will be unreliable.
Regardless, this is yet another not-so-codeless step that’ll slow down the testing workflows in your software development lifecycle (SDLC).
What to look for
When you’re considering the adoption of a codeless automated testing tool, check to see if it’s truly codeless, or if it simply offers a way to create tests without (much) code.
If you have to interact with or use code when creating or updating tests, it’s going to prevent non-technical members of the team from contributing to test maintenance, and/or it’s going to slow down the test maintenance process, even for team members with coding skills. Either way, it’s going to bottleneck your processes.
Rainforest QA requires zero code to create, update, and execute tests, and to analyze test results. In fact, you’ll never see any code when you’re performing these workflows (though you can integrate Rainforest tests into your release workflow with our API or CI integrations).
When you visit the Rainforest visual editor, Rainforest previews your app on a virtual machine. To create a test step, select an action (e.g., “Click”, “Fill”, “Scroll”) and then drag-and-drop a box to select the element to apply the action to.
Anyone on your team can help maintain your automated test coverage, and no one has to dig around to match CSS classes or Xpaths to your app elements — which can slow down even your technical team members.
Talk to us about setting up a Rainforest account and see how you can kick off your first automated test in five minutes or less.
Most codeless tools are complex and difficult to learn
Despite its connotations, “no code” doesn’t mean “easy to use.”
This concept explains the biggest challenge teams encounter when they test-drive these tools:
Most codeless test automation tools are complex (some more than others) and require a considerable learning curve to use with any success.
These tools have complicated workflows and user interfaces (UIs) — you can’t just jump in and expect to start creating automated tests just because you don’t need to know how to code. You’ll need time to learn and absorb the intricacies of whatever tool you’re using. (And that’s not even considering the additional drag on your productivity when you have to deal with code as part of test maintenance.)
Katalon Studio
Take Katalon, for example. To create the most basic of tests (e.g., clicking buttons and entering text into fields) without using any code, you can use its recorder. But for everything else, you’ll need to build tests step-by-step in its test editor.
There are over one hundred different types of actions (or “keywords”) you can add to a test step in Katalon Studio’s test editor. Some of them have meanings you can understand intuitively, like Click, Delete All Cookies, and Mouse Over.
But what about Switch to Window Index, Wait for Angular Load, and Switch to Default Content? What do they do, and when would you want to use them in a test?
Having a lot of options is great for expert power users, but otherwise it just means it’s going to take a lot of learning and practice to effectively write (and to understand) more than just basic tests.
For instance, for a workflow as common and basic as testing login functionality, how would a new Katalon user know that you don’t use the Set Text action to enter text into a password field, but rather the Set Encrypted Text action?
There are just so many nuances and minutia to learn, and that’s not even considering the tool’s complicated workflows, settings, and UI.
TestComplete
But to be fair, Katalon isn’t even the most complicated no-code testing tool out there. That distinction probably goes to TestComplete.
While recorded test steps don’t include conventional code, TestComplete’s puzzling naming conventions for test actions and other features make tests look like they might as well have been written in code.
The developers of TestComplete likely know how difficult it is to read a test in their tool: they specifically include a “Description” field for every step, so a test author can explain in plain language what’s actually happening at each stage of the test.
TestComplete feels like it was designed to scare away anyone but people dedicated to spending their working hours becoming experts on this tool. Even important functionality — like verifying the visual appearance of an element in your application — requires 15+ steps within the configuration windows of an unintuitive UI.
TestRigor
Of course, TestComplete is a bit of an extreme example.
Consider TestRigor, a truly codeless software testing app: anyone can write, read, and update automated tests using natural language in plain English. It’s even got a recorder for creating basic tests. Seems pretty straightforward, right?
Here’s the catch: TestRigor’s language model isn’t like ChatGPT. You can’t just write test steps using whatever words and expressions you want and assume TestRigor will be able to understand them.
TestRigor supports a list of just over 40 test actions (e.g., click, type, scroll). Each of those actions will only work correctly if you use them in the correct syntax.
For example, here’s the specific syntax you’d use to create a test step that clicks on a ‘delete’ link within a table:
click on "Delete" within the context of table "actions" at row containing "id1" and column "Actions"
It’s pretty easy to understand when you read it, but TestRigor’s way of describing the location of an element in your app might not align with how you’d describe the location of that element.
So, TestRigor is much like other record-and-playback tools in that you need to learn the functionalities of its many test actions to create anything but basic tests. Plus, you need to learn and practice speaking TestRigor’s version of plain English.
What to look for
If you want to adopt a test automation tool that’ll help your team move quickly while maintaining product quality, no-code is necessary but not sufficient. No-code allows anyone on the team to contribute to test creation and (ideally) test maintenance, so you don’t have to hire expensive QA engineers or put all the onus of software testing on your developers (who are just trying to ship code).
Your codeless test automation solution needs to be intuitive and its automated test cases need to be in plain English so anyone on the team can quickly interpret and update test steps when necessary.
Your team shouldn’t have to learn or memorize complex workflows, terminology, or other minutiae to easily understand and update test cases. Otherwise, automated test creation and maintenance will become painful bottlenecks in your release process.
That’s why we designed Rainforest QA so anyone can get started right away and execute quickly without any special training.
It’s intuitive, with a modern, user-friendly interface and straightforward workflows.
All test steps in Rainforest are in plain English, so anyone can immediately interpret them.
Unlike other tools with an overwhelming number of options, there are only ~15 test actions (e.g., “Click”, “Fill”, “Scroll”) to choose from. Rainforest doesn’t need as many built-in test actions as other tools because its actions are more flexible.
Plus, all the actions in Rainforest are easy to understand because they represent real-word user actions, not esoteric testing-tool functionality.
To try it out, talk to us about setting up a Rainforest account.
Most codeless tools focus on functional testing, not visual testing
Most codeless automation testing tools focus on testing the functionality of an app while disregarding its visual appearance. These tools don’t, by default, test what a user sees on the screen — they interact with and evaluate what happens in the DOM, the code layer behind what appears on the screen. This puts the user experience at risk.
For example, a DOM-based test looking for the presence of a particular button would pass the test if the button’s locator (e.g., its element ID, CSS class, or Xpath) appeared in the DOM. But the tester wouldn’t know anything about the appearance of the button, including its position, size, color, shape, or content. Even if the button was obscured by a popup or accidentally colored to match the background of the page, the test would still pass.
Ranorex Studio
Some of these tools can perform automated UI testing — evaluation that includes both functional testing and visual testing — but they don’t do it without extra, time-consuming work. For each visual test of a screen or element (like a button or headline) you want a test to perform, you, as the user, have to follow an often-complex workflow.
On top of that, the image-matching algorithms in tools like TestComplete and Ranorex Studio are unsophisticated: a test will fail every time the visual appearance of a tested element doesn’t exactly match its baseline image. Even minor tweaks to your app could cause test failures you then have to investigate and resolve.
Other tools don’t even include the built-in ability to validate the appearance of your app. They require you to pay for and integrate a third-party plugin like Applitools that specializes in adding visual-validation capabilities to other testing tools.
Testim
Take Testim, for example. It generally avoids the other two issues we’ve already discussed in this post: it’s relatively easy to use and you can (mostly) avoid interacting with code. But Testim isn’t capable of testing the appearance of your app without an integration with Applitools, and that option is only available on the plan that starts at $1,000/month.
Perfecto
Perfecto offers a “scriptless” option to automate tests, but if you want to validate the visual appearance of your web or mobile app, you need to use code like Java or C#. There’s no codeless way to add visual testing to your tests in Perfecto.
What to look for
You app can be functional without being usable. That’s why it’s important to adopt a testing tool that performs both functional testing and visual testing testing, by default. You shouldn’t have to jump through hoops to do both.
Further, the algorithms in your testing tool that evaluate the visuals of your app should account for the fact that your app’s appearance will experience (at least) minor evolutions over time. You don’t want to have to debug and update failed tests every time there are changes that a human tester wouldn’t notice or care about. Avoid “exact match” visual testing algorithms and look for ones that will pass minor and expected changes using AI or other intelligent methods.
Rainforest’s unique approach combines functional and visual testing in every test step, by default.
Rainforest QA’s proprietary automation locates elements in your app using visual appearance, not the DOM — just like a real user would. Rainforest also interacts with these elements via the visual layer of the UI, just like a real user would.
Because of this approach, unlike other tools, Rainforest can locate and interact with anything that appears on a screen — inside or outside the browser. You can test browser extension settings, download and open files, and do anything else a real user could do on a Windows machine.
For example, here’s a recording of a Rainforest test that downloads and installs Brave browser, including interactions with Windows 10 and its file system:
Plus, Rainforest avoids the test-brittleness issues you find in tools that identify app elements using DOM-based locators. When you use these other testing tools, changing the name of an element’s locator in your code could lead to failed tests even if the functionality and appearance of your app hasn’t changed. (Some tools offer “self-healing” for broken locators, but the effectiveness of these features is hit-or-miss depending on the self-healing method.)
Rainforest tests fail when they’re supposed to: when functionality doesn’t work or when UI elements don’t appear as expected. (Using machine learning algorithms, Rainforest can ignore any minor visual changes in your app that a human wouldn’t notice or would otherwise ignore, so you can avoid false-positive test failures.) Unlike other automated tests, they don’t fail just because the DOM-based locator (e.g., CSS class) of an element has changed.
Talk to us about setting up a Rainforest plan that fits your needs.
Make the right trade-offs for your team
Unless you’re already paying for QA engineers who spend all their time dealing with the intricacies of most of these tools, you’re going to struggle to execute with them. Test creation and maintenance will, to some degree, be painful bottlenecks in your release process and agile devops workflows.
We’ve specifically designed Rainforest to address the challenges that are otherwise common to complex, not-really-codeless test automation tools. Rainforest’s test-creation and -maintenance workflows are truly zero-code, it’s intuitive, and it presents tests in plain English. Anyone on your team can get started quickly and contribute to your automated test coverage with no special training.
Plus, Rainforest is the only no-code test automation tool that evaluates both the functionality and appearance of your app to protect the user experience and give you confidence when you ship.
Rainforest is an all-in-one, no-code test automation platform, so you don’t need to provision any additional products or services to get everything you need to create and run an automated test suite:
- A proprietary no-code automation framework for creating and maintaining end-to-end tests.
- Cloud-based virtual machines (VMs) — designed for scalability — running browsers on Windows 10 for test execution. (Our Unlimited plan includes VMs with other versions of popular operating systems and browsers.)
- Test reports featuring video recordings, plain-English test steps, HTTP logs, and browser logs for debugging.
- Integrations with email, Slack, and MS Teams for notifications and with JIRA for automatically generating tickets for bugs.
- Test scheduling plus an API, CLI, and CI integrations for continuous testing in your CI/CD pipeline.
Of course, no one testing solution is going to be perfect for everyone. For example, Rainforest is optimized for end-to-end, UI testing of web applications — it doesn’t automate mobile testing for native apps, so if testing native apps for iOS or Android mobile devices is your use case, you might want to check out Testim, Katalon. For automated API testing, mabl and Katalon both offer solutions.
Talk to us about setting up a Rainforest plan that fits your needs.
