An automated testing strategy answers the who, what, when, why, and how of software test automation.
While automation can be used for many kinds of software testing, in this post, we’re going to focus on the steps you should take when developing a strategy for functional UI testing (also known as end-to-end testing).
End-to-end testing simulates real user conditions to test the functionality of an application from the front-end user interface.
Whether you’re just getting started with test automation or you already have a few automated scripts in your testing suite, defining your test automation strategy will help you improve test coverage and get faster, more consistent test results.
In this post, we’ll cover why each of the following steps is important and how to implement each one:
- Define Your Goals: Find a balance between product quality and release speed.
- Assign Testing Responsibilities: Ensure testing gets done and stays up-to-date.
- Decide What Tests to Automate: Plan out which features you’ll test and with what methodology.
- Decide When to Do Functional Testing: Ensure it’s happening often enough during the development lifecycle to meet your goals.
- Choose a Test Automation Tool: Understand which tool best fits your needs.
Once you’ve created your automation test strategy, talk to us about quickly and easily creating an automated test suite with no code using Rainforest.
#1. Define Your Goals
Quality assurance is always a balancing act between release speed and test coverage. It’s impossible to catch every single bug during the testing process, and trying to do so would grind your release process to a halt. It’s better to think of QA as an exercise in risk mitigation: How can you reduce the risk of releasing software with major bugs that could impact your bottom line or reputation?
Each company has a different tolerance for bugs based on a variety of factors, including the industry, the expectations of the users, and the competitiveness of their market segment.
In some industries, minor bugs might be so damaging to a company’s reputation that it’s worth delaying releases by several weeks in order to thoroughly test everything. In other settings, it’s much more important to get a requested feature into customers’ hands by a certain deadline, even if it means the feature comes out with a few minor bugs.
Your company’s goals for quality and speed of release will impact your automated testing strategy in a variety of ways because they will determine how much time, money, and human resources you’re willing to invest into testing up-front.
Answering the following questions will help you determine the right balance between quality and speed for your organization:
- How many bugs can your customers tolerate before losing patience with your product?
- How much more does it cost to fix bugs after release versus finding them before release?
- What kinds of bugs are completely unacceptable to your customers, and which ones are just a nuisance?
- How fast do you need to be able to release software updates to meet customer demands?
#2. Assign Testing Responsibilities
Automated testing isn’t a set-it-and-forget-it exercise. Successful test automation requires an ongoing commitment to writing test scripts, running and interpreting the results, and maintaining the test suite.
If no one is explicitly in charge of each of the duties related to automated testing, the test suite can easily become stale and unusable.
To make sure your test suite stays up-to-date, all of the responsibilities related to test automation need to be clearly assigned.
Typically, development teams will set up and run automated unit tests and integration tests as part of their software development processes. But for functional testing, there are a variety of ways to organize responsibilities, especially if you’re using an accessible automated testing tool like Rainforest QA that doesn’t require programming skills to use.
In that case, you can have a mix of responsibilities. Perhaps the developers write the functional tests after they finish writing a feature and run them as part of their release process, but a QA testing team interprets the results and takes care of ongoing test maintenance. Or perhaps the QA specialist writes and maintains the tests, but the developer is the one to actually run them and interpret the results.
In most cases, having developers stay more involved in testing will help testing move faster. But since every minute a developer spends on testing is a minute not spent building new features or fixing bugs, many teams try to shift a lot of the duties to QA specialists.
Regardless of your testing approach, testing responsibilities need to be clearly defined and understood by everyone involved so your automation suite stays up-to-date.
#3. Decide What Tests to Automate
For most web applications, it’s not practical or feasible to test every single user path through an app. And not all user paths make sense to test with automation. Some types of tests are always better candidates for manual testing.
To figure out which features to test, you first need to create a map of all of your app’s user paths, ranked by priority. This map will also help you create a test plan after you’ve worked out your strategy.
Here is a sample functionality map for an early version of the Slack web app:
In general, you should prioritize your testing efforts around the most important user paths to make sure they are working properly. We like to use the analogy of a snow plow clearing a city’s streets after a snowstorm. The streets that see the most traffic get cleared first, and some of the side streets may never get cleared because so few people travel on them.
Once you know what user paths to test, you need to decide which tests are best suited for automated vs. manual testing. We’ll share a few general rules here, but we go into detail about Manual vs. Automated Testing in this article.
Test cases are ready for automation when they don’t change very often and get run frequently, such as smoke tests or regression tests.
Smoke tests: Tests that run through the most essential user paths to make sure the application is stable enough to move on to further testing.
Regression tests: Tests that check the original features of the application to make sure that the latest changes or updates haven’t broken anything.
Use manual testers for tests you only run once (such as exploratory tests), tests where the software — and therefore the data set — changes between each run, or tests that require human interpretation (e.g. if the feature includes a CAPTCHA).
#4. Decide When to Do Functional Testing
When you’re developing your automation process, you need to decide how often each kind of test should be run and what kind of changes merit running your full test suite.
The ideal solution is to run your entire functional test suite after every product change, no matter how small. To do this, you need to have a highly optimized automated test suite. Using a tool like Rainforest QA to create and run your tests can make this feasible by making testing fast and affordable.
Even though it might seem like overkill for small changes, running your full test suite after every change has two major benefits:
- It gives an incentive to everyone on your team to keep the test suite running as efficiently as possible.
- It’s the simplest option, with no possibility of confusion. No one has to remember which kinds of changes go with which set of tests.
However, this may not be feasible for your organization. In that case, you may need to bundle changes together before running your functional test suite, or even divide your test suite into groups of tests that need to be run for different kinds of changes.
(In the Rainforest platform, you can create Run Groups containing different sets of tests so you can easily run subsets of your entire test suite.)
Keep it as simple as you can — if you’re not careful, you can end up with a complicated list of conditional criteria for different kinds of changes. If a change falls in a gray area where it’s unclear if it meets the criteria for running the full test suite or not, we recommend running the full suite.
#5. Choose a Test Automation Tool
There are many different automated testing tools available, including open source tools and no-code tools. Choosing the right tool will impact:
- Who can create and maintain tests.
- How quickly new features can be released.
- What functionalities you’re able to test.
- How much it costs to run your functional test suite.
Whether you’re using an open-source testing framework like Selenium or a tool that generates code for you, all test automation frameworks (except Rainforest QA) test the underlying code of an application. Testing the underlying code means you’ll eventually run into one of the following problems:
- You’ll need a (high-salaried) software engineer to create tests, maintain tests or understand test failures.
- You’ll experience a bottleneck effect in your release pipeline because maintaining tests is time-consuming.
- You won’t be able to test actions outside of the browser or test the actual user interface.
Rainforest QA is the only automated software testing tool for web apps that doesn’t test the underlying code. Instead, Rainforest uses pixel-matching to simulate a real user interacting with the visual layer of the app. Anyone can create, maintain, and run automated tests with Rainforest’s no-code user-friendly interface.
Here are just a few of the additional benefits that come with Rainforest QA:
Test Automation That Doesn’t Break with Minor, Behind-the-Scenes Code Changes
Because Rainforest tests the visual layer of the application, its automated test scripts are less susceptible to breaking due to minor code changes that don’t affect the UI — unlike tools that test the underlying code.
Let’s say you you work at Notion and want to test whether your “Get Notion free” button is working after you’ve made some other changes to your web application. In Rainforest, you’d create a test that includes a step to click on the “Get Notion free” button. You’d tell the test where to click by click-and-dragging a selection box around the button.
When you run the test, Rainforest’s automation will look all over the page for pixels that match your image selection. If those pixels exist anywhere on the page, the test will find them and will be able to keep going, even if the underlying code changed.
In contrast, if you were using Selenium to test this button, you could run into a variety of situations where the test could fail even though the UI was flawless.
For example, if the CSS class of the button changed from “try for free button” to “tryforfreebtn2,” a Selenium test could break, but a Rainforest test would not as long as the pixels of the button on the page did not change.
A Rainforest test would fail only if the actual visual appearance of the button changed. For example, if the button was accidentally hidden due to a bug in a recent code push, the test would (and should) fail.
To be fair, Rainforest QA tests are not totally immune from “false positives” — or, test failures for which there’s no actual bug — in that a minor UI change, such as a button slightly changing shape, may break a test. But most Rainforest users find that test maintenance is less of a headache than with app testing tools like Selenium because:
- You can optionally use text matching to avoid test failures when you know the UI of a visual element is likely to change, and
- Anyone can quickly update tests in the visual editor without having to involve an engineer or write any code.
An All-in-One Solution with Usage-Based Pricing
Rainforest QA is an all-in-one UI test automation solution, which means it has built-in tools to allow users to create, run, and manage any number of automated tests all at once, meaning you don’t need to pay for things like a cloud-based testing grid, project management tools, and team collaboration tools just to manage your automation efforts.
All Rainforest QA plans include:
- A built-in “test grid” featuring more than 40 web browsers and operating systems running on virtual machines in our cloud, allowing you to test applications against a wide variety of platforms.
- Text execution in parallel. No matter how many test cases you run at a time, they’ll run simultaneously, getting you fast results.
Our Pay as You Go plan also includes human-powered add-on services:
- A test writing service. Provide a text or video description of the tests you’d like to run, and any data sets that go with them, and our QA professionals will create passing tests in Rainforest for you.
- A worldwide community of QA professionals available on-demand, 24×7 to run tests that can’t be automated, or that need subjective feedback.
With Rainforest QA, you can run as many tests as you want, in parallel. Rather than capping the number of concurrent tests you can run on a given pricing tier, we offer usage-based pricing. So you can scale testing up or down as needed, without having to make an expensive upgrade to any of your testing infrastructure tools.
Fix Broken Tests Fast with Video Playback
Rainforest automatically records a video of every test it runs (whether the test passes or fails). When a test fails, you can review the video to identify exactly where it failed. This can dramatically speed up the process of reviewing and classifying test failures because it makes it much easier to understand why a test failed.
Test Actions Outside of the Browser
Because Rainforest tests run on virtual machines (including Windows, MacOS, mobile devices), you can design tests that leave the browser and go into other apps.
For example, here’s a video walkthrough of creating a test that saves a file to the desktop from one location, then uploads the file to Google Drive. If you ever need to validate downloads or test file uploads, this is a really useful feature.
A Built-in Library of Data
Rainforest has a built-in library full of auto-generated test data, including random emails, passwords, addresses, virtual credit cards, and more. Plus, with virtual email inboxes, you can test emails.
Integrate into Your CI/CD Pipeline via Our API or CLI
Testing can quickly become a bottleneck unless it’s integrated into your CI/CD pipeline. Rainforest QA offers plugins to integrate with various tools, including CircleCI, so that developers can kick off a suite of Rainforest tests at the same time as their unit tests and integration tests.
You can also download our CLI and run Rainforest tests in any development environment. For custom workflows and more complex data analysis, our open source API makes it possible to integrate Rainforest into any software stack. With a fully-automated CI/CD pipeline, it’s much easier to achieve your team’s test coverage goals without slowing down development in an agile environment.
Put Your Automated Testing Strategy to Use with Rainforest QA
Rainforest is a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with an automated testing strategy or QA-mature teams regularly running 500+ quality software tests.
Talk to us about setting up a Rainforest plan that fits your needs.
