An automated testing strategy answers the who, what, when, why, and how of software test automation.
While automation can be used for many kinds of software testing, in this post, we’re going to focus on the steps you should take when developing a strategy for functional UI testing (also known as end-to-end testing).
End-to-end testing simulates real user conditions to test the functionality of an application from the front-end user interface.
Whether you’re just getting started with test automation or you already have a few automated scripts in your testing suite, defining your test automation strategy will help you improve test coverage and get faster, more consistent test results.
In this post, we’ll cover why each of the following steps is important and how to implement each one:
Once you’ve created your automation test strategy, sign up for Rainforest QA to save time and money on test automation — you can run up to five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
Quality assurance is always a balancing act between release speed and test coverage. It’s impossible to catch every single bug during the testing process, and trying to do so would grind your release process to a halt. It’s better to think of QA as an exercise in risk mitigation: How can you reduce the risk of releasing software with major bugs that could impact your bottom line or reputation?
Each company has a different tolerance for bugs based on a variety of factors, including the industry, the expectations of the users, and the competitiveness of their market segment.
In some industries, minor bugs might be so damaging to a company’s reputation that it’s worth delaying releases by several weeks in order to thoroughly test everything. In other settings, it’s much more important to get a requested feature into customers’ hands by a certain deadline, even if it means the feature comes out with a few minor bugs.
Your company’s goals for quality and speed of release will impact your automated testing strategy in a variety of ways because they will determine how much time, money, and human resources you’re willing to invest into testing up-front.
Answering the following questions will help you determine the right balance between quality and speed for your organization:
Automated testing isn’t a set-it-and-forget-it exercise. Successful test automation requires an ongoing commitment to writing test scripts, running and interpreting the results, and maintaining the test suite.
If no one is explicitly in charge of each of the duties related to automated testing, the test suite can easily become stale and unusable.
To make sure your test suite stays up-to-date, all of the responsibilities related to test automation need to be clearly assigned.
Typically, development teams will set up and run automated unit tests and integration tests as part of their software development processes. But for functional testing, there are a variety of ways to organize responsibilities, especially if you’re using an accessible automated testing tool like Rainforest QA that doesn’t require programming skills to use.
In that case, you can have a mix of responsibilities. Perhaps the developers write the functional tests after they finish writing a feature and run them as part of their release process, but a QA testing team interprets the results and takes care of ongoing test maintenance. Or perhaps the QA specialist writes and maintains the tests, but the developer is the one to actually run them and interpret the results.
In most cases, having developers stay more involved in testing will help testing move faster. But since every minute a developer spends on testing is a minute not spent building new features or fixing bugs, many teams try to shift a lot of the duties to QA specialists.
Regardless of your testing approach, testing responsibilities need to be clearly defined and understood by everyone involved so your automation suite stays up-to-date.
For most web applications, it’s not practical or feasible to test every single user path through an app. And not all user paths make sense to test with automation. Some types of tests are always better candidates for manual testing.
To figure out which features to test, you first need to create a map of all of your app’s user paths, ranked by priority. This map will also help you create a test plan after you’ve worked out your strategy.
Here is a sample functionality map for an early version of the Slack web app:
In general, you should prioritize your testing efforts around the most important user paths to make sure they are working properly. We like to use the analogy of a snow plow clearing a city’s streets after a snowstorm. The streets that see the most traffic get cleared first, and some of the side streets may never get cleared because so few people travel on them.
Once you know what user paths to test, you need to decide which tests are best suited for automated vs. manual testing. We’ll share a few general rules here.
Test cases are ready for automation when they don't change very often and get run frequently, such as smoke tests or regression tests.
Smoke tests: Tests that run through the most essential user paths to make sure the application is stable enough to move on to further testing.
Regression tests: Tests that check the original features of the application to make sure that the latest changes or updates haven’t broken anything.
Use manual testers for tests you only run once (such as exploratory tests), tests where the software — and therefore the data set — changes between each run, or tests that require human interpretation (e.g. if the feature includes a CAPTCHA).
When you’re developing your automation process, you need to decide how often each kind of test should be run and what kind of changes merit running your full test suite.
The ideal solution is to run your entire functional test suite after every product change, no matter how small. To do this, you need to have a highly optimized automated test suite. Using a tool like Rainforest QA to create and run your tests can make this feasible by making testing fast and affordable.
Even though it might seem like overkill for small changes, running your full test suite after every change has two major benefits:
However, this may not be feasible for your organization. In that case, you may need to bundle changes together before running your functional test suite, or even divide your test suite into groups of tests that need to be run for different kinds of changes.
(In the Rainforest platform, you can create Run Groups containing different sets of tests so you can easily run subsets of your entire test suite.)
Keep it as simple as you can — if you’re not careful, you can end up with a complicated list of conditional criteria for different kinds of changes. If a change falls in a gray area where it’s unclear if it meets the criteria for running the full test suite or not, we recommend running the full suite.
There are many different automated testing tools available, including open source tools and no-code tools. Choosing the right tool will impact:
Whether you’re using an open-source testing framework like Selenium or a tool that generates code for you, all test automation frameworks (except Rainforest QA) test the underlying code of an application. Testing the underlying code means you’ll eventually run into one of the following problems:
Rainforest QA is the only software testing tool that doesn’t test the underlying code. Instead, Rainforest uses pixel-matching to simulate a real user interacting with the visual layer of the app. Anyone can create, maintain, and run automated tests with Rainforest’s no-code user-friendly interface.
Here are just a few of the additional benefits that come with Rainforest QA:
Because Rainforest tests the visual layer of the application, its automated test scripts are less susceptible to breaking due to minor code changes that don’t affect the UI — unlike tools that test the underlying code.
Let’s say you want to test whether your “Try for Free” button is working after you’ve made some other changes to your web application. In Rainforest, you’d create a test that includes a step to click on the “Try for Free” button. You’d tell the test where to click by click-and-dragging a selection box around the button.
When you run the test, Rainforest’s automation will look all over the page for pixels that match your image selection. If those pixels exist anywhere on the page, the test will find them and will be able to keep going, even if the underlying code changed.
In contrast, if you were using Selenium to test this button, you could run into a variety of situations where the test could fail even though the UI was flawless.
For example, if the CSS class of the button changed from "try for free button" to "tryforfreebtn2," a Selenium test could break, but a Rainforest test would not as long as the pixels of the button on the page did not change.
A Rainforest test would fail only if the actual visual appearance of the button changed. For example, if the button was accidentally hidden due to a bug in a recent code push, the test would (and should) fail.
To be fair, Rainforest QA tests are not totally immune from “false positives” — or, test failures for which there’s no actual bug — in that a minor UI change, such as a button slightly changing shape, may break a test. But most Rainforest users find that test maintenance is less of a headache than with app testing tools like Selenium because:
Rainforest QA is an all-in-one UI test automation solution, which means it has built-in tools to allow users to create, run, and manage any number of automated tests all at once, meaning you don’t need to pay for things like a cloud-based testing grid, project management tools, and team collaboration tools just to manage your automation efforts.
All Rainforest QA plans include:
Our Pay as You Go plan also includes human-powered add-on services:
With Rainforest QA, you can run as many tests as you want, in parallel. Rather than capping the number of concurrent tests you can run on a given pricing tier, we offer usage-based pricing. So you can scale testing up or down as needed, without having to make an expensive upgrade to any of your testing infrastructure tools.
Rainforest automatically records a video of every test it runs (whether the test passes or fails). When a test fails, you can review the video to identify exactly where it failed. This can dramatically speed up the process of reviewing and classifying test failures because it makes it much easier to understand why a test failed.
Because Rainforest tests run on virtual machines (including Windows, MacOS, mobile devices), you can design tests that leave the browser and go into other apps.
For example, here’s a video walkthrough of creating a test that saves a file to the desktop from one location, then uploads the file to Google Drive. If you ever need to validate downloads or test file uploads, this is a really useful feature.
Rainforest has a built-in library full of auto-generated test data, including random emails, passwords, addresses, virtual credit cards, and more. Plus, with virtual email inboxes, you can test emails.
Testing can quickly become a bottleneck unless it’s integrated into your CI/CD pipeline. Rainforest QA offers plugins to integrate with various tools, including CircleCI, so that developers can kick off a suite of Rainforest tests at the same time as their unit tests and integration tests.
You can also download our CLI and run Rainforest tests in any development environment. For custom workflows and more complex data analysis, our open source API makes it possible to integrate Rainforest into any software stack. With a fully-automated CI/CD pipeline, it’s much easier to achieve your team’s test coverage goals without slowing down development in an agile environment.
Rainforest is a scalable, all-in-one quality assurance solution that’s appropriate for small teams just getting started with an automated testing strategy or QA-mature teams regularly running 500+ quality software tests.
Our free-forever Essentials plan makes software test automation accessible to anyone. This plan has everything you need to get started with automated testing — parallel testing on virtual machines and unlimited team members — without any hidden costs. You get up to 5 hours of free testing every month and it's only $5/hr after that. Get started for free.
With this practical guide, understand the difference between manual vs automated testing and when to use each method.
Testing the visual layer of your app is more reliable than testing the underlying code, especially for automated UI testing.
Learn how Signagelive built 500 tests in record time and dramatically sped up development cycles with automated tests.
The right codeless test automation tools make it easy for anyone to speed up their QA process without trading quality.