Almost all software and web-based companies (SaaS, e-commerce, and more) that care about a bug-free user experience will eventually need a systematic and repeatable process to test their products.
And for most software teams, the only way to feasibly accomplish testing fast enough that it doesn’t become a bottleneck in the software development lifecycle is to incorporate automation tools. Done well, automated testing can speed up software testing cycles, improve the efficiency of QA teams, and ultimately increase quality.
But setting up a high-functioning, reliable, and scalable automated testing operation is not a trivial task.
Most teams run into two challenges:
We built Rainforest QA, a no-code automated testing platform, to solve these challenges and have helped hundreds of companies develop automation strategies.
With Rainforest QA, you can start automating tests within a couple of hours—and continue to build a more rigorous QA strategy and culture as you go.
Rainforest QA is more scalable than a code-based approach like Selenium because it’s an all-in-one solution to automated testing—allowing you to run an infinite number of automated tests in parallel.
And, as we’ll point out, Rainforest QA can cut down on the maintenance burden of test automation compared to Selenium because it’s much more resistant to breaking from minor code changes that don’t affect the user.
In this tutorial, we’ll outline everything you need to know to get started with automated testing using test automation tools from Rainforest QA and help you avoid common mistakes along the way.
Table of Contents
Before you dive into creating your first automated test, we recommend that you first create a basic test coverage map. Many Rainforest customers tell us they wish they’d spent a bit more time planning out their automation strategy in the beginning because it would have saved them time in the long run.
Your coverage map should list all of the main features and user flows within your app. From there, you can pick which flows you want to cover with your first few tests.
We’ve noticed many testers make the mistake of wanting to set up test cases for every single user flow and edge case. Their first instinct is to try to test for the weird things they think are most likely to break their app, like someone entering a name with non-Roman characters.
But that makes setting up and getting value from tests super time-consuming. And it’s not very helpful because you end up creating a ton of test cases for obscure paths that most users never even experience.
Instead, you want to do the opposite. Your first priority with automation is to simply make sure the most essential features of your software work the way you expect them to.
Aim to identify the five to ten most common user paths and create automated tests for those paths. In QA terms, this is called smoke testing, or sometimes sanity testing.
Smoke testing is much less rigorous than other types of testing, such as functional testing, regression testing, integration testing, and performance testing. (We'll define these terms later.) The smoke testing process is meant to catch major errors in software at the earliest stage possible so that teams don't waste time building features on unstable code.
Here are two examples of smoke tests:
If you’re already familiar with the product you’re testing, then you should be able to create this list quickly. If you’re not a product person—let's say you're a devops engineer—you'll want to sit down with someone who knows how users experience the app and map out the most important features and user paths.
If you’re already doing some manual testing, another way to decide on the right tests to automate is to start with the five to ten manual tests that are currently the most time-consuming to do by hand.
As we mentioned before, automated testing can be brittle, meaning your tests can break when the product changes. And if your team uses agile methodologies or is running a CI/CD pipeline, your developers are going to be updating the product all the time.
You should only create automated tests for user paths that don’t change frequently. Otherwise you’ll have to maintain your tests all the time.
You also don’t want to automate tests that require or could benefit from human judgment, such as any user path that involves filling out a Captcha test or evaluating images. While a human will notice at a glance that an image is warped, stretched, turned sideways, or super pixelated, it’s very hard to write tests that will catch errors like those.
With Rainforest QA, you can create functional, UI-level tests in a matter of minutes, even if you have no technical background. That means the first test can be written by someone on the product team, the design team, or even the customer success team.
You don’t have to download anything because the entire end-to-end testing system is in the Rainforest web app. You don’t have to wait for an engineer to set up Selenium, integrate it with cloud-based browser testing software (a grid), and code the tests.
You create (or edit) every test step in Rainforest’s visual editor with a click or drag-and-select of the mouse.
For example, here’s a test created in Rainforest QA’s visual editor to validate the signup functionality on the www.rainforestqa.com website:
As you can see, anyone looking at the set of steps on the left can understand exactly what’s going on in the test.
This 2-minute video will walk you through the process of setting up your first test.
The steps are:
For a walkthrough of a slightly more complicated test, check out this 4-minute video, which shows you how to create a sign-up test for Airbnb.
You can start incorporating automated testing into your release workflow with just two or three tests. You don’t need to wait until you’ve created all of the tests in your coverage map to begin seeing value and making an impact on quality.
So the next step is to simply start running those tests!
Running a test (or group of tests) against Rainforest QA’s automation is as quick as selecting the test environment and browser/platform. Within minutes, you can get a set of test results that are easy for anyone to interpret.
Plus, your team’s engineers can kick off automated tests for every code release using our API, CLI, or CircleCI integration.
In Rainforest, every test result includes a video recording of the test that the automation performed, so you can see exactly where something went wrong. This significantly decreases the amount of time you’ll spend debugging test failures in Rainforest as compared to Selenium.
In Selenium, when a test fails, you have to recreate every test step to figure out what happened. (You can add code to automatically take a screenshot and/or record of the HTML DOM when a test fails, but these don’t always reveal the cause of the failure.)
In Rainforest, you can identify the exact point at which the test failed by watching the video. If it failed because the test itself was broken (e.g., due to a product change), you can jump into the visual test editor to fix it. Rainforest even offers smart suggestions for fixing broken test steps with just a click.
If it was a meaningful failure because of an actual bug in the application under test, you can immediately click a button to add tickets to Jira, or download the relevant video recording and HTTP logs for your engineering team.
Once you’re comfortable with your first few tests and you have worked out any early issues, your next milestone should be to build out a more complete test suite. As a general rule, we recommend:
If that’s all going well, you can begin to break out features of your application into categories and create a variety of tests within each category.
You might also need to incorporate any or all of the following types of tests at different points in the software development cycle:
Some companies might have a total of 40 tests for their entire application, and that’s enough. Other companies might need 500 or more tests. Rainforest QA can support an infinite number of tests and run them all in parallel, so the only limit to how many tests you can create is how much time you have available to maintain them.
For a more in-depth exploration of the trade offs of automation versus manual testing, check out this post: When is a Test Case Ready for Test Automation?
After you get familiar with basic automated testing, it’s a best practice to create a test plan so that you can share the responsibility of testing with more team members and keep everyone on the same page. You want everyone on your team to be able to see what user paths are being tested, how those tests are going, and what other tests still need to be created or rewritten.
A test plan should answer the following questions:
A good test plan will tell you what data you need in your test environment, and what you need to do to reset your environment between test runs so that each new test runs exactly like the first one.
To save time and headaches in the future, you need to think about repeatability of your tests when you build them.
If your test involves changing a setting in the application under test, then you need to make sure that the next time you run the test, the application starts in the same state. Otherwise, your test will break, or it’ll finish without actually testing whether that setting can be changed.
In order to avoid having to manually go into the test environment and reset the variable you were testing between each test—which almost completely negates the value of automating the test—you need to include set-up and tear-down steps in the test itself.
Let’s look at the example of setting your status to “away” in Slack.
The default setting is “active.” If you write a test and tell it to click on “Set yourself as away” and don’t include a reset step in the test, then the second time you try to run the test, you will get a false failure. The test is looking for the button that says “Set yourself as away,” but the app will show the button “Set yourself as active.”
One way to make sure your tests are repeatable is to create a brand new account every time you run the test. So for the example above, the whole test would look like:
Go to website > sign up with random email > go to email inbox > get verification code > create my first workspace > set my status to away > verify that it was set to away.
Rainforest QA makes it easy to automate the creation of accounts by providing a random email and password generator and a virtual email inbox for the types of test steps described above.
These options are built into the drop-down editor, and you can see how this works in this video of how to create a test for a sign-up flow.
Planning for repeatability of your tests can be tricky, but Rainforest QA makes it as easy as possible by including a library of actions that can automatically generate random test data (like phone numbers, names, mailing addresses, etc.) to fill in forms.
When tests create their own randomized data, instead of pulling from an external database, they’re easier to set up, more reliable, and more repeatable. You don’t have to spend any time creating or maintaining a database of test data, and you never have to worry about the test breaking because someone renamed or moved the database.
Sometimes, it’s not possible or practical to create a new account for every test. In those cases, you’ll need to create some dedicated testing accounts and include a tear-down step in each of your tests to reset the variable you were testing. So in the Slack example, the process would be:
Login with test account > create my first workspace > set my status to away > verify that it was set to away > set my status to active.
That final reset step is the tear-down step that ensures that your test can be repeated with the same results an infinite number of times—future-proofing your test.
When you first set up test automation, it can be super frustrating whenever your tests break.
And as much as we would like to tell you that Rainforest QA makes tests that never break, that’s simply not possible. No matter what tool you’re using to run automated tests, you will get tests that break.
But that doesn’t mean you’re doing something wrong; that’s just how automation works. But with Rainforest, most of our customers find that test maintenance is less of a headache than with tools like Selenium because anyone can quickly update tests in the visual editor without having to involve an engineer or write any code.
Additionally, since Rainforest tests the visual layer of the user interface, instead of the underlying code, your tests won’t break from behind-the-scenes changes to the codebase that don’t affect the user, like they could in Selenium.
Rainforest tests can sometimes break because of changes to the UI, but the app will identify many of these errors on its own and make smart suggestions to allow you to fix tests with one click.
Remember, even if you have to spend time on test maintenance, you’re still gaining a lot of benefits from automated testing:
As you gain familiarity with test automation, you’ll want to level up your quality assurance strategy to continue getting more efficient and effective. Here are some resources that can help:
Rainforest QA makes it easy to get started with automated testing and build as you go. You don’t need to learn a new programming language, you don’t need to buy additional services from other vendors to run or manage your tests, and anyone on your team can create, run, maintain, debug, and interpret tests.
It’s a fast and scalable all-in-one solution that’s appropriate for teams that are just dipping their toes into automated testing, as well as QA-mature teams running 500+ software tests on a regular basis.
You can try Rainforest QA yourself by starting a 14 day free trial.
We provide practical guidance on how to start automation testing from scratch and how to choose a test automation tool.
10 automated front-end testing tools that can help you release new features faster with fewer bugs.
With these test automation maintenance tips, prevent automated test breakage and spend less time fixing broken tests.
In this post, you'll learn how to automate regression testing without having to know any code.