How to Automate Testing: A Non-Technical Guide

Contents

Almost all software and web-based companies (SaaS, e-commerce, and more) that care about a bug-free user experience will eventually need a systematic and repeatable process to test their products.

And for most software teams, the only way to feasibly accomplish testing fast enough that it doesn’t become a bottleneck in the software development lifecycle is to incorporate automation tools. Done well, automated testing can speed up software testing cycles, improve the efficiency of QA teams, and ultimately increase quality.

But setting up a high-functioning, reliable, and scalable automated testing operation is not a trivial task.

Most teams run into two challenges:

Engineering Headcount: Most teams assume they need QA engineers to set up code-based test scripts using one of the two original open-source frameworks, Selenium or Cypress. This is expensive and isolates QA testing from the other members of the team who are also heavily invested in product quality (namely product and customer success teams).
Cost of Maintaining Tests: Most teams underestimate the cost of maintaining tests. Specifically, if they use a code-based automated testing tool, the burden of test maintenance means they end up needing more and more QA engineers.

We built Rainforest QA, a no-code automated testing platform, to solve these challenges and have helped hundreds of companies develop automation strategies.

With Rainforest QA, you can start automating tests within a couple of hours—and continue to build a more rigorous QA strategy and culture as you go.

Rainforest QA is more scalable than a code-based approach like Selenium because it’s an all-in-one solution to automated testing—allowing you to run an infinite number of automated tests in parallel.

And, as we’ll point out, Rainforest QA can cut down on the maintenance burden of test automation compared to Selenium because it’s much more resistant to breaking from minor code changes that don’t affect the user.

In this tutorial, we’ll outline everything you need to know to get started with automated testing using test automation tools from Rainforest QA and help you avoid common mistakes along the way.

Table of Contents

Step 1: List Out Your Most Common User Paths (Not Edge Cases)
Step 2: Write Your First Test (without Any Code)
Step 3: Start Automation Testing with the Click of a Button
Step 4: Scale Up Your Automation Testing with a Test Plan
Step 5: Future-Proof Your Tests with Set-up, Tear-down, and Randomized Test Data

Contents

Step 1: List Out Your Most Common User Paths (Not Edge Cases)

Before you dive into creating your first automated test, we recommend that you first create a basic test coverage map. Many Rainforest customers tell us they wish they’d spent a bit more time planning out their automation strategy in the beginning because it would have saved them time in the long run.

Your coverage map should list all of the main features and user flows within your app. From there, you can pick which flows you want to cover with your first few tests.

We’ve noticed many testers make the mistake of wanting to set up test cases for every single user flow and edge case. Their first instinct is to try to test for the weird things they think are most likely to break their app, like someone entering a name with non-Roman characters.

But that makes setting up and getting value from tests super time-consuming. And it’s not very helpful because you end up creating a ton of test cases for obscure paths that most users never even experience.

Instead, you want to do the opposite. Your first priority with automation is to simply make sure the most essential features of your software work the way you expect them to.

Aim to identify the five to ten most common user paths and create automated tests for those paths. In QA terms, this is called smoke testing, or sometimes sanity testing.

Smoke testing is much less rigorous than other types of testing, such as functional testing, regression testing, integration testing, and performance testing. (We’ll define these terms later.) The smoke testing process is meant to catch major errors in software at the earliest stage possible so that teams don’t waste time building features on unstable code.

Here are two examples of smoke tests:

For an e-commerce company, the most common user paths are search, add to cart, check-out, place order, etc. By testing these paths, you’re guaranteeing that users can always do the most important thing on your site: buy something.
For a B2B web app, something like Slack, you’d want to test things like signing up with a new email, creating a new workspace, sending a DM, and creating a channel.

If you’re already familiar with the product you’re testing, then you should be able to create this list quickly. If you’re not a product person—let’s say you’re a devops engineer—you’ll want to sit down with someone who knows how users experience the app and map out the most important features and user paths.

If you’re already doing some manual testing, another way to decide on the right tests to automate is to start with the five to ten manual tests that are currently the most time-consuming to do by hand.

What Not to Test

As we mentioned before, automated testing can be brittle, meaning your tests can break when the product changes. And if your team uses agile methodologies or is running a CI/CD pipeline, your developers are going to be updating the product all the time.

You should only create automated tests for user paths that don’t change frequently. Otherwise you’ll have to maintain your tests all the time.

You also don’t want to automate tests that require or could benefit from human judgment, such as any user path that involves filling out a Captcha test or evaluating images. While a human will notice at a glance that an image is warped, stretched, turned sideways, or super pixelated, it’s very hard to write tests that will catch errors like those.

Step 2: Write Your First Test (without Any Code)

With Rainforest QA, you can create functional, UI-level tests in a matter of minutes, even if you have no technical background. That means the first test can be written by someone on the product team, the design team, or even the customer success team.

You don’t have to download anything because the entire end-to-end testing system is in the Rainforest web app. You don’t have to wait for an engineer to set up Selenium, integrate it with cloud-based browser testing software (a grid), and code the tests.

You create (or edit) every test step in Rainforest’s visual editor with a click or drag-and-select of the mouse.

For example, here’s a test created in Rainforest QA’s visual editor to validate the signup functionality on the www.rainforestqa.com website:

As you can see, anyone looking at the set of steps on the left can understand exactly what’s going on in the test.

This 3-minute video will walk you through the process of setting up your first test.

The steps are:

Add the URL of the site you want to test to Rainforest in settings (detailed instructions here).
Use the drop-down menu of available actions (click, fill, type, press key, etc.) to choose the first action. Click-and-drag with the mouse to create small screenshots that indicate where on the page the action will happen. For example, if you use the “click” action, you will take a screenshot of the button you want the test to click.
Add as many actions as you need to complete the test.
Add an “Observe” action to verify that the expected result happened. For example, “Observe that the signup success message is visible.”
Hit “Save changes”.
Run a preview of your test to make sure it’s working properly. Watch as the automation follows your test steps in a browser on a virtual machine.
Hit “Run Test” and let the automation begin.

For a walkthrough of a slightly more complicated test, check out this 4-minute video, which shows you how to create a sign-up test for Airbnb.

Step 3: Start Automation Testing with the Click of a Button

You can start incorporating automated testing into your release workflow with just two or three tests. You don’t need to wait until you’ve created all of the tests in your coverage map to begin seeing value and making an impact on quality.

So the next step is to simply start running those tests!

Running a test (or group of tests) against Rainforest QA’s automation is as quick as selecting the test environment and browser/platform. Within minutes, you can get a set of test results that are easy for anyone to interpret.

Plus, your team’s engineers can kick off automated tests for every code release using our API, CLI, or CircleCI integration.

Rainforest Keeps Video Replays of Every Test So You Can See Exactly What Happened

In Rainforest, every test result includes a video recording of the test that the automation performed, so you can see exactly where something went wrong. This significantly decreases the amount of time you’ll spend debugging test failures in Rainforest as compared to Selenium.

In Selenium, when a test fails, you have to recreate every test step to figure out what happened. (You can add code to automatically take a screenshot and/or record of the HTML DOM when a test fails, but these don’t always reveal the cause of the failure.)

In Rainforest, you can identify the exact point at which the test failed by watching the video. If it failed because the test itself was broken (e.g., due to a product change), you can jump into the visual test editor to fix it. Rainforest even offers smart suggestions for fixing broken test steps with just a click.

If it was a meaningful failure because of an actual bug in the application under test, you can immediately click a button to add tickets to Jira, or download the relevant video recording and HTTP logs for your engineering team.

Expand Your Test Suite As You Go

Once you’re comfortable with your first few tests and you have worked out any early issues, your next milestone should be to build out a more complete test suite. As a general rule, we recommend:

Five to ten smoke tests: Tests that run through the most essential user paths to make sure the application is stable enough to move on to further testing.
Fifteen happy path tests: These make sure the most important user paths will work when users enter everything properly.
Fifteen regression tests: Tests that check the original features of the application to make sure that the latest changes or updates haven’t broken anything.

If that’s all going well, you can begin to break out features of your application into categories and create a variety of tests within each category.

You might also need to incorporate any or all of the following types of tests at different points in the software development cycle:

Unit Testing: Usually done by the development team, unit tests are automated tests that ensure each unit of source code behaves as expected.
Integration testing: Combining modules that have already passed unit testing to make sure they work together.
Functional testing: Testing the user interface, APIs, database, and security and usability of the application under test to make sure it meets all of the basic requirements.
Performance testing: Testing the speed, response time, stability, and reliability of the application.
Exploratory testing: Using manual testers to hunt for bugs in an unscripted fashion.

Some companies might have a total of 40 tests for their entire application, and that’s enough. Other companies might need 500 or more tests. Rainforest QA can support an infinite number of tests and run them all in parallel, so the only limit to how many tests you can create is how much time you have available to maintain them.

For a more in-depth exploration of the trade offs of automation versus manual testing, check out this post: When is a Test Case Ready for Test Automation?

Step 4: Scale Up Your Automated Testing with a Test Plan

After you get familiar with basic automated testing, it’s a best practice to create a test plan so that you can share the responsibility of testing with more team members and keep everyone on the same page. You want everyone on your team to be able to see what user paths are being tested, how those tests are going, and what other tests still need to be created or rewritten.

A test plan should answer the following questions:

What are all of the user paths or features you’re going to test? Again, you can’t test everything, so you need to prioritize the most common user paths.
What are you trying to validate in each test? You need to get very specific here. It’s not just “can the user buy something?”, it’s “when the user clicks ‘add to cart,’ does the item show up in the cart correctly?”.
What set of conditions have to be true to validate what you’re trying to test? The conditions that have to be true are called dependencies. If you’re testing a web app like Slack, and you want to test sending a direct message, the dependencies are that you have to have an existing workspace, and that workspace must have two users in it.

A good test plan will tell you what data you need in your test environment, and what you need to do to reset your environment between test runs so that each new test runs exactly like the first one.

Step 5: Future-Proof Your Tests with Setup, Tear-down, and Randomized Test Data

To save time and headaches in the future, you need to think about repeatability of your tests when you build them.

If your test involves changing a setting in the application under test, then you need to make sure that the next time you run the test, the application starts in the same state. Otherwise, your test will break, or it’ll finish without actually testing whether that setting can be changed.

In order to avoid having to manually go into the test environment and reset the variable you were testing between each test—which almost completely negates the value of automating the test—you need to include set-up and tear-down steps in the test itself.

Let’s look at the example of setting your status to “away” in Slack.

The default setting is “active.” If you write a test and tell it to click on “Set yourself as away” and don’t include a reset step in the test, then the second time you try to run the test, you will get a false failure. The test is looking for the button that says “Set yourself as away,” but the app will show the button “Set yourself as active.”

One way to make sure your tests are repeatable is to create a brand new account every time you run the test. So for the example above, the whole test would look like:

Go to website > sign up with random email > go to email inbox > get verification code > create my first workspace > set my status to away > verify that it was set to away.

Rainforest QA makes it easy to automate the creation of accounts by providing a random email and password generator and a virtual email inbox for the types of test steps described above.

These options are built into the drop-down editor, and you can see how this works in this video of how to create a test for a sign-up flow.

Planning for repeatability of your tests can be tricky, but Rainforest QA makes it as easy as possible by including a library of actions that can automatically generate random test data (like phone numbers, names, mailing addresses, etc.) to fill in forms.

When tests create their own randomized data, instead of pulling from an external database, they’re easier to set up, more reliable, and more repeatable. You don’t have to spend any time creating or maintaining a database of test data, and you never have to worry about the test breaking because someone renamed or moved the database.

Sometimes, it’s not possible or practical to create a new account for every test. In those cases, you’ll need to create some dedicated testing accounts and include a tear-down step in each of your tests to reset the variable you were testing. So in the Slack example, the process would be:

Login with test account > create my first workspace > set my status to away > verify that it was set to away > set my status to active.

That final reset step is the tear-down step that ensures that your test can be repeated with the same results an infinite number of times—future-proofing your test.

Final Advice: Look For a Tool That Minimizes Test Maintenance

When you first set up test automation, it can be super frustrating whenever your tests break.

And as much as we would like to tell you that Rainforest QA makes tests that never break, that’s simply not possible. No matter what tool you’re using to run automated tests, you will get tests that break.

But that doesn’t mean you’re doing something wrong; that’s just how automation works. But with Rainforest, most of our customers find that test maintenance is less of a headache than with tools like Selenium because anyone can quickly update tests in the visual editor without having to involve an engineer or write any code.

Additionally, since Rainforest tests the visual layer of the user interface, instead of the underlying code, your tests won’t break from behind-the-scenes changes to the codebase that don’t affect the user, like they could in Selenium.

Rainforest tests can sometimes break because of changes to the UI, but the app will identify many of these errors on its own and make smart suggestions to allow you to fix tests with one click.

Remember, even if you have to spend time on test maintenance, you’re still gaining a lot of benefits from automated testing:

Lightning fast test turnaround
The ability to run infinite tests in parallel
The ability to focus on higher value tasks—no more mindless manual clicking through the application
Reduced human error
Improved overall product quality

As you gain familiarity with test automation, you’ll want to level up your quality assurance strategy to continue getting more efficient and effective. Here are some resources that can help:

Get Started With Automated Software Testing in Two Hours or Less with Rainforest QA

Rainforest QA makes it easy to get started with automated testing and build as you go. You don’t need to learn a new programming language, you don’t need to buy additional services from other vendors to run or manage your tests, and anyone on your team can create, run, maintain, debug, and interpret tests.

It’s a fast and scalable all-in-one solution that’s appropriate for teams that are just dipping their toes into automated testing, as well as QA-mature teams running 500+ software tests on a regular basis.

Talk to us about setting up a Rainforest account so you can try it out for yourself.