Both manual and automated testing have their place in the software development lifecycle. Understanding the pros and cons of each testing method — and the tools available for each — will help you find the most effective balance for your team.
In this article, we’ll talk about:
Sign up for Rainforest QA to save time and money on automated test execution — you can run up to five hours of no-code automated tests for free, every month. It’s only $5/hour after that.
Manual testing is any type of testing where someone physically clicks through the user-path being tested — either by following written instructions to test a specific feature or by clicking around in an unscripted way. Manual testing is good for:
The kinds of testing that are good for manual testing include:
Manual testing is easy to set up, and if you’re only ever running a specific test once, or you know the software you’re testing is going to be changed between each test run, it’s a perfect fit.
However, as soon as you start running the same test multiple times, like with regression testing, manual testing presents a lot of downsides:
The right tool can help mitigate these downsides, but for stable, repetitive tasks, automated testing is usually the best option.
Automated testing uses software tools or coding frameworks to automatically run tests that validate user paths, UI workflows, input fields, and more.
Automation tests are much cheaper to execute, take less time to run, and provide more consistent results than manual tests.
If a software team is running a continuous development pipeline, they can hook a suite of automated tests into their release process so that when they finish writing code, they can launch the tests with the click of a button before sending the code to the next stage. Because automated tests can be run on-demand and only take a few minutes to execute, they can be run much more frequently.
More frequent testing means there’s less new code to evaluate in each test run, which means issues are easier to isolate and debug.
Frequent testing also means you’ll find bugs closer to the time the code was written (minutes or days later, instead of weeks later). The new code will be fresher in the developer’s mind and they’ll have an easier time remembering the assumptions and decisions they made.
Testing types that are good for automated testing include:
Because it usually takes more time to write automated tests than to write instructions for human testers, the initial investment can be steep with automated testing, but the long-term benefits often outweigh the cost.
The main downside of automation testing is that someone on the team will have to spend time maintaining the tests to reflect any changes you make to your product.
Before deciding which tests to automate, you need to determine if your application and your testing process are ready for automation.
The most important factor in automation readiness is the stability of your product. If you have a consistent series of tests that don't change very often and get run frequently, as in smoke or regression tests, it’s likely that automation could save you a lot of time and money. If your product is still in the early development phase and lots of major code changes are occurring on a regular basis, it’s not time to automate yet — wait until the application is more stable.
Other reasons to look into automated testing:
Before you can get started with an automation tool, you may need to do some work to set up the proper infrastructure for automated testing. You will likely need:
Once you’ve determined that your team is ready for automated testing, a clear QA strategy will help you determine which test scripts to automate.
Quality assurance is an exercise in risk mitigation — how many bugs can your customers find before they lose faith in the company? Once you know the answer to that question, you’ll be able to build an effective strategy that appropriately balances release speed and product quality for your company.
When considering what test cases to include in an automated testing suite, we like to use our Snowplow Strategy. Think of all of the possible user paths through an app like a city map with hundreds of streets. After a blizzard, snowplows work to clear the most trafficked streets first because they affect the most people. The side streets may get plowed later, but in large cities, some streets never get plowed.
Likewise, you should prioritize your testing process around the most important user paths. These are the tests that should be automated, if possible.
Few software testing tools offer solutions for both manual and automated testing, making it a hassle to combine and transition between the two.
Rainforest QA gives you the speed of automation and the flexibility and intelligence of humans by offering automated testing and crowdsourced testing in one unified platform.
Rainforest QA’s no-code automation means anyone can write and maintain test scripts.
To create (or edit) any test step in Rainforest QA, simply select a preset action (like “click”, “select”, or “type”) and then click-and-drag to select the element you want to apply the action to. This screenshot is used to find elements and verify visibility.
Once you’ve created each action along the user path you want to test, you can play back the actions you created to verify that the test will do what you intended.
Then, when you’re ready to test, you launch it with the click of a button in the Rainforest platform or via the API or CLI. Our automation service will run your tests concurrently on virtual machines in our cloud.
For a more detailed look into how to create a full test, check out this 4-minute video.
Every Rainforest test gets recorded (whether it passes or fails) and can be played back at any time. This makes understanding test failures much easier and quicker because you can see how the application performed without having to recreate the exact situation.
Lastly, Rainforest tests mimic real user behavior by testing the visual layer of the user interface using pixel-matching. We go into more detail as to why we think testing the visual layer gives better results than testing the underlying code (as you would with automated testing tools like Selenium IDE) in this blog post.
The short version is that testing the visual layer is closer to the end user experience which means you’ll be catching more bugs that your users actually care about.
Note: You can also use our test writing service and have our group of Rainforest experts build a suite of 20 tests for you within a week.
For all the test cases that never make sense to automate, it can be helpful to outsource manual testing. That’s where crowdtesting platforms can be super helpful, and with Rainforest, you can manage a suite of automation tests and manual tests using the same user-friendly tool.
To use Rainforest’s Crowd Testing Platform, you can write test instructions using our library of standard actions or in plain English and send them to our worldwide community of QA specialists. Each test gets run by at least two specialists simultaneously (and more if needed), and results are returned on average within 17 minutes.
Our on-demand crowd testing service enables some of the fastest manual testing available.
Not only is our crowd testing platform fast, it also helps mitigate miscommunications and human error. Our cloud-based editor makes writing tests easy and provides straightforward communication between your team and the tester community. You can use either the visual editor or the plain-text editor to write tests for the tester community.
The plain-text editor lets you write instructions in free-form English. Each step has two parts: a tester instruction and a tester confirmation.
The visual editor lets you write tests in the same way you would for automated tests. Then, each step is converted into clear sentences for manual testers to follow.
If you’re not ready for automation (because your product is still changing frequently) but you need more testing and faster results to keep up with your release schedule, crowdsourced testing can help.
Testing can quickly become a bottleneck unless it’s integrated into your CI/CD pipeline. Rainforest QA offers plugins to integrate with various tools, including CircleCI, so that developers can kick off a suite of Rainforest tests at the same time as their unit tests and integration tests.
You can also download our CLI and run Rainforest tests in any environment. For custom workflows and more complex data analysis, our open source API makes it possible to integrate Rainforest into any software stack. With a fully-automated CI/CD pipeline, it’s much easier to achieve your team’s test coverage goals without slowing down development in an agile environment.
Our free-forever Essentials plan makes software test automation accessible to anyone. This plan has everything you need for test automation — parallel testing on virtual machines and unlimited team members — without any hidden costs. You get up to 5 hours of free testing every month and it's only $5/hr after that.
To use crowdsourced testing, you can opt for the Pay As You Go plan. This plan allows you to scale your testing up or down to meet project requirements at a moment's notice while only paying for what you use.
You can also reach out to us and create a Custom plan to fit your individual needs.
With Rainforest QA, anyone can automate and maintain end-to-end tests without learning a new programming language. Whether they're using our tester community or automated tests, QA teams can create and run tests fast enough to keep up with agile development teams and teams doing continuous integration and continuous delivery.
It’s a scalable, all-in-one test automation solution that’s appropriate for small teams just getting started with automated testing or QA-mature teams regularly running 500+ software tests.
Follow these 5 steps to create an automated testing strategy that answers the who, what, when, why, and how of software test automation.
Testing the visual layer of your app is more reliable than testing the underlying code, especially for automated UI testing.
Learn how Signagelive built 500 tests in record time and dramatically sped up development cycles with automated tests.
The right codeless test automation tools make it easy for anyone to speed up their QA process without trading quality.