Continuous testing is the practice of testing software as frequently as possible (e.g., multiple times per day) and as early as possible in the development lifecycle. With more frequent updates and faster bug fixes, your product can be more responsive to market changes and the needs of customers.
However, achieving truly continuous testing is challenging, especially when using a testing framework like Selenium, where it can take hours or days to run a full automated test suite and triage the test results. This makes it impractical to run a full suite of tests for every minor change, which is why many teams end up with larger releases and slower than ideal release cycles. Without the right testing tool, testing may take so long that you have to decide between releasing the new build with possible bugs or delaying the release process.
In this post, we cover four essential features a software testing platform should have to support continuous testing. Then, we compare five continuous testing tools according to those factors, starting with our solution, Rainforest QA.
Table of Contents
Ready to release faster with continuous testing? Sign up for Rainforest QA—you can run up to five hours of no-code automated software testing for free, every month. It’s only $5/hour after that.
Many testing tools offer basic support for continuous testing by integrating with CI tools like CircleCI Orb or Jenkins. But, being able to kick off a suite of regression tests automatically is only one part of making testing a seamless part of your delivery pipeline. To effectively support continuous testing, the tool needs to have the following capabilities:
With continuous testing, you need to constantly create new tests to cover new features and edit existing tests to reflect changes in existing functionalities. If you can’t write and edit tests fast enough, you’ll have to slow down each release or reduce your test coverage.
That’s why being able to write and edit tests as quickly as possible is essential to continuous testing. A no-code tool can help speed up test creation, however, it’s important to choose the right kind. The most common type of no-code tool is a record-and-playback tool, where you click through a test case manually and the tool turns your actions into a Selenium test script. But record-and-playback tools have a couple of downsides:
That’s why we recommend using a true no-code testing tool that can handle complex test cases and allows you to make any needed changes without coding.
Rainforest QA is the only true no-code tool that makes it possible for anyone to write and maintain tests without writing a single line of code. Not only does this mean anyone can assist with testing, but it also means that developers will be able to write or update tests in less time.
As mentioned above, the ability to quickly create and update test scripts is essential for continuous testing. Another way to speed up the process of creating tests is to avoid writing the same exact sequence of test steps multiple times. For example, if a lot of your tests start by logging into a test account, it would save time if you could write that sequence just once and then insert the entire sequence into any test that starts by logging in.
Different tools have different names for this capability (we call them “embedded tests”) and some tools don’t offer it at all.
Additionally, when it comes to updating commonly used sequences, some tools require that you update the sequence in each test case individually. With Rainforest QA, you can update the sequence in just one place and it will automatically get updated across all tests that use that sequence.
Not every test failure represents a real bug. Sometimes tests will fail because of hiccups in the testing environment or because of outdated tests. While it’s usually ideal to fix broken tests immediately and not move on to the next step in the deployment cycle until you get 100% of tests to pass, that’s not always feasible.
You may need to instead confirm that the failed tests were broken tests, not real bugs. But figuring out why a test failed is often one of the most time-consuming parts of software testing and can halt the release process if you don’t have tools to help you determine the cause of test failures.
When it takes a long time to figure out why a test failed, it can be tempting to assume the test was broken, but the software is fine. Of course, this defeats the purpose of testing because it provides no insight into the quality of your software.
Video replays are one of the most effective and efficient ways to determine the cause of test failures. They require no coding skills to understand, and you can even compare a successful test run to a failed test run in order to understand why the test failed. That’s why we recommend using a tool like Rainforest QA that automatically records a video of every test run and stores all recordings indefinitely.
The pricing structure of many automated testing tools puts limits on the number of tests you can run each month, depending on how much you pay. This can be a problem for continuous testing, because developers need to be able to kick off tests whenever they’re ready, without checking to see if they have enough testing credits left in the month.
If you use up all the available testing credits before the month is over, you have to:
Additionally, many testing tools put limits on the number of users in each price tier. With continuous testing, it’s common to involve multiple people in the testing process, but the typical pricing structure disincentivizes sharing the labor between team members.
We recommend choosing a tool—like Rainforest QA—that lets you:
Rainforest QA makes it possible for anyone to handle all aspects of functional software testing.
Here’s how it works.
Rainforest tests use pixel-matching to interact with the visual layer of an application. This means Rainforest closely mimics the way end-users interact with web applications.
There are two basic parts to writing any Rainforest test step:
For more complex test cases, you’ll also be able to add additional wait times, embed one test in another test (more on this in the next section), leave comments for others who may view the test, and much more.
Then, you can preview the entire test or individual steps to check your work.
When you’re ready for a test run, you can run multiple tests at once using our extensive network of virtual machines. With the virtual machines, you’ll have access to multiple operating systems including Windows, MacOS, iOS, and Android. You’ll also have access to newer and older versions of the four major browsers: Safari, Microsoft Edge, Chrome, and Firefox.
To speed up test creation and test maintenance, you can embed one Rainforest test into another Rainforest test. Instead of adding an individual test step, you simply search for the name of a test and select the appropriate one.
Then, you can expand the embedded test to show each step or collapse it to save space.
If you edit the embedded test steps in one place, that edit will automatically carry over into any other test that uses those steps. Instead of writing the same sequence of steps or making the same edit in 30 different places, embedded tests let you write the sequence or make the edit just once.
Rainforest makes it easy to understand why tests fail by automatically recording every test run, pass or fail, and storing the recordings indefinitely.
Next to each recording, you’ll see the test steps so you can follow along as it plays out.
Under every test step is an ‘Investigate Action’ button that will take you to a screen with further details about how the result was determined. You’ll also have access to HTTP logs and browser logs, which are often the key to identifying if the test failed due to something in the test environment rather than a bug in the software.
Once you’ve determined the cause of the failure, you can categorize the failure with custom labels.
Rainforest QA makes it really easy to quickly understand and categorize test failures. However, minimizing the number of failed tests that happen in the first place is the best way to save time and resources.
Sometimes tests will fail because of system overload, hiccups in the test environment, or for no immediately obvious reason (if this issue continues for a particular test, it’s commonly referred to as a ‘flaky test’).
While there are many ways to mitigate flaky tests, one of the most common methods is to rerun failed tests. This typically takes care of test failures caused by temporary glitches in the test environment or system.
Rainforest QA offers the ability to configure automatic retries so you don’t have to manually restart each failed test. You can configure the test to automatically re-run up to three times, and it will rerun until it passes or it hits the specified maximum number of retries, whichever comes first.
Rainforest QA offers a CLI so you can start a Rainforest test run from any CI/CD or DevOps tool (e.g., Jenkins, Bamboo, GIT, etc.). We offer direct integrations with CircleCI Orb and GitHub Actions. These integrations provide pre-written commands to download and unpack the CLI and then run Rainforest as a step or job in the delivery process.
Rainforest also comes with a Jira integration to help optimize the debugging process. Anyone on the QA team can automatically create a Jira ticket for the development team that includes the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.
Finally, Rainforest integrates with Slack and Microsoft Teams, so you can get instant notifications of any test failure.
Our free-forever Professional plan gives you five hours of free testing every month. Then, it’s only $5/hour for the time spent running tests, and there are no limits on the number of users. You won’t be locked into a contract that limits the amount of tests you can run. And, you won’t be charged for any of the time spent writing/editing tests, categorizing test failures, or otherwise managing your tests.
You’ll have access to everything you need to write, run, and manage a full suite of functional tests as part of your CI/CD pipeline.
Selenium is the original software testing framework for automating interactions with web browsers. Only someone with programming experience will be able to write or edit test scripts in Selenium, but you will have a variety of programming languages to choose from (i.e., Ruby, Java, Python, PHP, etc.).
Selenium is open-source, so you won’t have to worry about a rigid pricing structure. However, you will eventually need other tools in order to run multiple tests simultaneously, organize your test cases, or improve your workflow. Additionally, you may need to hire additional software engineers as your testing needs outgrow the capacity of your team.
Certain aspects of Cypress are open source (such as their test script editor) but eventually, you’ll need to upgrade to one of their paid tools to run multiple tests or efficiently manage your test suite.
Katalon offers desktop, web, and mobile app testing as well as API testing and some performance testing. Katalon’s pricing structure is based on the number of users who will need access to your test suite.
Tricentis Tosca is an enterprise-focused automated testing tool, built for teams practicing agile methodologies. They support continuous testing by connecting to various continuous integration tools. Tricentis Tosca offers real-time business risk assessment so you can decide what testing is the most important at any given moment.
Finally, Tricentis Tosca offers cloud and on-premise versions of their software, but you have to contact their team for pricing details.
With Rainforest QA, anyone can manage a full suite of tests as part of the software development process. It integrates with any CI/CD ecosystem to fit seamlessly into your delivery process. It’s a scalable, all-in-one test automation solution that’s appropriate for small teams just getting started with automated software testing or QA-mature testing teams practicing continuous delivery and regularly running 500+ software tests.
Get started with Rainforest QA for free. You get 5 hours of free testing every month. It’s only $5/hour after that—and you only pay for what you use.
You can only move as fast as your QA process allows. That’s why in order to do continuous delivery, you need to adopt an equally continuous QA process.
Testing earlier helps teams catch bugs earlier and move faster. Here are three ways that three different companies approach testing to help speed up development and deploy more confidently.
You probably heard the news, CI is cool. In this post I'm going to walk you through the basics of what CI is, and why you need to use it, like now.
In this post, we explore modular testing, one facet of continuous testing, and learn how it contributes to a fast, reliable QA strategy.