Continuous testing starts from the beginning of development and continues through production. The aim of continuous testing is to ship your code faster and give your customers high quality releases. We advocate “shifting left,” or moving testing earlier in the development process. The more you can “shift left” with continuous testing, the earlier you’ll find bugs and the faster you’ll be able to get your product to market — being more responsive to market changes and the needs of customers.

However, achieving truly continuous testing is challenging, especially when using a testing framework like Selenium, where it can take hours or days to run a full automated test suite and triage the test results. This makes it impractical to run a full suite of tests for every minor change, which is why many teams end up with larger releases and slower than ideal release cycles. Without the right testing tool, testing may take so long that you have to decide between releasing the new build with possible bugs or delaying the release process. 

In this post, we cover four essential features a software testing platform should have to support continuous testing. Then, we compare five continuous testing tools according to those factors, starting with our solution, Rainforest QA. 

Table of Contents

  • ‍Four Essential Features of a Continuous Testing Tool
  • Rainforest QA: No-code Test Automation for Faster Releases
    • Let Anyone Write or Edit Tests without Coding
    • Speed Up Test Creation and Maintenance with Embedded Tests
    • Triage Test Results in Minutes with Video Replays‍
    • Mitigate Test Failures with Automatic Retries‍
    • Run Tests from Any CI/CD or DevOps Tool‍
    • Add Unlimited Team Members and Only Pay for Time Spent Running Tests
  • Four Other Continuous Testing Tools

Ready to release faster with continuous testing? Talk to us about a Rainforest plan that fits your needs.

Four Essential Features of a Continuous Testing Tool

Many testing tools offer basic support for continuous testing by integrating with CI tools like CircleCI Orb or Jenkins. But, being able to kick off a suite of regression tests automatically is only one part of making testing a seamless part of your delivery pipeline. To effectively support continuous testing, the tool needs to have the following capabilities:

1. No-code test creation and maintenance that anyone can use

With continuous testing, you need to constantly create new tests to cover new features and edit existing tests to reflect changes in existing functionalities. If you can’t write and edit tests fast enough, you’ll have to slow down each release or reduce your test coverage.

That’s why being able to write and edit tests as quickly as possible is essential to continuous testing. A no-code tool can help speed up test creation, however, it’s important to choose the right kind. The most common type of no-code tool is a record-and-playback tool, where you click through a test case manually and the tool turns your actions into a Selenium test script. But record-and-playback tools have a couple of downsides: 

  • You’re typically pretty limited in the types of test cases you can create without having to write additional code
  • You may have to re-record the entire test step or edit the code directly to make any changes

That’s why we recommend using a true no-code testing tool that can handle complex test cases and allows you to make any needed changes without coding.

Rainforest QA is the only true no-code tool that makes it possible for anyone to write and maintain tests without writing a single line of code. Not only does this mean anyone can assist with testing, but it also means that developers will be able to write or update tests in less time.

2. The ability to save and reuse commonly repeated sequences

As mentioned above, the ability to quickly create and update test scripts is essential for continuous testing. Another way to speed up the process of creating tests is to avoid writing the same exact sequence of test steps multiple times. For example, if a lot of your tests start by logging into a test account, it would save time if you could write that sequence just once and then insert the entire sequence into any test that starts by logging in. 

Different tools have different names for this capability (we call them “embedded tests”) and some tools don’t offer it at all. 

Additionally, when it comes to updating commonly used sequences, some tools require that you update the sequence in each test case individually. With Rainforest QA, you can update the sequence in just one place and it will automatically get updated across all tests that use that sequence. 

3. Automatic video recordings of every test run 

Not every test failure represents a real bug. Sometimes tests will fail because of hiccups in the testing environment or because of outdated tests. While it’s usually ideal to fix broken tests immediately and not move on to the next step in the deployment cycle until you get 100% of tests to pass, that’s not always feasible. 

You may need to instead confirm that the failed tests were broken tests, not real bugs. But figuring out why a test failed is often one of the most time-consuming parts of software testing and can halt the release process if you don’t have tools to help you determine the cause of test failures. 

When it takes a long time to figure out why a test failed, it can be tempting to assume the test was broken, but the software is fine. Of course, this defeats the purpose of testing because it provides no insight into the quality of your software. 

Video replays are one of the most effective and efficient ways to determine the cause of test failures. They require no coding skills to understand, and you can even compare a successful test run to a failed test run in order to understand why the test failed. That’s why we recommend using a tool like Rainforest QA that automatically records a video of every test run and stores all recordings indefinitely. 

4. No limits on users or tests

The pricing structure of many automated testing tools puts limits on the number of tests you can run each month, depending on how much you pay. This can be a problem for continuous testing, because developers need to be able to kick off tests whenever they’re ready, without checking to see if they have enough testing credits left in the month.

If you use up all the available testing credits before the month is over, you have to: 

  • Upgrade to the next pricing tier (and possibly pay for far more credits than you need)
  • Wait to run your tests until the start of the next month (or try to cover the gap with manual testing)
  • Risk deploying buggy software into the production environment

Additionally, many testing tools put limits on the number of users in each price tier. With continuous testing, it’s common to involve multiple people in the testing process, but the typical pricing structure disincentivizes sharing the labor between team members.

We recommend choosing a tool—like Rainforest QA—that lets you: 

  1. Run as many tests as you need without worrying about overages.
  2. Add as many team members as you need without additional charges.

Rainforest QA: No-code Test Automation for Faster Releases

Rainforest QA makes it possible for anyone to handle all aspects of functional software testing. 

Here’s how it works. 

Let Anyone Write or Edit Tests without Coding

Rainforest tests use pixel-matching to interact with the visual layer of an application. This means Rainforest closely mimics the way end-users interact with web applications. 

There are two basic parts to writing any Rainforest test step: 

  1. Select an action from the dropdown menu such as click, double-click, observe, etc. 
  2. Take a screenshot of the element you want to apply the action to by clicking and dragging your cursor over the element (this screenshot is used to find and interact with the element during test runs).

For more complex test cases, you’ll also be able to add additional wait times, embed one test in another test (more on this in the next section), leave comments for others who may view the test, and much more.

Add a Click Action in Rainforest QA: Try for Free Button example

Then, you can preview the entire test or individual steps to check your work.

When you’re ready for a test run, you can run multiple tests at once using our extensive network of virtual machines. With the virtual machines, you’ll have access to multiple operating systems including Windows, macOS, iOS, and Android. You’ll also have access to newer and older versions of the four major browsers: Safari, Microsoft Edge, Chrome, and Firefox.

Speed Up Test Creation and Maintenance with Embedded Tests

To speed up test creation and test maintenance, you can embed one Rainforest test into another Rainforest test. Instead of adding an individual test step, you simply search for the name of a test and select the appropriate one. 

Rainforest Signup Flow: Embed a Test

Then, you can expand the embedded test to show each step or collapse it to save space. 

If you edit the embedded test steps in one place, that edit will automatically carry over into any other test that uses those steps. Instead of writing the same sequence of steps or making the same edit in 30 different places, embedded tests let you write the sequence or make the edit just once. 

Triage Test Results in Minutes with Video Replays

Rainforest makes it easy to understand why tests fail by automatically recording every test run, pass or fail, and storing the recordings indefinitely. 

Next to each recording, you’ll see the test steps so you can follow along as it plays out. 

Airbnb example: Download HTTP Logs, Browser Logs, Video

Under every test step is an ‘Investigate Action’ button that will take you to a screen with further details about how the result was determined. You’ll also have access to HTTP logs and browser logs, which are often the key to identifying if the test failed due to something in the test environment rather than a bug in the software.

Book a Flexible Trip: Investigate Action example

Once you’ve determined the cause of the failure, you can categorize the failure with custom labels. 

Failed Tests Summary: Run Results and Custom Labels

Rainforest QA makes it really easy to quickly understand and categorize test failures. However, minimizing the number of failed tests that happen in the first place is the best way to save time and resources. 

Mitigate Test Failures with Automatic Retries

Sometimes tests will fail because of system overload, hiccups in the test environment, or for no immediately obvious reason (if this issue continues for a particular test, it’s commonly referred to as a ‘flaky test’). 

While there are many ways to mitigate flaky tests, one of the most common methods is to rerun failed tests. This typically takes care of test failures caused by temporary glitches in the test environment or system. 

Rainforest QA offers the ability to configure automatic retries so you don’t have to manually restart each failed test. You can configure the test to automatically re-run up to three times, and it will rerun until it passes or it hits the specified maximum number of retries, whichever comes first. 

Test Retries: Failed tests will be retried up to 2 times.

Run Tests from Any CI/CD or DevOps Tool

Rainforest QA offers a CLI so you can start an automated test in CI/CD from any CI/CD or DevOps tool (e.g., Jenkins, Bamboo, GIT, etc.). We offer direct integrations with CircleCI Orb and GitHub Actions. These integrations provide pre-written commands to download and unpack the CLI and then run Rainforest as a step or job in the delivery process. 

Rainforest also comes with a Jira integration to help optimize the debugging process. Anyone on the QA team can automatically create a Jira ticket for the development team that includes the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest.

Finally, Rainforest integrates with Slack and Microsoft Teams, so you can get instant notifications of any test failure. 

You’ll have access to everything you need to write, run, and manage a full suite of functional tests as part of your CI/CD pipeline. 

Talk to us about setting up a Rainforest plan that fits your continuous testing needs.

Four Other Continuous Testing Tools

Selenium

Selenium homepage: Selenium automates browsers.

Selenium is the original software testing framework for automating interactions with web browsers. Only someone with programming experience will be able to write or edit test scripts in Selenium, but you will have a variety of programming languages to choose from (i.e., Ruby, Java, Python, PHP, etc.). 

Selenium is open-source, so you won’t have to worry about a rigid pricing structure. However, you will eventually need other tools in order to run multiple tests simultaneously, organize your test cases, or improve your workflow. Additionally, you may need to hire additional software engineers as your testing needs outgrow the capacity of your team.

You can read more about the advantages and disadvantages of Selenium here. 

Cypress.io 

Cypress homepage: The web has evolved. Finally, testing has too.

Cypress is a test automation tool designed for writing test scripts in JavaScript. If you already know JavaScript, this could be a good choice, but many users find there’s a steep learning curve as a beginner. Many teams end up looking for alternatives to Cypress because it limits who can manage testing and because it can be time-consuming to use, which slows down the software delivery pipeline. 

Certain aspects of Cypress are open source (such as their test script editor) but eventually, you’ll need to upgrade to one of their paid tools to run multiple tests or efficiently manage your test suite. 

Katalon

Katalon homepage: An all-in-one test automation solution

Katalon is a software testing solution built on the Selenium WebDriver. Its main offering is a record-and-playback tool which allows non-technical testers to create simple test cases without writing code. It also provides a drag-and-drop manual editor and tools for more technical users to add custom code with Selenium or JavaScript for advanced test automation scripts. 

Katalon offers desktop, web, and mobile app testing as well as API testing and some performance testing. Katalon’s pricing structure is based on the number of users who will need access to your test suite. 

You can learn more about Cypress vs. Selenium vs. Katalon here.

Tricentis Tosca

Tricentis Tosca: Intelligent test automation

Tricentis Tosca is an enterprise-focused automated testing tool, built for teams practicing agile methodologies. They support continuous testing by connecting to various continuous integration tools. Tricentis Tosca offers real-time business risk assessment so you can decide what testing is the most important at any given moment.

Finally, Tricentis Tosca offers cloud and on-premise versions of their software, but you have to contact their team for pricing details. 

Speed Up Your Software Release Cycle with Rainforest QA

With Rainforest QA, anyone can manage a full suite of tests as part of the software development process. It integrates with any CI/CD ecosystem to fit seamlessly into your delivery process. It’s a scalable, all-in-one test automation solution that’s appropriate for small teams just getting started with automated software testing or QA-mature testing teams practicing continuous delivery and regularly running 500+ software tests.

Talk to us about setting up a Rainforest plan that fits your testing needs.