Today, we’re launching our no-code QA platform, making software test automation accessible to any product contributor in any company.
Rainforest QA started in 2012 as a crowdsourced testing platform — QA specialists from our worldwide community would follow plain English instructions to run customers’ test cases.
After two years of development, we’ve now added a proprietary, no-code automation service to the platform, including a visual test editor anyone can use to create, update, and run complex, automated test cases without knowing any code.
This launch marks a major milestone in our mission to make QA accessible to everyone responsible for product quality and user experience. And it reflects the fact that modern QA means optimizing not only for quality, but also for speed of execution. Specifically, the speed required by automated software development pipelines and the companies that use them to stay competitive.
Read on to learn:
- Why QA matters, and how it’s been creating unhappy tradeoffs in modern software development processes.
- How many product development teams have been excluded from the benefits of test automation.
- More details on exactly what we’re announcing today, including how it addresses the obstacles development teams have been facing.
The QA Challenge
I should start by thanking Paul Graham (or “PG”, as we came to know him), the legendary founder of Y Combinator.
In 2012, after my cofounder, Russ, and I had spent a month at Y Combinator, trying and failing to ‘make something people want’, PG offered some pointed feedback that completely changed the trajectory of our fledgling company.
With characteristic directness, he told us our idea was “dumb”, suggested that we probably wouldn’t be passionate about it for the long haul, and recommended that we stop working on it.
In trying to prove him wrong about our idea (which was completely unrelated to what we’re doing today), we realized pretty quickly: he was right.
So, with a month to go before Demo Day, we found ourselves without an idea. Having built many prototypes that people weren’t interested in, we emailed everyone else in our YC batch, asking if there was any software development problem they’d pay us at least one thousand dollars a month to solve for them.
The most consistent response? Quality assurance (QA).
In software product development, QA is necessary. It’s what keeps bugs and other oversights from hurting the brand, from eroding revenue, and even from hurting customers.
The only question tech companies face isn’t whether or not to do QA, it’s how.
That’s why the software testing market is huge: it’s a >$45B market (projected to grow to $110B by 2027), and accounts for around one-fourth of enterprise IT budgets, on average.
But small companies like the ones in our YC class typically can’t afford to hire extra headcount just to do QA. And the time their (expensive) developers and product managers spend testing the product is time not spent working on new code.
That’s why, for nine of the companies in our YC cohort, it was worth one-thousand dollars a month for Russ and I to do their QA for them.
Of course, we couldn’t build a scalable business based on two guys manually going through test scripts.
Around that time, we found the solution when another founder introduced us to Amazon Mechanical Turk: Leveraging a huge network of globally-distributed workers seemed like an ideal way to execute test scripts, at scale.
We seemed to have found our not dumb business idea: “crowd testing”.
The Rise of Continuous Integration / Continuous Delivery (CI/CD)
Not only did our YC classmates point us to our business idea, they also pointed the way to the future of software development and, by proxy, the future of QA.
Something all of the companies in our cohort had in common was that they released code quickly and frequently — often many times a day. They were on the leading edge of a trend most of us in tech see everywhere today: releasing software product updates at a high velocity as a competitive advantage.
Specifically, they had adopted continuous integration and continuous delivery (CI/CD) pipelines in their development processes. In short, CI/CD is a methodology that automates various stages of software product development so that teams can release new code to customers more rapidly, efficiently, and frequently.
We were certain that CI/CD would be a key part of the increasingly-agile future of software development.
But QA was (and is) still a labor-intensive, manual process. In an automated pipeline, manual processes become the bottleneck. For CI/CD pipelines, QA is often the bottleneck. Startups are faced with a frustrating dilemma: slow down your team and your release process to do QA, or move fast and break things.
Widespread, successful adoption of CI/CD would require a step-change in software testing:
- Ease of use. Anyone on the team responsible for quality needs to be able to create and update tests. Otherwise, those stakeholders are left depending on specialized QA teams, which are frequently time-strapped and backlogged.
- Affordability. If you’re deploying code several times a day and running QA tests for every release, costs can add up, fast. Testing has to be affordable enough to execute with every release.
- Speed. Testing has to be fast enough to support an automated pipeline. Developers can’t feel like QA is a bottleneck, otherwise they might skip it in favor of faster releases.
- Quality of results. Being affordable and fast is meaningless unless test results are reliable and trustworthy.
Rainforest QA, Version 1
Given this vision of the future, we operated Rainforest QA for several years around addressing a single question:
Can we build an easy-to-use system in which crowd testers return high-quality test results quickly and affordably enough for CI/CD?
Easy?
Our customers could write test instructions for our tester community in plain English. We heard from many people on product teams saying, “Wow, this is great, I can use it without going to the QA team!”
Of course, we also discovered that open-ended English language instructions can be unhelpfully vague. If you tell me, “Visit the park”, I can think of 100 different ways to get there, including different routes, modes of transportation, speeds, and more. The best test scripts are detailed and specific; they spell out exactly what should happen at each step.
High-quality?
Can crowd testers consistently deliver reliable test results? When you’ve got the right people with the right training and systems in place, the answer is yes.
We specifically developed three systems to ensure consistently high-quality results from our community of testers (who we initially sourced from Mechanical Turk, and subsequently contracted with, directly):
- Training. We implemented automated training to verify English language comprehension and writing, ability to follow nuanced instructions, and more.
- Reproducibility. Every test is executed by at least two testers to confirm a consensus in the results. Our quality algorithms add more testers as necessary in real-time.
- Reputation scores. Every tester at Rainforest has an algorithmically-based reputation score that is constantly adjusted based on their work. Factors include the consistency of their test results with high- and low-reputation testers, their response time to new tasks, their training performance, and more.
Fast and affordable (enough)?
We optimized crowd testing to be as fast as it could possibly be.
First, we recruited thousands of QA specialists worldwide, so that any test could be run on-demand, 24×7. There’s effectively no lag, waiting for someone to run your tests.
Further, for each set of test cases someone submits to Rainforest, we run the test cases in parallel. That means executing that set of tests only takes as long as the longest single test case in the set.
These days, for sets of tests run against our tester community, results come back in 17 minutes, on average.
That’s incredibly fast for manual testing, and has been wildly popular with customers whose internal teams were spending hours, days, or even weeks running tests manually. Those customers love that they can expand their test coverage and get speedy test results in an affordable way, without adding headcount.
But, the fact is, for many rote tasks, humans will never be as fast (or as affordable) as the processing power of computers.
Critically, we found that human-powered crowd testing, in most cases, would never be fast and affordable enough for CI/CD pipelines.
Even for companies who haven’t adopted automated development pipelines, speeding up the release of product updates is among their most important priorities.
Inaccessible Test Automation Solutions
The logical conclusion is that teams should simply automate as much testing as possible.
But test automation has been largely inaccessible to many of the product teams who need it.
The skills barrier
Historically, you’ve needed to hire a coder (for example, a Quality Engineer) to write and maintain your test cases using a test automation framework like Selenium. That precludes any product team without specialized QA folks from taking advantage of automation.
Even having a QA Engineer on the team doesn’t solve the accessibility problem. There are roles on the team — like Product Managers and UX Designers — who’re responsible for the quality of the product experience, likely don’t know how to code, and therefore have no immediate control over what gets tested and how.
For the developers who do know how to code, their time may be better spent building software, not test suites.
The cost barrier
Open source test automation frameworks like Selenium are “free”, but they come with the not-so-hidden costs required to use them:
- Testing infrastructure. You need to procure machines you’ll run tests on, either locally or via a 3rd-party service like BrowserStack.
- Headcount. Ziprecruiter pegs the current average salary for Selenium developers in the U.S. at $125K.
Automation in the realm of QA hasn’t been living up to its promise as a cost-effective, affordable solution.
We’ve seen the effects of these cost barriers manifest in the adoption rates of CI/CD. Even as it’s gained wider adoption, there’s an adoption gap between smaller and larger companies.
Relative to larger companies, smaller companies can’t afford to add specialized engineering headcount for functional testing, so they keep manually running regression tests — tests to confirm important product flows are working — before every release.
And that’s if they’re lucky enough to have a person dedicated to manually running these QA test scripts. In many cases, “QA” is limited to product managers and other people on the team passionate about quality running ad hoc tests after each release, and emailing or Slacking the developers with any issues they find.
Either way, there’s a painful tradeoff for the business between the speed of releasing code and the thoroughness of testing that code.
If you can’t afford to automate a meaningful portion of your QA testing, you can’t fully automate your development pipeline and release code at the velocity needed to compete. QA becomes the bottleneck.
Rainforest QA, Version 2
Two years ago, we made a big decision to pivot the company around these findings.
Now we’re finally ready to share how we’ve addressed each one of these trends, challenges, and needs in software testing.
Today, we’re making software test automation truly accessible to all companies and product contributors with a novel no-code approach.
True no-code test automation that anyone can use
Automation accessible to anyone on the product team
Creating or updating a test step in Rainforest’s visual editor is as easy as selecting an action (like click, fill, or scroll) then click-and-dragging your mouse to highlight the target of the action. Everything is human-readable.
Now anyone on the team can write, maintain, and run tests and Rainforest automation will return detailed results (including video recordings of every test) in just minutes — less than four minutes, on average.
The flexibility to test any scenario
Simple, straightforward functionality doesn’t mean limited features.
We’ve built Rainforest to be incredibly flexible so it can accommodate all sorts of testing scenarios:
- Complete forms with randomly-generated names, emails, passwords, and more.
- Validate emails with virtual inboxes.
- Check login flows that include two-factor authentication.
…and so much more.
Tests that reflect the actual user experience
Our proprietary automation service tests the visual layer of your application or website, to give you the confidence you’re testing what your users and customers will experience.
Compare this to the popular, open source framework Selenium, which tests the code behind the visual layer. Minor, behind-the-scenes code changes that don’t affect the UI can break Selenium tests (i.e., create false positives), but won’t break Rainforest tests.
Integrating additional ‘senses’, like understanding the text and the HTTP requests, gives us a foundation to continue decreasing brittleness over time.
CI/CD integrations for rapid, high-quality product delivery
Are you a developer? Integrate Rainforest tests into your CI/CD pipeline via our API, CLI, or CircleCI integration and enjoy fully-automated continuous development.
With fully-automated CI/CD pipelines, product teams can finally deliver on both quality and speed.
A world-class QA team, on demand
Not all tests can, or should, be automated. Computers are nowhere near able to replace the ingenuity and judgement of people.
That’s why our worldwide community of QA pros — who have each been with us at least five years — isn’t going anywhere. Nor are the processes and technologies we’ve innovated to ensure the quality and speed of their test results.
Note: The tester community is only available through our paid plans.
For running test scripts
With the click of a button, you can choose to run any test in Rainforest on our proprietary automation service or with specialists from our tester community. Rainforest is the only QA platform that provides on-demand access to both no-code automated testing and manual testing by QA specialists.
Our most successful customers use a combination of manual (crowd) and automated testing, applying each to the appropriate use-cases:
- Automation is perfect for rote test scripts like regression tests that run before every release.
- The tester community is your best bet for executing test scripts that could benefit from subjective feedback (“The images on the page are a bit blurry.”).
No matter which method you use, all tests run on Rainforest run in parallel, giving you fast-as-possible results.
For exploratory testing and test creation & maintenance
Beyond executing scripted test cases, there are plenty of other time-consuming tasks needed to do QA well. Tasks that can’t be automated.
For example, you can engage QA specialists from our tester community to run unscripted, exploratory tests to uncover gnarly bugs and edge cases in your major releases or in fast-changing areas of your app.
Further, even though we’ve made test creation and maintenance as simple as possible, those are still tasks that require someone’s time and attention — time and attention that many teams don’t have to spare.
That’s why we created a test writing service that offloads test writing and maintenance from your burdens and puts them in the hands of our QA experts. They’ll deliver passing tests optimized to run on Rainforest.
The Next Version of Rainforest QA
Our mission — to make QA accessible to the people responsible for product quality — means we’re on a continuing path to making our solution even more easy, fast, and powerful.
Talk to us about setting up a Rainforest plan that fits your testing needs.
