A great customer of ours, Ian Hunter, the CTO of BuzzStarter, was kind enough to share a few thoughts with me about Rainforest and what he and other CTOs actually care about. I realized after speaking with hundreds of CTOs from a sales perspective, I could also add some color to the discussion. Hopefully this is helpful in giving some more context to what Rainforest is all about.
Here’s a list of common questions we get from CTOs, and both my and Ian’s answers to them. Ian is a customer of ours, so a little less biased :)
Ian: You’re not a basic integration and you already know it. It’s definitely not as easy as dropping in a Google Analytics
tag. I think for us it took about 2.5 weeks to get up and running but that largely was due to a unique infrastructure we have setup over here.
One of the things I try to impress upon customers when they’re making the decision to integrate Rainforest is the investment required in terms of time and resources.
Potential customers often inquire ‘we want to just try it out and write a few tests and see if it works’. The product absolutely works. We have hundreds of paying customers that can attest to that. But when I first joined Rainforest, I agreed with the second point; why not just write a few quick tests and take it for a whirl? After working through a few implementations firsthand, it became clear there is a fair amount of planning (and training!) that goes into getting a test suite up and running.
That’s true of Rainforest, and it’s true of any other solution one might be considering. At Rainforest, the order of magnitude is something along the lines of: Budget 4-6 weeks before you see substantial value.
Rainforest also makes a substantial investment in customers during this time. We do so through the Customer Success team. The guys and gals on the CS team teach good test writing practices and how to technically integrate with Rainforest.
Some examples of our investments:
We work with you on all of the above, and more.
There’s lots more on our ‘Roadmap to Success’. Without going too deep into the details, the crucial point is it’s a mistake to think you’ll see immediate value.
Ian: Aaron, our Head of Product is actually responsible for writing tests on our side. It was really helpful that you were able to indoctrinate him into the art of test writing. I think there’ll always be the question around who’s the designated person responsible for these. Logically I think this is the Director/Head of Product but could be different per org.
This is such a great question and a topic of hot debate at Rainforest.
The beauty of plain English testing is anyone can contribute to testing. Customer success, Support, even Executive Assistants have been known to leverage Rainforest.
To me the real beauty of Rainforest is that Product now has the power to control quality. For example, when a CTO reaches out to discuss, she usually asks who else to bring to the party. I suggest a representative from the product team. When I was a product manager, I would usually throw requirements and tests over the wall to my 30 person QA team. After a nervous weekend I’d return to the office to find a spreadsheet filled with colors and a lot of inconsistencies. What was I supposed to do with this? How could I release this feature and sleep well at night? It was inscrutable in terms of actual verifiable results. Reproducing these bugs in the real world was a nightmare. The whole thing was just a huge pain (sound familiar?!).
Today, our customers - many of whom are Product Managers - enter similar tests in Rainforest. In fact they often start by simply importing an existing user story into a test. On average, they tend to get results within about 20 minutes. We run our 40 test regression suite across three browsers in about 20 minutes. That would take one human, sequentially testing, about 17 man-hours.
Would you rather wait 2 days or 20 minutes to deploy?
Product folks should be stoked about the opportunity inherent in Rainforest. It’s more power to control your destiny. That doesn’t obviate the need for skilled in-house testers, but it should make QA as a whole much more collaborative.
With Rainforest rigorously testing for cross-browser functional regressions, in-house testers are freed up for more interesting and context sensitive work. That is, exploratory testing.
Ian: We have some rather complicated interfaces, some single-page apps in particular, and we wanted to ship fast which usually means lots of prod bugs. It got so out of hand that we even had a hard time getting our admins to use some of the admin features. I looked at a QA engineer, which in SF would be like $125k-$140k, you guys were less expensive and much faster / concurrent.
Great points on both the utility and the comparative cost from Ian. A few other notes I’ve heard from customers are:
Ian: Accuracy will always be a question that engineering managers will ask. I think the thing that wasn’t really pointed out during our demo is the fact that you run up to 3 testers through the individual test and use a weighted determination to rule out false positives. This is a very good thing to make note of. Also, it’s nice when I’m able to tell the story of early on I noted a failure but considered it a false positive and shipped when in fact Rainforest caught a feature regression. From now on I take far more note of failing tests.
Accuracy is an interesting question. If I’m reading between the lines, the question is: Does our crowd of 50k testers have 100% accurate results all the time? Um, no. We certify, train, and continually assess our crowd of testers. That propels us quite a bit forward, towards the ideal of 100% accuracy.
However, fundamentally, it’s important to remember that testers are still human. And humans are susceptible to error.
You might be interested to learn Rainforest employs two PHDs in Mathematics. They develop and deploy the algorithms we use to manage tester quality. So we have algorithms that help monitor these humans and tracks them over time to ensure the quality bar is high. We mix these sophisticated algorithms with more basic ones, like the quorum algorithm.
That’s a summary of the common questions we get about Rainforest in a nutshell. Our goal was to show you a real world scenario of how we work, without (most of!) the usual marketing spiel. If you you think we missed something important here, please let me know!
Special thanks to Ian and BuzzStarter for sharing their story.