Everything a CTO Should Know About Rainforest title image

Everything a CTO Should Know About Rainforest

A great customer of ours, Ian Hunter, the CTO of BuzzStarter, was kind enough to share a few thoughts with me about Rainforest and what he and other CTOs actually care about. I realized after speaking with hundreds of CTOs from a sales perspective, I could also add some color to the discussion. Hopefully this is helpful in giving some more context to what Rainforest is all about.

Here’s a list of common questions we get from CTOs, and both my and Ian’s answers to them. Ian is a customer of ours, so a little less biased :)

Common Question 1: How hard is it to integrate?

Ian: You’re not a basic integration and you already know it. It’s definitely not as easy as dropping in a Google Analytics tag. I think for us it took about 2.5 weeks to get up and running but that largely was due to a unique infrastructure we have setup over here.

One of the things I try to impress upon customers when they’re making the decision to integrate Rainforest is the investment required in terms of time and resources.

Potential customers often inquire ‘we want to just try it out and write a few tests and see if it works’. The product absolutely works. We have hundreds of paying customers that can attest to that. But when I first joined Rainforest, I agreed with the second point; why not just write a few quick tests and take it for a whirl? After working through a few implementations firsthand, it became clear there is a fair amount of planning (and training!) that goes into getting a test suite up and running.

That’s true of Rainforest, and it’s true of any other solution one might be considering. At Rainforest, the order of magnitude is something along the lines of: Budget 4-6 weeks before you see substantial value.

Rainforest also makes a substantial investment in customers during this time. We do so through the Customer Success team. The guys and gals on the CS team teach good test writing practices and how to technically integrate with Rainforest.

Some examples of our investments:

  1. To make Rainforest fast, we send testers in parallel. Naturally, that means they’ll need more than one account for testing. Should some accounts be provisioned as admins? Should some accounts have an associated activity history? This is called ‘state set-up’ – and we help you find and deploy the right tool based on your infrastructure.
  2. We care about business results. That is, your business results. A large chunk of our process is dedicated to understanding deployment cadences, how Rainforest can speed those up, time savings, cost savings, resource savings, etc. This allows us to keep our goals aligned with yours.
  3. We like training by pairing. You’d think because Rainforest is written in plain English it’s easy to write tests. You’d think that, but consider: Out of the 500+ possible tests for your product, which are worth working on first? What’s the best way to filter for internal jargon and confusing technical terms? How do you set the tests up so they’re modular and adhere to the DRY principle?

We work with you on all of the above, and more.

There’s lots more on our ‘Roadmap to Success’. Without going too deep into the details, the crucial point is it’s a mistake to think you’ll see immediate value.

Common Question 2: Do I have to do everything or what kind of non-technical people can use the product?

Ian: Aaron, our Head of Product is actually responsible for writing tests on our side. It was really helpful that you were able to indoctrinate him into the art of test writing. I think there’ll always be the question around who’s the designated person responsible for these. Logically I think this is the Director/Head of Product but could be different per org.

This is such a great question and a topic of hot debate at Rainforest.

The beauty of plain English testing is anyone can contribute to testing. Customer success, Support, even Executive Assistants have been known to leverage Rainforest.

To me the real beauty of Rainforest is that Product now has the power to control quality. For example, when a CTO reaches out to discuss, she usually asks who else to bring to the party. I suggest a representative from the product team. When I was a product manager, I would usually throw requirements and tests over the wall to my 30 person QA team. After a nervous weekend I’d return to the office to find a spreadsheet filled with colors and a lot of inconsistencies. What was I supposed to do with this? How could I release this feature and sleep well at night? It was inscrutable in terms of actual verifiable results. Reproducing these bugs in the real world was a nightmare. The whole thing was just a huge pain (sound familiar?!).

Today, our customers - many of whom are Product Managers - enter similar tests in Rainforest. In fact they often start by simply importing an existing user story into a test. On average, they tend to get results within about 20 minutes. We run our 40 test regression suite across three browsers in about 20 minutes. That would take one human, sequentially testing, about 17 man-hours.

Would you rather wait 2 days or 20 minutes to deploy?

Product folks should be stoked about the opportunity inherent in Rainforest. It’s more power to control your destiny. That doesn’t obviate the need for skilled in-house testers, but it should make QA as a whole much more collaborative.

With Rainforest rigorously testing for cross-browser functional regressions, in-house testers are freed up for more interesting and context sensitive work. That is, exploratory testing.

Common Question 3: What problems is Rainforest actually solving?

Ian: We have some rather complicated interfaces, some single-page apps in particular, and we wanted to ship fast which usually means lots of prod bugs. It got so out of hand that we even had a hard time getting our admins to use some of the admin features. I looked at a QA engineer, which in SF would be like $125k-$140k, you guys were less expensive and much faster / concurrent.

Great points on both the utility and the comparative cost from Ian. A few other notes I’ve heard from customers are:

  • Deploy faster. Not by minutes or hours but by days and weeks.
  • Delay the first hire, and avoid the bad ones altogether. Rainforest will leverage true QA talent. When you get around to hiring a great QA analyst, we’ll help them scale. Avoid talent dilution caused by mediocre hires.
  • Test continuously. Continuous Delivery is the dream most teams would like to work towards. Fast + maintainable releases need fast + maintainable tests. Testing with Rainforest ensures testing can keep up with the rest of your org.

Common Question 4: How accurate is it? Does it work?

Ian: Accuracy will always be a question that engineering managers will ask. I think the thing that wasn’t really pointed out during our demo is the fact that you run up to 3 testers through the individual test and use a weighted determination to rule out false positives. This is a very good thing to make note of. Also, it’s nice when I’m able to tell the story of early on I noted a failure but considered it a false positive and shipped when in fact Rainforest caught a feature regression. From now on I take far more note of failing tests.

Accuracy is an interesting question. If I’m reading between the lines, the question is: Does our crowd of 50k testers have 100% accurate results all the time? Um, no. We certify, train, and continually assess our crowd of testers. That propels us quite a bit forward, towards the ideal of 100% accuracy.

However, fundamentally, it’s important to remember that testers are still human. And humans are susceptible to error.

You might be interested to learn Rainforest employs two PHDs in Mathematics. They develop and deploy the algorithms we use to manage tester quality. So we have algorithms that help monitor these humans and tracks them over time to ensure the quality bar is high. We mix these sophisticated algorithms with more basic ones, like the quorum algorithm.

Wrapping up

That’s a summary of the common questions we get about Rainforest in a nutshell. Our goal was to show you a real world scenario of how we work, without (most of!) the usual marketing spiel. If you you think we missed something important here, please let me know!

Special thanks to Ian and BuzzStarter for sharing their story.