In the Rainforest Customer Spotlight series, we highlight the superstars of the Rainforest QA platform: the Rainforest project managers that write tests, manage databases and help build better quality processes for their organizations.
For this post, we interviewed Megan Bruce, a release manager-turned-product marketing manager at Curalate who has been using Rainforest QA since May 2016. Read on to learn how Megan measures the success of Rainforest at Curalate, and her suggestions for creating the most efficient test suite from day one.
Curalate enables discovery-driven commerce at scale. Through our platform, brands can ingest media from multiple platforms and sources; tag products to that media; and distribute it to touchpoints like on- and off-site shoppable galleries, email, and blogs, helping consumers discover new products and driving traffic and revenue. It’s really exciting to be able to work with some of the world’s largest brands, like Urban Outfitters, Nordstrom, Crate & Barrel, and Sephora.
I started at Curalate as a support engineer. Being in support, I saw a lot of issues coming in that could have been caught with more robust regression testing. We had found out about Rainforest, and it seemed like a great option to be more innovative about regression testing, rather than investing in a team of manual testers.. I then stepped into a release engineering role and began managing our deployments as well as our Rainforest implementation.
I’ve now moved onto the product marketing team, which felt like a natural progression since I was handling a lot of our release and deployment processes, including writing release notes and assisting with product launch plans.
We currently deploy to our web app twice a week, and we’re working on increasing the frequency of those deployments. Before each deployment, we run our Rainforest test suite and make sure there are no issues. We prioritize testing the critical paths. We know that our clients have incredibly busy days and need to move quickly in our platform, and so we don’t want any critical-path issues slowing them down or causing a frustrating experience.
Rainforest is our primary tool for doing regression testing for our web app. We love it. In addition to Rainforest, we rely on a combo of unit tests, integration tests, and Protractor for some basic automated UI tests.
As a release engineer, I helped managed the deployment process, including deploying to a staging environment for regression testing, and then deploying the QAed build to production. I was typically able to determine within an hour of running our Rainforest test suite whether or not all tests had passed. If there were failures, I’d work with the developers to triage the issues and determine whether or not we needed to block the deployment for a fix.
In a typical week, I’d also spend time maintaining current tests, like updating tests based on any recent code changes or product launches.
I noticed a significant drop in the number of critical-path bugs reported in production - the kind of bugs that previously would have required a developer to drop everything and fix. It’s really exciting to review a test and find a failure, because it means that Rainforest caught something that would have otherwise gotten through to our clients. It’s really been awesome to use Rainforest to catch that stuff without having to use manual time for our staff to find those issues.
The tester video feature really helps a lot. It’s so helpful to be able to play back that video and see exactly what the tester was hitting. We take any issue that’s reported pretty seriously, and if there’s an edge case we want to catch it. If we see one failure and there’s some weird error that popped up, we want to dig into why that happened. Looking back at the video allows us to reproduce the issue easily when needed.
Demonstrating the cost-efficiency. The fact that I’m able to tell the team that we’ve been able to cut the regression testing time from 5-6 hours of manual testing to JIRA to measure the amount of tickets produced by Rainforest tests.
In addition to making sure that the developers understand the value of Rainforest, we’ve made an effort to involve them in the conversation about scaling Rainforest into the dev workflow. Rainforest’s CTO Russ came in to speak with one of our our lead developers about integrating Rainforest with Github and writing the tests in markdown [with Rainforest DevEx]. We were able to show our developers that Rainforest can be integrated into their workflow really easily.
I wish I had known not to get too granular with our tests. We initially spent a lot of time building out a full regression test suite and running all those tests. What we found was that a really large regression test suite was challenging for a single person to maintain and also that we were using a large amount of steps for each run, when really our priority was on ensuring our critical path tests were green. Really focusing on testing the paths that matter the most for us would have been a more efficient strategy from the start.
Focus on the critical path and getting those up and running. Make sure you’re never writing the same test twice. Break things into chunks as much as possible, use embedded tests and think about your tests in small, reusable units. That’s something we’re trying to implement now, in order to make tests easier for our developers to integrate into their workflow.
Learn more about what Megan and her team are working on by visiting the Curalate website. And don’t forget to check out our previous installment of the Rainforest Customer Spotlight series with Sammy Samperi from iOffice.