To speed up their deployment cycles and increase confidence in releases, Exaptive implemented Rainforest QA to create a scalable, reliable testing process that keeps development resources focused on new features, not fixing bugs.
Exaptive is a data application studio and marketplace for applications designed to help teams gain stronger insights into their data. With a small team of developers and no full-time QA resources in house, Exaptive’s engineering team needed confidence that every release was up to par, as any issue in production required pulling devs off of other projects to fix them. Exaptive’s Engineering Director, Cory White, turned to Rainforest QA for a scalable approach to software testing that would give his team the assurance they needed.
Reducing Friction and Stress Around Releases
Before using Rainforest, Exaptive relied on a junior developer to perform tests before every release. “The process involved last-minute, manual end-to-end testing by one guy,” says Cory White, Engineering Director. “There was a checklist of smoke tests to go through, and a few particular regressions. It took a long time to do it. We were the typical case of ‘release a bunch of stuff, break a bunch of stuff, rinse and repeat.’”
This ad-hoc testing approach resulted in missed bugs, longer QA cycles and higher stress for the team. “There was a lot of friction built up around every release. That made the list of things that went into a release longer, which made every one more risky. It wasn’t sustainable or scalable,” says Cory.
A New QA Process for Speed and Repeatability
The Exaptive team has seen an impact in the quality of their product and process since implementing the on-demand QA platform.“ The overall effect of Rainforest has been a material uptick in quality. Even the process of writing tests helps us catch bugs,” says Cory. “Anybody on the dev team can sit down and write down a test that works well for us. The opportunities for scalability are great. I’ve been impressed by how consistent the testing was.”
When they first implemented Rainforest, Cory’s team put Rainforest to the test, running it head-to-head with their manual testing process to compare results: “For the first few releases we ran Rainforest alongside our manual QA process and got the same results in less time. Rainforest is more reliable than what we were doing before.” With Rainforest, Exaptive’s team can see the results of their entire test suite in about 30 minutes, without pulling their QA off of other projects to execute tests. “Now I’m able to kick off our whole smoke suite 10 times in a single day. I never, ever could have done that with one guy. The scalability of Rainforest is huge.”
Now that the engineering team has seen the difference Rainforest makes to their testing process, they’re eager to incorporate Rainforest fully into their development workflow. “Everyone has bought into the efficacy of it, and now they’re seeing how easy it is to add to the test suite, so they’re jumping in to do it.”
Minimizing Risk for Every Release
Since incorporating Rainforest into their QA process, Exaptive’s dev team has been able to increase the frequency of their releases and reduce the time required to prepare for releases. “The biggest impact of Rainforest was reducing our lead time for release by an order of magnitude. We were releasing typically every few weeks to a month with a ton of risk. Now we’re down to releasing every week. Our development to production time has shrunk dramatically,” says Cory.
Rainforest test results provide the context Cory’s team needs to fix issues quickly. Every test run provides the context needed to pinpoint the issue and replicate it. “One of the best things about Rainforest is when a test fails we can go in and see the click trail and the videos -- it fast forwards you to the point at which the test failed. Within seconds we can see what went wrong,” Cory explained.
This speed has increased the organization’s confidence in the product and the team. “We’ve seen an increase in confidence internally. When someone finds a bug internally, they’re confident that they’ll see it fixed quickly. Everyone has a lot less stress about releases.”