America's Test Kitchen is a cooking media and publishing company that includes 2 TV shows, 5 websites and 2 magazines and multiple books published each year. With a small, fast-moving engineering team, ATK must make sure that they keep testing as speedy and efficient as possible. But their initial goal of minimizing manual QA and automating as much testing as possible was thwarted by their need to move faster. Business demands outstripped their bandwidth for implementing automation alongside new features, and they frequently found themselves falling back on manual QA practices to hit their deadlines.
Welling LaGrone, Director of Software Engineering, set out to explore options for crowdsourced testing to supplement his in-house team and improve their QA strategy. He found that Rainforest fit their needs the best. “The Rainforest model was much more in line with a process that we felt we could support. We can have our own QA lead orchestrate the test case development and implementation, and essentially farm out the execution of those tests,” Welling says. America’s Test Kitchen currently uses Rainforest for two of their five websites, with plans to extend it to cover all of their existing sites and services in the future.
Driving a Strategic Approach to Quality Assurance
One of the major concerns for America’s Test Kitchen was finding a scalable but affordable way to do QA. One of their primary reasons for turning to Rainforest was because they wanted to avoid building out a large, costly QA team. “Rainforest offers us a cost-efficient way to offload [the] commodity aspect of QA and focus on the strategic parts of what we need to do to ensure a quality product,” Welling told us. With Rainforest, America’s Test Kitchen was able to increase test coverage without taking developers away from building new features.
As a result, Rainforest has allowed America’s Test Kitchen to increase test coverage and product quality without increasing their in-house QA resources. This frees up their developers and QA engineers for other, more strategic activities. “Offloading repetitive testing tasks allows us to invest in higher caliber QA team members and focus our QA capacity on the strategic QA challenges, rather than the mechanics of testing,” says Welling.
Building Test Coverage from the Command Line with Rainforest DevEx
Rainforest has allowed America’s Test Kitchen to execute regression testing much more effectively. Working towards continuous deployment is an important part of America’s Test Kitchen’s overall development strategy, and Rainforest offers the ability to write tests once, then run them on-demand. Tests that were previously executed manually can now be run automatically via Rainforest.
Because ATK’s engineers help drive their push for increasing test coverage, the ability to write and run tests from the command line using Rainforest’s DevEx and CLI tools was a major draw. “We love that we can drive Rainforest from the command line and control the scope of the tests and cherry pick the browsers we want to run tests on,” Welling told us. With Rainforest, engineers can now write tests for new features without leaving their workflow, helping America’s Test Kitchen to develop and test more rapidly.
Improving Product Quality and Processes
Overall, Rainforest has helped America’s Test Kitchen streamline their QA processes so they can focus on driving quality forward: “Having our tests run regularly and continuing to fine tune them has let us understand our tests better. We’ve been able to catch and clean up some functional processes.”
In addition to improved testing processes, leveraging crowdsourced testing has also had an impact on America’s Test Kitchen’s product quality: “We’ve seen an uptick in quality with Rainforest. It has essentially protected us from introducing bad stuff,” says Welling.