Twyla is expanding the art market and providing a new way for people to connect with artists online. While the Austin-based company doesn’t have an in-house QA team, Twyla’s VP of Engineering, Douglas Ferguson, puts a major emphasis on product quality. To ensure that quality standards are met without overextending resources, Twyla has taken some quality-first initiatives, including company-wide “Test Fests” on a weekly basis, as well as crowdsourced regression testing via Rainforest. More recently, Twyla’s team implemented Rainforest On-Premise in order to take their in-house testing activities to the next level.
Supporting a Culture of Quality
A unique manifestation of Twyla’s quality-first culture is the Test-Fest they hold each week. During this hour, every employee is encouraged to join in testing the product. “It’s a moment to socialize progress and celebrate what’s happening on the team,” says John Fitch, Twyla’s product manager. “In a way it’s our version of an internal demo day. Everyone can come and see new progress from the development team, ask questions, and get up to speed on the product.”
This team-wide collaboration on QA testing fosters sense of product ownership and gets everyone interested in UX. “Our Test-Fests help build patience and empathy. We have an organization where half of the team is very technical and driven by optimization, and the other half is very creative and driven by aesthetics. The Test-Fest process allows the two to empathize with one another,” says Douglas.
Developing Structured Testing with Rainforest On-Premise
In order to make this grassroots testing process more efficient, Twyla turned to Rainforest, which they were already using for regression testing. John mentioned that they were already seeing some of the benefits of ad hoc testing with Rainforest testers: “A side product of using Rainforest is that the Rainforest testers can give you user experience feedback via their comments and their notes. That’s valuable because they can see things in a way that a new user would. It’s also a critical part of the process to have the testers find unknown UX friction points, as it leads to moments of clarity for the Twyla team.”
John decided to implement Rainforest On-Premise to provide structure to in-house testing as well. Rainforest On-Premise provides the platform interface, VMs and structured test format used for crowdsourced regression testing, but allows Twyla’s employees to execute tests internally, rather than utilizing Rainforest’s network of 50K testers. “Putting our employees through the same step-question interface that the Rainforest testers utilize helps streamline the testing process. They can focus on identifying user experience flaws. Over time, it changes everyone’s thinking about how we build our product,” says John. “We’re training everyone to be a UX thinker at Twyla.”
Rainforest On-Premise also enables Twyla to plug test results into their engineers’ workflow more seamlessly, allowing them to get a more organized, structured view of the feedback that the Twyla employee-testers provide. John also noted that by using Rainforest to standardize their ad-hoc testing format, each member of the Twyla team feels more comfortable engaging in testing even outside of organized Test-Fests: “Sometimes people are busy and they wish they could be at Test-Fest, but they can’t. What’s great about Rainforest is that I can push test drafts to our entire company. Everyone knows and respects that if you have a free moment, anyone can knock out a few tests.”
QA Coverage without a QA Team
In addition to regular in-house testing, Twyla uses Rainforest Regression testing to provide more testing coverage. Because Twyla does not have a full-time QA team, this allows them to integrate testing into their developers’ workflow more easily: “It’s a very easy interface for the engineers to use to review issues and address them quickly,” John says, “Overall, the quality of our product has increased. It’s like having a QA person that doesn’t have to eat, drink or sleep, so you don’t feel guilty about maxing out your QA person. Therefore, we can set expectations for quality that are unrealistic for a human tester to reach.”
Douglas adds that Rainforest has helped Twyla’s testing keep pace with their daily production schedule. As a result, Douglas’ team can release code more confidently by reducing the need to run comprehensive tests in-house before every deployment: “We have Rainforest set up to test on the build, and we release to production every day. Rainforest gives us that last acceptance confirmation before we push to production. It’s nice to have that little extra feeling of confidence before the push. Our developers don’t have to test every browser every time.”