Librato Ensures Comprehensive QA Coverage for Highly-Visual Monitoring and Analytics Platform
Librato is a real-time SaaS operations analytics solution that allows companies to collect, correlate, and visualize metrics to help them glean meaningful information from their data. Librato’s streamlined team of developers deploys at breakneck speeds, averaging about 20-25 pushes per day. Testing Librato’s highly visual platform as quickly as code is ready to ship can be challenging with traditional QA methods, especially because Librato doesn’t have an in-house QA team.
Scaling Testing for Continuous Deployment
With a platform as complex as Librato, scaling testing to keep up with continuous deployment isn’t always easy. “We’re aggressive about changing things in our environment really fast,” said Matt Sanders, Director of Engineering at Librato. “In order to do that, much of our process around pushing code is highly automated.” Despite the team’s efforts to automate testing wherever possible, Librato’s highly visual interface still required a large amount of manual testing.
While the limitations of testing automation haven’t slowed down Librato’s aggressive deployment pace, they have made their development processes riskier. “We spent a lot of time doing manual testing, and we were also more uncomfortable about potentially breaking things. We felt that we didn’t have a good feedback loop,” Matt told us.
But eventually, Librato hit a ceiling with testing automation tools. “Our product includes a lot of drawing graphs, lines and different variations of visualizations,” Matt explained, “Those features are challenging to write automated tests for that aren’t brittle.”
It became clear that automation didn’t offer the agility and flexibility needed to handle Librato’s testing needs. “As we scaled up, we found that we had a lot of challenges with our existing automated CI tools, both in terms of how long it took to run those tests, and the overall stability of those test suites. When a test takes 10 minutes to run and it occasionally fails on some flapping test that’s ok, but when it takes 40 minutes to run and half of the tests are flapping, then you can’t trust your suite.”
Improve Product Quality with Tester Feedback
With on-demand testing and over 50K testers available on-demand, Rainforest made it possible for Librato to break through the testing ceiling. Rainforest keeps Librato moving at their breakneck deployment speed with the peace of mind that they aren’t going to break things.
“For a long time it was common that when we made changes to our visualization layer, we would roll it out only for our team, wait 2 or 3 days, and then turn it on for everyone else,” Matt told us. “We tend not to do that anymore because at this point we feel like the feedback loop is fast enough that if we break something we’re going to find out pretty fast. Now that we’re using Rainforest as a safety net, it’s more acceptable to move fast.”
Rainforest provides a safety net that makes Librato’s largely automated deployments run more smoothly. And with a steadily growing user base, ensuring that automated testing results are accurate is more important than ever for Librato.
Leveraging Human Testers to Stay Connected with Customer Needs
As Librato grew, keeping in contact with customers became more challenging. Matt says, “As we’ve grown, part of what’s changed is the level of contact we have with every customer. We have always tried to be very hands-on and helpful with our support. With a larger customer base, that dynamic has shifted -- just because we haven’t heard about it doesn’t mean that customers aren’t running into issues and churning.”
Now that Librato is catering to a wider, more diverse audience, their users are more likely to run into edge cases, such as using less popular browsers. With the ability to test multiple browsers in parallel, Librato gained faster, more effective insight into how a wider set of their customers use the platform. “The ability to do parallel cross-browser testing with humans as a smoke test is pretty killer,” says Matt. Rainforest has given Librato the ability to better see their product from the perspective of a broad range of users, without wasting time on manual tests.
Getting Results with Rainforest QA
Matt explains that by using Rainforest, organizations can save time testing and prevent downtime. “Rainforest is a slam dunk for sites that do very complex visual interactions like we do. If you have a robust organization and are doing complicated front-end work, and you’re burning a bunch of time doing manual verification, you can really save some time and benefit from the cross-browser parallel testing feature. I think that’s huge because no automated system is going to test in four or six browsers at once.”