After integrating data analytics platform Librato into their suite of products, SolarWinds leveraged Rainforest to scale Librato for a global audience while saving over $3MM annually on QA costs.
SolarWinds’ Librato platform is a real-time SaaS operations analytics solution that allows companies to collect, correlate, and visualize metrics to help them glean meaningful information from their data. SolarWinds’ streamlined team of developers for Librato deploys at breakneck speeds, averaging about 20-25 pushes per day. Testing the highly visual Librato platform as quickly as code is ready to ship can be challenging with traditional QA methods.
Challenge: Scaling Testing without Losing Speed
With a platform as complex as Librato, scaling testing to keep up with continuous deployment isn’t always easy. “We’re aggressive about changing things in our environment really fast,” said Matt Sanders, Director of Engineering at SolarWinds. “In order to do that, much of our process around pushing code is highly automated.” Despite the team’s efforts to automate testing wherever possible, Librato’s highly visual interface still required a large amount of manual testing.
While the limitations of testing automation haven’t slowed down SolarWinds’ aggressive deployment pace, they have made their development processes riskier. “We spent a lot of time doing manual testing, and we were also more uncomfortable about potentially breaking things. We felt that we didn’t have a good feedback loop,” Matt told us.
Eventually, SolarWinds hit a ceiling with testing automation tools. “Our product includes a lot of drawing graphs, lines and different variations of visualizations,” Matt explained, “Those features are challenging to write automated tests for that aren’t brittle.”
It became clear that automation didn’t offer the agility and flexibility needed to handle SolarWinds’ testing needs for the Librato platform. “As we scaled up, we found that we had a lot of challenges with our existing automated CI tools, both in terms of how long it took to run those tests, and the overall stability of those test suites. When a test takes 10 minutes to run and it occasionally fails on some flapping test that’s ok, but when it takes 40 minutes to run and half of the tests are flapping, then you can’t trust your suite.”
Maintaining Quality for a Diverse User Base
With on-demand testing and over 60K testers available on-demand, Rainforest made it possible for SolarWinds to break through the testing ceiling. Rainforest keeps SolarWinds moving at their breakneck deployment speed with the peace of mind that they aren’t going to break things. “For a long time it was common that when we made changes to our visualization layer, we would roll it out only for our team, wait 2 or 3 days, and then turn it on for everyone else,” Matt told us. “We tend not to do that anymore because at this point we feel like the feedback loop is fast enough that if we break something we’re going to find out pretty fast. Now that we’re using Rainforest as a safety net, it’s more acceptable to move fast.”
Rainforest provides a safety net that makes SolarWinds’ largely automated deployments run more smoothly. And with a steadily growing user base, ensuring that automated testing results are accurate is more important than ever.
Leveraging Human Testers to Stay Connected with Customer Needs
After Librato became part of SolarWinds and was adopted by a larger user base, keeping in contact with customers became more challenging. Matt says, “As we’ve grown, part of what’s changed is the level of contact we have with every customer. We have always tried to be very hands-on and helpful with our support. With a larger customer base, that dynamic has shifted -- just because we haven’t heard about it doesn’t mean that customers aren’t running into issues and churning.”
Now that the Librato platform is catering to a wider, more diverse audience, SolarWinds users are more likely to run into edge cases, such as using less popular browsers. With the ability to test multiple browsers in parallel, SolarWinds gained faster, more effective insight into how a wider set of their customers use the Librato platform. “The ability to do parallel cross-browser testing with humans as a smoke test is pretty killer,” says Matt. Rainforest has given SolarWinds the ability to better see their product from the perspective of a broad range of users, without wasting time on manual tests.
Getting Results with Rainforest QA
Matt explains that by using Rainforest, organizations like SolarWinds can save time testing and prevent downtime. “Rainforest is a slam dunk for sites that do very complex visual interactions like we do. If you have a robust organization and are doing complicated front-end work, and you’re burning a bunch of time doing manual verification, you can really save some time and benefit from the cross-browser parallel testing feature. I think that’s huge because no automated system is going to test in four or six browsers at once.”