Back to all case studies
The situation: Push Security's software team struggled with end-to-end testing. Since manual testing was too time-consuming and unreliable, they adopted a no-code tool to transition to automation — but staying on top of test maintenance was unmanageable for their developers.
The result: Rainforest test managers joined Push Security’s workflows and took over test creation and maintenance, allowing Push’s developers to focus on releasing features. Tyrone now has a sense of trust in the testing process, especially after Rainforest caught a number of critical bugs.
I recommend Rainforest to startups blindsided by bugs in production, which is probably everyone. But particularly for people who run more modern microservices-type companies, especially with the CI/CD mindset where everyone's pushing to production multiple times a day.
Push Security is an identity security platform for cloud-first businesses. It uses a lightweight browser extension to give you real-time visibility of all your employees’ cloud identities and the vulnerabilities putting your business at risk.
Their software team had solid unit and integration test coverage, but struggled with end-to-end (e2e) testing, especially given the complexity of their architecture. In isolation, their many subsystems worked, but they kept finding bugs with the interactions between these systems.
Like many startups, they started with manually-executed e2e tests, but that was time-consuming and unreliable. Running the manual tests was "mindless and error-prone," which indicated it was time for automation. Tyrone decided to adopt a no-code tool that’d make it easy for his team to automate their e2e test suite.
But it turned out to be difficult for developers on the team to stay on top of test maintenance, even with a simple-to-use, no-code tool.
"The maintenance burden started hitting us. And we realized it was because there's so many moving parts in our system that if something breaks, it could have been one of five people that touched it this week. So, who should investigate it? I realized if you don't make it someone’s responsibility, it becomes no one's responsibility."
Meanwhile, developers consistently pushed feature updates, which only exacerbated the maintenance problem: feature changes would, as expected, break tests that were looking for a now-obsolete version of the app. Those broken tests would result in test failures that required investigation and subsequent updating of the affected tests.
"The really painful part was triaging. Why did this error occur? What changed? Most of the time, it was expected, minor feature tweaks causing the test failures — it wasn't bugs being found. We found that really painful."
Without staying on top of maintenance, the test suite was becoming outdated, which meant the team was losing its confidence in the test suite’s ability to catch actual bugs and issues.
This unsustainable situation with developers on the team owning test maintenance prompted Tyrone to start thinking about hiring someone to own these painful QA automation tasks.
But spinning up an entirely new discipline within the team wasn’t an appealing proposition.
"I was thinking, ok, I need to hire someone in a QA position. I've done that before and it's really painful to run an engineering team and a QA team as well. They’re very different disciplines in my opinion, even though they're very tightly related. Starting a new team with a new discipline and focus and training is difficult at this stage in a startup, when everything is about racing to market and land-grabbing."
Plus, hiring expensive full-time employees in non-core disciplines didn’t align with Push Security’s hiring philosophy. They only hire in roles that are "absolutely core" to the business, and outsource everything else. This allows them to focus on the core differentiators that'll help them win the market.
So, he briefly explored outsourcing QA, but none of the QA outsourcing firms he spoke with inspired any confidence "that they knew what they were talking about."
Then he spoke with Rainforest about an alternative to hiring and conventional QA outsourcing.
From the start, he felt more confident about the Rainforest team’s insights and expertise.
"This team has been around the block. They understand what I'm saying. It was just a good conversation. I actually learned some things in that conversation. That's how you know that it's the start of something good."
He could see Rainforest making an impact quickly, just like an experienced hire could.
"That's one of the biggest reasons I ended up going with Rainforest, is because I could see there's a path here to this team coming in and hitting the ground running."
Push Security quickly integrated a Rainforest strategist and test managers into its workflows. They set up a shared Slack channel for the test managers and treat them like "an extension of them team."
The test managers are responsive throughout the workday, and feel like part of the team.
"They respond seemingly at all hours of the day and night. They're there in the morning and there at night posting messages. It's dark magic.
They're really polite, cool people. It's a very similar style to people internally."
The Rainforest test managers have completely taken over test creation and maintenance and triaging of test errors, giving the software team time back to focus on shipping features.
Tyrone’s team simply provides overview videos showing where test coverage is needed for new features.
"The flow has been really great. They're really proactive, they triage all the errors and things that happen in tests, they fix up the tests, they write new tests, and they ask crazy-insightful questions on the product."
In fact, test managers have inspired so much trust that the Push Security software team has stopped paying attention to test failures altogether. They count on the test managers to highlight any suspected issues with the Push Security app, along with the details necessary to make issues "really easy to debug."
The new process isn’t just giving the software team time back to stay on top of new features. It's also been a “game-changer” for improving confidence in the quality of Push Security’s product.
"We're catching way more stuff before it gets to production. It's been a game-changer for our general quality initiative. In the last few months Rainforest has saved our ass I don't even know how many times. Like literally before things get into production, the day before. Critical bugs being stopped from going into production."
And people on Tyrone’s team are seeing the difference. He's had "people from around the team saying, ‘Rainforest just caught another thing. Damn, they’re good.’"
"Rainforest has become a safe pair of hands for us. Core flows don't break, which is amazing and important. It's exceeded my expectations."
