Back to all case studies
The situation: PayHOA’s five developers had been shipping with urgency and expanding the team — but they started breaking things multiple times a day, without a reliable testing process to catch the issues. Dale, CTO, needed an efficient and scalable way to handle QA, but hiring a full-time QA person was an expensive and ultimately unsuccessful process.
The result: Dale and team spent minimal effort onboarding Rainforest Test Managers, who have taken over automated test creation, test maintenance, and review of test failures. Now Dale’s team can focus on shipping fast, confident that if something breaks, they’ll get an alert in Slack. Rainforest allowed PayHOA to complete what would have been a month-long platform upgrade in just four days.
We’ve quickly built up our test coverage and Rainforest gives me a lot more confidence in pushing regularly without scrutinizing every change. The Test Managers even find things in the existing code. They’re excellent!
PayHOA’s software for homeowners associations handles payments, maintenance requests, accounting, and all the SMS and email communications in between.
PayHOA had hit a big milestone — profitability — but the “ship-at-all-cost” habits that got them there would have to change. As Dale Smith, CTO, explains it:
“We’d been going full throttle developing new features, chasing customers, and chasing revenue. When we started to expand our dev team, we hit this tipping point where all that pushing meant we were breaking multiple things a day. We weren’t slowing down to test.”
Dale knew that if they didn’t fix the QA problem, adding developer headcount might make things worse. More people and more commits could mean more bugs and more time-consuming hotfixes.
All the new issues introduced so much risk that it made Dale reluctant to ship the team’s ambitious features and upgrades.
“We didn’t know what was safe to push. We had a lot of big stuff that we wanted to do, but I was hesitant to make moves on a lot of it because I didn’t know what else was going to break.”
But in a small, budget-constrained startup, it was tough to make a case for hiring anyone other than a developer.
"It's hard for me to convince leadership that we should spend money on somebody who doesn't deliver code — even though we all acknowledge the value of not letting bugs into production."
Dale successfully made the case for hiring. He and his team didn’t know how long it’d take, but committing time to the recruiting and hiring process on top of their work felt like a worthwhile investment. Without QA and without feeling confident about each push, they couldn’t scale.
That said, no one on Dale’s team had deep experience in QA. Some had written automated tests before with Selenium, but they didn’t know the ideal QA process or the right skills to look for.
“We interviewed three or four people and selected one. It took a month and a half, from the first interview to their first day. Finding applicants wasn’t hard — we received hundreds of resumes — but we weren’t 100% sure what we were looking for.”
In many ways, they were relying on that individual to tell them how QA should go. They made the hire — and at first the hire seemed promising.
But months into that person’s tenure, it was clear they weren’t delivering. PayHOA parted ways with them.
“We had wanted them to modernize our processes but that didn’t happen. In six months they hadn’t produced a single automated test. We thought, ‘Crap, we changed our process to account for this QA person.’”
Having been burned by the hiring experience, Dale and his team considered their other options.
Dale knew about Rainforest from a prior role and began his search there. He learned that Rainforest offered a managed service on top of its AI-powered platform.
The managed part could save his team of developers from time-consuming and painful test creation, test maintenance, and review of test failures, allowing them to focus on shipping code.
“I was aware of Rainforest from several years ago but I had no idea they offered a service to build the tests for us as well. That was a double whammy.”
They already knew that traditional, overseas QA outsourcing services could present security risks and communication issues, which could hurt productivity. But Rainforest reassured Dale confidence on these fronts.
“While an overseas team can be hit or miss on quality, it was reassuring that Rainforest assigned people in-country, in our time zone. As for security, we didn’t want to put sensitive data out there with an overseas team, but Rainforest resolves that by not needing access to the code, using secrets, and destroying the virtual machine after each run.”
Plus, trying Rainforest was a reversible decision: using Rainforest’s 60-day money-back guarantee and then switching to something else would still be far less disruptive than hiring another QA person or writing all their own tests.
The Rainforest setup was easier than they thought, and the Rainforest Test Managers got up to speed on PayHOA’s app faster than they expected.
“We really didn't have to put all that much effort into onboarding them. I want to give kudos to our Test Managers for jumping in and more or less figuring the app out on their own. We gave them a long list of things to test. They dove right in and figured it out based on our four-year-old video tutorials.”
Now, it’s like having a QA team without managing a QA team. The Test Managers review test failures, update the tests, and report the real bugs in Slack for the PayHOA team to fix.
Debugging issues with Rainforest is easy with detailed test results, including video recordings.
“Compared to Selenium it’s night and day. If a Selenium test fails, if you pre-programmed it to record the screen and take screenshots every time it clicked, you’re going to have to rewatch that whole video to figure out what failed. In Rainforest, if I click that ‘Failure’ button, it takes me right to the screen. It doesn’t get much simpler.”
The primary value has been the improvements in velocity and peace of mind. Dale’s team can push changes at “wildfire speed” and trust that they won’t be derailed by surprise bugs.
“Rainforest QA allows us to move really, really quickly. When I push a commit and see a test fail and then a few minutes later, I see a rerun and it passed, I'm like, ‘Okay, that wasn't something I broke, it was environmental and the Test Managers are on it.’ I didn't waste any time. Whereas if it’s a legitimate failure, there’s a message in the chat from the Test Manager.”
Rainforest’s testers have helped them catch more issues, including things that aren’t necessarily bugs, but negatively impact app behavior and create usability issues.
“That’s amazing. Our vendor changed something and not only did they catch and fix it, they made us aware. And sometimes they’ll message and say, ‘This thing works, but it’s kind of weird.’”
The boost in confidence in the QA process has helped PayHOA modernize its infrastructure in ways that will scale. They needed to make big upgrades on their platform, and with Rainforest, they finally could.
“We have a lot more confidence in pushing code in general. Before, we weren't confident that we had good enough test coverage to catch the things we really care about. But now, last week, we killed PHP 7.4 in production and are on 8.1 and instead of taking a month, it took four days.”
That’s just the start. They’re only just beginning to realize the added productivity.
“We reached the point of feeling confident about this testing. Now I expect we’ll see a huge velocity improvement.”
