How DataRobot Optimizes Test Automation with Rainforest QA title image

How DataRobot Optimizes Test Automation with Rainforest QA

Teams should leverage test automation. Used strategically, it increases efficiency and speeds up software delivery. Yet, not all tests are a great fit for automation, and teams often find that implementing it creates more challenges than it solves.

But that doesn't mean test automation can't be leveraged to create an optimized QA process. And the team over at DataRobot is doing just that! We sat down for a webinar with their VP of Engineering and Rainforest customer, Dustin Burke, and picked his brain on how they've optimized their QA strategy. Here is what we learned:

DataRobot Uses "Quality Assistance" to Build a Culture of Quality

Every team and product is unique, with varying testing needs. While there's no one-size-fits-all way to do QA -- especially when deciding how many tests to automate -- there are innovative approaches all successful teams share.

Quality Assistance vs. Quality Assurance

Inspired by the Atlassian team's transition from Quality Assurance to Quality Assistance, DataRobot implemented a QA process that empowers developers to manage code quality with automated regression testing. Quality engineers (QEs) couple with developers as their quality coaches. Efficiently providing advice on common testing guidelines and overall regression test coverage.

QEs do more high-value testing, like exploratory testing, rather than spending their time on regression testing. -Dustin Burke
With every member of the team contributing to product quality at each stage of the development cycle, QEs have more bandwidth to focus on high-impact testing initiatives. This approach doesn't just shift quality earlier in the development cycle. In DataRobot's case, this approach has created a culture of quality while optimizing resources.

DataRobot is Strategic with their Approach to Test Automation

It can be tempting to try automating everything when it comes to software testing. But not all tests are a good fit for automation, and targeting a meaningless goal is not a strategic way to manage QA. To get the benefits of test automation, you need to blend automated and human-executed testing by determining which test cases are a great fit for automation.

While DataRobot operates within a continuous delivery pipeline and automates as many tests as possible, they are strategic about which tests they automate. Furthermore, DataRobot finds ways to automate aspects of their testing process that are not a good fit for test automation.

We are heavily invested in test automation. We even have a dedicated team to make our automated tests scalable and cost-efficient. - Dustin Burke

DataRobot Automates their Manual Testing Process with Rainforest QA

Strategically automating tests is just one piece of DataRobot's QA success. Additionally, the team automates their manual testing process with the Rainforest platform.

Rainforest provided us with a new capacity to automate the equivalent to manual regression testing for flaky UI tests, previously done by human testers on our team. - Dusin Burke

DataRobot's Delivery Pipeline with Test Automation and Rainforest QA:

  1. Test Case Management: QA team manages Rainforest test suites, which are version controlled in their Github repository. Tests are written in plain English.
  2. Test Execution Trigger: A Jenkins job runs Rainforest tests, provisioning access to the staging environment. The Rainforest CLI automatically populates the test cases from the latest version.
  3. Deployment Integration: The Jenkins job is integrated with DataRobot's staging deployment pipeline. It's triggered by a 'switchover' (i.e. the automated tests passed, and a Blue/Green deployment was initiated).
  4. Slack Notifications: The DataRobot team gets Rainforest test results through a Slack channel which is managed by their QA and Cloud Ops team, and uses the feedback to decide whether or not to ship.
What is rainforest?

Rainforest is a unified platform for software testing. Quickly build no code QA tests that can be run with automated or crowd execution. Works across browsers, platforms, and mobile.