How NBCUniversal Implemented Continuous Delivery at Enterprise Scale

Picture of Ashley Dotterweich
Ashley Dotterweich, Tuesday April 18, 2017

Implementing fast development methodologies requires strategic thinking for any organization. For large enterprises, getting to continuous delivery can be a serious challenge. Jeremy Ocampo, NBCUniversal's VP of Software Quality Engineering & Agile Management, recently spoke at Developer Week about his team's transition to continuous testing.

Over the past few years, NBCUniversal has adopted a continuous delivery methodology to keep up with the demands of their global audience. Using continuous testing that ensures that QA moves as fast as development does. In this Developer Week talk, Ocampo revealed his major takeaways from implementing continuous testing that other enterprise teams can leverage to bring their teams up to the speed of continuous.

Break Big Work into Small Pieces

Enterprise organizations often have massive, monolithic code bases, which presents some issues for implementing continuous delivery. To release code faster, NBCUniversal had to reconsider the scope of their development and testing projects.

Think small to move fast. Running tests against an entire code base is time consuming, and doesn’t provide actionable results quickly enough for the development team to respond to. “We divvy up the features into small chunks, similar to any agile process. For each of these features, we develop a series of tests,” says Ocampo. By dividing their code base into functional chunks, they can keep the testing cycle as fast as possible.

Testing happens on day one. Instead of waiting weeks to start testing, Ocampo's team starts testing as soon as they begin developing. NBCUniversal’s team must execute comprehensive regression & security tests before merging into delivery pipeline, but running an enormous amount of tests at once can delay releases. Switching to continuous testing allows the team to test in smaller increments and receive real-time feedback on their code.

Rethink the Development and QA Team Divide

For enterprises, development and software quality testing can be fulfilled by completely discrete teams. Siloing these teams creates a lag in communication that continuous delivery can’t afford. Smaller organizations often resolve this by either combining the development and QA teams into a single group, or avoiding hiring a QA team entirely.

Collaboration is critical. Because merging teams isn’t always an option for larger, more complex organizations, enterprises must create the culture and processes that for tight collaboration between the two teams. NBCUniversal’s development and quality engineering teams work together closely to meet their common goal of rapidly shipping high-quality code.

Build Test Coverage for Business Value

Because balancing testing time with coverage is critical to doing continuous delivery at scale, Ocampo’s team builds test coverage that will provide the highest business value. When a team is working against the clock to release code as quickly as possible, every QA test must make a difference.

Test whatever is changing. Areas of the codebase that are not changing do not need to be tested as rigorously as more dynamic areas. Ocampo’s team knows that they are most likely to encounter issues with code that has recently changed. By focusing on testing new code, the team can find and resolve show-stopping issues rapidly without wasting resources.

Automate wherever possible. Another component of building coverage effectively is choosing the testing methods that provide the fastest possible results. Ocampo stresses the importance of moving towards as much automation as possible, but he also notes that automation can be a resource-intensive process. Identifying the most valuable test cases to automate helps keep the automation scalable while maximizing its benefits.

Identify & Measure What You’re Not Testing

Ocampo notes that knowing what you’re not testing is just as important as knowing what coverage you have. Code changes that don’t have the appropriate amount of testing “Hypothetically speaking, a code commit could go to production at any time. We need to understand what our risks are to identify and ensure what we are developing is being tested appropriately.”

Find and quantify “Quality Holes.” There’s no way to test everything, but understand where you have testing gaps, or “Quality Holes,” can help you understand where to allocate resources. “You need to have the right instrumentation in place to understand what are the gaps in your application and your testing process,” says Ocampo. NBCUniversal’s development team is working towards measuring their testing gaps to determine if a lack of coverage should be a show-stopper for any given release.

Using Continuous Testing to Do Continuous Delivery at Scale

For NBCUniversal, continuous testing accelerates continuous delivery. Ocampo’s team is able to ship rapidly while avoiding incurring technical debt. Enterprise teams can drive faster development by adopting a similar continuous testing strategy that provides more assurance for each deployment.

To learn more about implementing continuous testing strategy that keeps up with continuous delivery and to find out how other organizations are speeding up their QA process, download our free guide, Continuous Testing for CTOs.