{"id":342,"date":"2023-01-24T23:08:00","date_gmt":"2023-01-24T23:08:00","guid":{"rendered":"http:\/\/rainforestqa.com\/ci-cd-automated-testing\/"},"modified":"2024-10-01T17:35:15","modified_gmt":"2024-10-01T17:35:15","slug":"ci-cd-automated-testing","status":"publish","type":"post","link":"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing","title":{"rendered":"CI\/CD automated testing: How to release fast, with confidence"},"content":{"rendered":"\n<p>If you\u2019ve got an agile team interested in shipping fast without breaking things, this post is for you.<\/p>\n\n\n\n<p>In this piece, I\u2019m going to explain how we at Rainforest QA approach automated testing in a continuous integration \/ continuous delivery (CI\/CD) pipeline, with a focus on end-to-end (E2E) functional testing. The aim of our testing and other DevOps methodologies is to maintain a healthy balance between speed and product quality.&nbsp;<\/p>\n\n\n\n<p>We\u2019ve built <a id=\"\" href=\"https:\/\/www.rainforestqa.com\" target=\"_blank\" rel=\"noopener\">a no-code test automation platform designed for CI\/CD<\/a> and we dogfood our product every day, so we think about this topic a lot. Plus, we\u2019ve seen what software testing practices work \u2013 and don\u2019t work \u2013 for hundreds of our customers, so it\u2019s fair to say we\u2019ve got an informed perspective.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Specifically, I\u2019m going to cover each of the four core testing processes involved in managing an automated test suite within a CI\/CD pipeline:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Adding tests<\/li>\n\n\n\n<li>Running tests<\/li>\n\n\n\n<li>Test maintenance<\/li>\n\n\n\n<li>Debugging test failures<\/li>\n<\/ul>\n\n\n\n<p>But first, it\u2019s worth noting that the model of ownership of automated testing within software development teams has changed in recent years, which changes how to think about who in your org owns each of these four processes.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Who_owns_the_automated_test_suite\" >Who owns the automated test suite?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Adding_tests\" >Adding tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Running_tests\" >Running tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Test_maintenance\" >Test maintenance<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Debugging_test_failures\" >Debugging test failures<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.rainforestqa.com\/blog\/ci-cd-automated-testing\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Who_owns_the_automated_test_suite\"><\/span><strong id=\"\">Who owns the automated test suite?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>With traditional automated testing tools like Selenium or Cypress, you\u2019d need to hire or allocate someone with the right technical skills to work within those frameworks. That person would own all aspects of creating, debugging, and maintaining tests because they\u2019d be the only ones with the expertise \u2013 and, therefore, responsibility \u2013 for doing so.&nbsp;<\/p>\n\n\n\n<p>But the person with the expertise isn\u2019t always the person who\u2019s in the best <em id=\"\">position<\/em> in the software development or CI\/CD process to take action. For example, developers are in the best position to update breaking tests because they\u2019re the most familiar with the details of each release. In many cases, they can quickly look at a failing test and know exactly what broke it.&nbsp;<\/p>\n\n\n\n<p>In a traditional test automation model, though, they\u2019d have to communicate back-and-forth with a quality assurance engineer doing the debugging. This slows things down \u2013 which is bad when you\u2019re trying to ship!<\/p>\n\n\n\n<p>On the other hand, when you\u2019re working with a no-code solution like Rainforest, you can assign testing responsibilities based on what makes sense for your software development process. (For us, that includes not having a QA team nor any conventional QA tester roles since <strong id=\"\">we believe <\/strong><a id=\"\" href=\"https:\/\/www.rainforestqa.com\/method#p1\" target=\"_blank\" rel=\"noopener\"><strong id=\"\">product builders should own quality<\/strong><\/a>.)<\/p>\n\n\n\n<p>For the purposes of this piece, we\u2019re going to assume you\u2019re using a no-code automation tool like Rainforest so you have the flexibility to assign testing responsibilities to the best-suited roles. Even if that\u2019s not the case for you and your team, you should still be able to learn from the underlying principles we follow.<\/p>\n\n\n\n<p>Here\u2019s how we\u2019ve set up automated testing within our CI\/CD pipeline here at Rainforest, including our underlying DevOps principles, pro tips, and owners of each part of the testing process.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Adding_tests\"><\/span>Adding tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You can\u2019t add new end-to-end (E2E) tests to your suite in an ad hoc way and expect to somehow get sufficient coverage of your most important features. It requires proactive planning.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Test planning<\/strong><\/h3>\n\n\n\n<p>For new features, the relevant product squad defines new E2E test coverage during the feature planning process. In fact,<strong id=\"\"> feature planning isn\u2019t considered complete until test coverage is defined<\/strong>.<\/p>\n\n\n\n<p>Given the coverage that already exists, the squad identifies new coverage that\u2019ll be needed. User flows and mockups guide most of this discussion for the new feature.&nbsp;<\/p>\n\n\n\n<p>The squad also determines where the tests will live within the structure of the test suite, since keeping the suite organized makes a lot of steps in the testing workflow easier.&nbsp;<\/p>\n\n\n\n<p>Finally, the squad considers if new functionality will replace old functionality, in which they\u2019ll need to remove or modify some existing tests. (In the <a id=\"\" href=\"#maintain\">Test Maintenance section<\/a>, we\u2019ll cover an easy shortcut for identifying tests that need to be updated.)&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Who adds tests to the suite?<\/strong><\/h3>\n\n\n\n<p>During the planning phase, the squad also determines who\u2019ll handle adding the new test coverage. Ultimately, the decision is guided by determining who has the bandwidth and the required context to author the tests.<\/p>\n\n\n\n<p>Usually, the product manager (PM) or lead developer on the squad is responsible for adding the tests. (The developer handles updating \/ removing tests \u2013 i.e., test maintenance \u2013 which we\u2019ll cover later in this piece.)&nbsp;<\/p>\n\n\n\n<p>If it\u2019s a large feature getting shipped that requires a lot of new coverage, the squad will usually share the work across its members: the PM, developer(s), and designer. Using a no-code tool means we can share the workload \u2013 even with non-technical team members \u2013 to get it done more quickly.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"960\" height=\"600\" src=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/03\/add-a-step-click-a-button-notion-1.gif\" alt=\"Adding a step in Rainforest QA\" class=\"wp-image-1146\"\/><figcaption class=\"wp-element-caption\">Adding a test step in Rainforest QA<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">When to add test coverage<\/strong><\/h3>\n\n\n\n<p>As far as <em id=\"\">when<\/em> to add new tests, the key is to <strong id=\"\">do it before code makes it to your users<\/strong>. Even though the value of this approach might be obvious \u2013 you want to make sure new functionality works properly before your users see it \u2013 it\u2019s not always followed, especially in the eagerness to ship.<\/p>\n\n\n\n<p>Depending on the configuration and policies of your CI\/CD pipeline, there are several different ways you could add test coverage to new code before it reaches users. We tend to use one of these three methods:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Run tests against a code branch<\/li>\n\n\n\n<li>Add a manual approval step before release to prod<\/li>\n\n\n\n<li>Ship code behind a feature flag<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">1. Run tests against a code branch<\/strong><\/h4>\n\n\n\n<p>This approach requires that you can deploy code to a specific environment (like QA) and can run your automated tests against that environment. Which means, for example, test data (like usernames and passwords for fake accounts) should be available to that environment.&nbsp;<\/p>\n\n\n\n<p>On our team, we create <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/using-branching\" target=\"_blank\" rel=\"noopener\">a branch in Rainforest<\/a> to stage our new and updated tests. Once these tests pass when run against the analogous code branch, we merge them both so our test suite stays in-sync with our code repository&#8217;s main branch.<\/p>\n\n\n\n<p>A nice thing about this approach is that it doesn\u2019t require blocking the release pipeline while we work on updating the test suite.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">2. Add a manual approval step before release to prod<\/strong><\/h4>\n\n\n\n<p>In our pipeline, we run our suite of automated Rainforest tests as a blocking step before pushing to prod. Once the pipeline reaches the point of test execution, we pause it to give us a chance to make any updates to the suite.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"277\" src=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-1-1024x277.png\" alt=\"\" class=\"wp-image-1216\" srcset=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-1-1024x277.png 1024w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-1-300x81.png 300w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-1-768x208.png 768w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-1.png 1050w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<p>In this diagram of our CircleCI pipeline, notice that <em id=\"\">trigger_rf_run<\/em> is paused \u2013 it requires manual approval. At this point, any new code has been moved to our staging\/QA environment (which happened in the <em id=\"\">update_release_qa_stg<\/em> step), where we can update our tests. When that\u2019s done, we manually resume the process.<\/p>\n\n\n\n<p>Once the updates give us passing test results, the code is moved into the next step of the pipeline which creates a production build and ships the code to production:<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"263\" src=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-2-1024x263.png\" alt=\"\" class=\"wp-image-1217\" srcset=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-2-1024x263.png 1024w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-2-300x77.png 300w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-2-768x197.png 768w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-2.png 1050w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<p>This approach gives you an easy way to control when code is pushed to prod without adding any meaningful overhead. Setting up a manual approval step can be as simple as adding a single line to the configuration of the CI\/CD tool you use in your software development environment. For example, CircleCI requires <a id=\"\" href=\"https:\/\/circleci.com\/blog\/manual-job-approval-and-scheduled-workflow-runs\/\" target=\"_blank\" rel=\"noopener\">adding \u201ctype: approval\u201d<\/a>. In Jenkins, you can add manual approval steps with the <a id=\"\" href=\"https:\/\/www.jenkins.io\/doc\/pipeline\/steps\/pipeline-input-step\/\" target=\"_blank\" rel=\"noopener\">Pipeline: Input Step plugin<\/a>.&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">3. Ship new code behind a feature flag<\/strong><\/h4>\n\n\n\n<p>In this approach, we push to prod, but hide new code from users behind a feature flag until we\u2019ve had a chance to update our tests.<\/p>\n\n\n\n<p>How you use this technique will depend on how your feature flags are implemented. At Rainforest, to test code protected by a feature flag, we have to add some seeds to our database. That is, we need login credentials for a fake user that has this feature enabled. We can then write tests against this login and add it to our test suite. Once the flag has become generally available, we remove the feature flag code and database seeds.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Other time-saving guidelines for adding tests<\/strong><\/h3>\n\n\n\n<p>One of the best ways to avoid creating unnecessary slowdowns and pain for your team is to <strong id=\"\">avoid adding more tests than you really need<\/strong>. Tests are expensive in terms of maintenance, which many teams learn the hard way. So, even though it can be tempting to test <em id=\"\">all the things, <\/em>&nbsp;don\u2019t add more tests to your suite than you can reasonably afford to maintain.&nbsp;<\/p>\n\n\n\n<p>Follow the <a id=\"\" href=\"https:\/\/www.rainforestqa.com\/blog\/automation-test-coverage\" target=\"_blank\" rel=\"noopener\">Snowplow Strategy<\/a> to prioritize your test creation, and \u2013 as a rule of thumb \u2013 don\u2019t don\u2019t create tests for functionality you wouldn\u2019t fix right away if it broke.<\/p>\n\n\n\n<p>Finally, keep each of your tests as short as possible. When you keep the scope of each of your tests as finite as you can, you\u2019ll find they finish more quickly and are easier to debug and maintain.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Running_tests\"><\/span>Running tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Who runs the automated tests?<\/strong><\/h3>\n\n\n\n<p>If you\u2019re running a CI\/CD pipeline, running E2E regression testing is presumably a step in your release process. (If it\u2019s not, it should be.) In which case, running tests happens automatically in every release. There\u2019s no \u201cwho\u201d involved, unless you count the developers on the team, who (typically) own the release process itself.&nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<p><em id=\"\">Note: while many teams call it the \u201cmaster branch,\u201d we here at Rainforest call it the \u201cmain branch,\u201d so that\u2019s the vernacular I\u2019m going to use in this piece.<\/em><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">When to run tests<\/strong><\/h3>\n\n\n\n<p>We don\u2019t want to run E2E tests <em id=\"\">every<\/em> time we\u2019re pushing or merging code &#8211; that\u2019d be overkill and would bog down the pipeline at some point. We mostly think about this in terms of these two scenarios:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>code is <em id=\"\">not<\/em> being merged to main or released to production, or<\/li>\n\n\n\n<li>code is being merged to main or released to production.&nbsp;<\/li>\n<\/ol>\n\n\n\n<p>To help elaborate, here\u2019s a simplified diagram of our CI\/CD environment-based workflow. All non-production environments are replicas of production \u2013 they have their own url, database, server, etc, that are all configured to be as close to production as possible, with some differences in how data is added.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"484\" src=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-diagram-1024x484.png\" alt=\"\" class=\"wp-image-1218\" srcset=\"https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-diagram-1024x484.png 1024w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-diagram-300x142.png 300w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-diagram-768x363.png 768w, https:\/\/www.rainforestqa.com\/blog\/wp-content\/uploads\/2023\/04\/cicd-pipeline-diagram.png 1050w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">Code is not being merged to main or released to production<\/strong><\/h4>\n\n\n\n<p>This scenario is the most common. We constantly push code to different branches, merge non-main branches together, and all the other operations that occur in a typical version-control workflow.&nbsp;<\/p>\n\n\n\n<p>This is the first step in the diagram: <em id=\"\">Dev&#8217;s local machine<\/em> pushes code to some GitHub branch. At this point, it doesn\u2019t make sense to run E2E tests because we know they\u2019ll likely fail \u2013 code on non-main branches is very often in a broken or incomplete state. Relatively speaking, E2E testing is more expensive in time and money than, say, unit tests, so it wouldn\u2019t be a good use of resources to run them at this stage.<\/p>\n\n\n\n<p>So, at this stage, we run very fast and almost-free checks on non-main branches, including things like type checking, linting, and unit tests. We deploy this code to our staging environment once all the checks pass, which allows non-devs from the product squad to do any manual testing or exploratory testing (like we do when we\u2019re releasing a big new feature).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">Code is being merged to main or released to production<\/strong><\/h4>\n\n\n\n<p>When it\u2019s time to start the release process, we merge code to the main branch, which kicks off a special workflow. This workflow includes all the checks from the previous scenario, plus additional steps: once those checks pass, we kick off an automated E2E test run. If all the tests pass, then the additional deployment steps (create a production build, deploy it to production servers, etc.) happen. At this point, we\u2019re confident our release contains no new bugs since we trust our test suite to catch any issues.<\/p>\n\n\n\n<p>It\u2019s worth noting that it\u2019s always good practice to tie your E2E test run to a release ID of some kind. For most teams this is the commit SHA of the pull request being merged to main, but it can be any unique identifier that your team uses. This practice allows you to quickly identify which chunks of code are responsible for test failures.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">Scheduled tests<\/strong><\/h4>\n\n\n\n<p>In addition to the <a href=\"https:\/\/www.rainforestqa.com\/blog\/when-to-run-e2e-tests\">E2E tests we run<\/a> in the release process, we also schedule tests. We have a smoke suite of automated tests that run daily in production to make sure important flows in Rainforest get frequent tire-kickings \u2013 we make sure the app is up <em id=\"\">and<\/em> that it\u2019s behaving as expected. We also have scheduled tests for our marketing site (which is separate from our core application).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Test_maintenance\"><\/span>Test maintenance<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The easiest way to identify which tests need to be updated is by running the automated test suite in the release process. This\u2019ll reveal which tests have broken as a result of product changes in the release.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Who\u2019s responsible for test maintenance?<\/strong><\/h3>\n\n\n\n<p>At Rainforest, developers are responsible for maintaining our automated test suite.&nbsp;<\/p>\n\n\n\n<p>Anyone on the product squad should know the user flows of the feature, but the <strong id=\"\">devs ultimately know the most about the exact code being shipped and have other helpful context<\/strong>. For example, maybe a test failure was caused by a data seeding issue in the staging environment \u2013 that\u2019s easy for a dev to recognize, but more difficult for a non-dev to figure out without some investigation.&nbsp;<\/p>\n\n\n\n<p>That\u2019s why devs are in the best position to unblock the process by bringing broken tests into a passing state.<\/p>\n\n\n\n<p>For a lot of devs, this idea might set off alarm bells. Test maintenance has gotten a bad rap thanks to open source frameworks like Selenium and Cypress and their derivatives. In these frameworks, if you didn\u2019t write the tests, they take some effort to decipher. And then you have to figure out which selectors in your tests match with which elements in your app. It\u2019s a bit of a slog. And all this assumes you know how to work in these frameworks in the first place. If not, you\u2019re going to wait on a QA engineer to hopefully get your test results to a passing state so you can ship your code.<\/p>\n\n\n\n<p>In fact, we have customers who specifically switched to Rainforest away from those frameworks because <a id=\"\" href=\"https:\/\/www.rainforestqa.com\/case-studies\/flux\" target=\"_blank\" rel=\"noopener\">they struggled to maintain their tests and ship fast<\/a>. With our no-code approach, anyone can look at a test and quickly understand it and update it.<\/p>\n\n\n\n<p>This quick video follows along as Rainforest test automation downloads and installs Brave browser on desktop \u2013 you can see that every test step is in easily-understandable English:<\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Rainforest QA Test - Download and Install Brave Browser\" width=\"500\" height=\"375\" src=\"https:\/\/www.youtube.com\/embed\/TshFfFKjl2Q?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Where to do test maintenance<\/strong><\/h3>\n\n\n\n<p>The Rainforest platform has <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/using-branching\">a branching system<\/a> that allows our different teams to make updates to the test suite without stepping on each other\u2019s toes. We have several CI\/CD pipelines in our company \u2013 for the frontend and backend codebases, for example \u2013 that all share the same tests. So, committing test changes to a branch and merging when we\u2019re done allows us to avoid collisions and delays during test maintenance.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Usually, if the required updates are minor, we\u2019ll commit these changes directly to main. If there\u2019s a more time-consuming refactor required (like when shipping a major update), we\u2019ll <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/using-branching\">make the changes on a branch<\/a> so we don\u2019t block other dev teams.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Debugging_test_failures\"><\/span>Debugging test failures<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>It\u2019s not enough to just run E2E tests as part of the release process \u2013 <strong id=\"\">failed tests <em id=\"\">must<\/em> block the release pipeline<\/strong>. Ignoring or disabling failed tests (which can be tempting when there\u2019s pressure to ship) invites a vicious circle that undermines product quality: when there are consistently failed tests that get ignored, people rightfully doubt the utility of the test suite, which leads to less investment in it. The test suite deteriorates, and more bugs make it to prod.<\/p>\n\n\n\n<p>A single disabled test might not break your app, but it begins to normalize behavior that eventually does have meaningful negative consequences. Put another way: A single hole may not sink your boat, but many holes will. When there\u2019s one hole, fixing the problem is manageable. But when we have too many, the boat will take on water faster than we can plug the holes.&nbsp;<\/p>\n\n\n\n<p>Therefore, as a fundamental practice of software testing hygiene, <strong id=\"\">fix broken tests right away<\/strong>. Your test suite is only useful if your team keeps it up-to-date.<\/p>\n\n\n\n<p>If you\u2019ve got a test that\u2019s failing because of a known bug that you continue to ignore, it might be time to ask yourself if it\u2019s time to remove that test from your suite. (Remember our rule of thumb: Only add test coverage for flows you\u2019d fix right away if they broke.)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">Categorize each test failure<\/strong><\/h3>\n\n\n\n<p>We get a <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/slack-integration\">Slack notification<\/a> to let us know when an automated test run has failed. When that happens, the first step is always to review the test failures and categorize them so they can be resolved. (Rainforest includes options for <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/how-to-categorize-failures\">categorizing test failures<\/a> and assigning them to the right people on your team for resolution.)<\/p>\n\n\n\n<p>Failures tend to fall into one of three categories:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The test needs to be updated<\/li>\n\n\n\n<li>There\u2019s a bug&nbsp;<\/li>\n\n\n\n<li>There are issues with the test environment<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">1. The test needs to be updated<\/strong><\/h4>\n\n\n\n<p>In this scenario, our code is working as expected and we\u2019ve added or changed some functionality that caused an existing test to fail. As mentioned earlier, this is an easy way to identify tests that need to be updated as a result of product changes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">2. There\u2019s a bug<\/strong><\/h4>\n\n\n\n<p>A bug happens when there\u2019s a problem with the code we\u2019re shipping and the unit tests didn\u2019t catch it. (That\u2019s why it\u2019s important to run E2E tests! We consider them to be integration tests that verify the various service APIs and functionalities covered individually by unit tests all work together properly.)<\/p>\n\n\n\n<p>In this case, we have two options: 1) merge a hotfix or 2) revert the changes.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\"><strong id=\"\">When to hotfix&nbsp;<\/strong><\/h5>\n\n\n\n<p>Merging a hotfix means we merge an additional change into the code that caused test failures. Hotfixing is a workable option, but you should <strong id=\"\">only consider it if the dev is <em id=\"\">very <\/em>confident that it\u2019ll fix the issue without introducing additional problems.<\/strong>&nbsp;<\/p>\n\n\n\n<p>After all, the dev was probably confident in the integrity of the original code that they tried to ship \u2013 how can they be certain they\u2019re right about the hotfix? (This is a reflection on the riskiness of hotfixes, not upon the competence of any developer.) The last thing you want to do is compound the issue and create more complexity you then have to untangle.<\/p>\n\n\n\n<p>Only consider a hotfix if:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>the \u201cfix\u201d is a very small code change that another dev can quickly review,<\/li>\n\n\n\n<li>the code change has very low complexity,<\/li>\n\n\n\n<li>you\u2019re sure the fix won\u2019t have any undesired consequences or side-effects, and<\/li>\n\n\n\n<li>there are no other outstanding issues that could cause failures. For example, if there were multiple test failures during the run, you need to be sure the fix will address <em id=\"\">all <\/em>of the failures.&nbsp;<\/li>\n<\/ol>\n\n\n\n<h5 class=\"wp-block-heading\"><strong id=\"\">When to revert<\/strong><\/h5>\n\n\n\n<p>If you\u2019re not deeply confident in a hotfix, the best course is to revert the code. It\u2019s simple to do, and you can try to ship the code again later once a fix has been applied.&nbsp;<\/p>\n\n\n\n<p>When reverting, we just merge in a \u201crevert\u201d PR. But this means the code has to go through the whole release pipeline again. While this may seem a bit annoying, it\u2019s a necessary step to protect the app: after the revert has been merged, we need to run our entire test suite to make sure the code was properly reverted and everything is back to a stable state.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong id=\"\">3. There are issues with the test environment&nbsp;&nbsp;<\/strong><\/h4>\n\n\n\n<p>Not infrequently, we speak with customers who deal with test failures stemming from a flaky staging or QA environment. We run into these issues, ourselves.&nbsp;<\/p>\n\n\n\n<p>It\u2019s ideal to make these environments mirror the production environment as much as possible, but it\u2019s not always practical. We\u2019re often willing to invest more time and money into production, but non-production environments simply need to be <em id=\"\">good enough<\/em> to get the job done. That\u2019s why prod and non-prod environments can and do differ in terms of performance, data seeding, network connectivity, and other variables.&nbsp;<\/p>\n\n\n\n<p>So <em id=\"\">some<\/em> amount of test failures due to environmental factors are expected. In these cases, you can often resolve these failures by simply rerunning the failed tests. (At Rainforest, we save time by <a id=\"\" href=\"https:\/\/help.rainforestqa.com\/docs\/test-retries\" target=\"_blank\" rel=\"noopener\">re-running failed tests automatically<\/a>.)&nbsp;<\/p>\n\n\n\n<p><strong id=\"\">When non-production environments cause a meaningful amount of test failures, it&#8217;s worthwhile to improve the state of those environments<\/strong>. The upfront costs may not be appetizing, but they\u2019ll pay for themselves in how much faster you\u2019re able to release code.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong id=\"\">When bugs are found on production<\/strong><\/h3>\n\n\n\n<p>Inevitably, users will find bugs in production. We\u2019ll never have 100% test coverage on every possible user interaction, so it\u2019s only a matter of time until a user finds a novel combination of clicks and keyboard strokes that breaks your application.<\/p>\n\n\n\n<p>When this happens, the flow is simple:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>file a bug ticket,<\/li>\n\n\n\n<li>fix the bug, and<\/li>\n\n\n\n<li>add test coverage to make sure users never see this bug again.<\/li>\n<\/ol>\n\n\n\n<p><strong id=\"\">If you\u2019re shipping a bug fix, always add test coverage for it.<\/strong> (If the bug was worth fixing, it\u2019s worth making sure it never escapes to prod, again.) However, the type of test depends on the nature of the bug \u2013 sometimes it\u2019s fine to add unit test coverage instead of automated E2E coverage. Either way, it\u2019s the dev\u2019s responsibility since they\u2019re the one in the best position to decide.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span><strong id=\"\">Conclusion<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Determining what environments you need, how they should be configured, what workflow is best for your development team, and all the other nuances that come with shipping code can be tricky. Regardless of how you decide to implement these things, it\u2019s always important to maintain high product quality standards.<\/p>\n\n\n\n<p>The practices I\u2019ve outlined in this article can be a lot to absorb, especially if you\u2019re setting up a CI\/CD pipeline for the first time. If nothing else, remember these five fundamental rules:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Plan test coverage as part of feature planning. Only test what you\u2019d fix right away.<\/li>\n\n\n\n<li>Add test coverage for new functionality before your users can access it.&nbsp;<\/li>\n\n\n\n<li>Run a regression test suite as part of every release.<\/li>\n\n\n\n<li>Failed tests always block the release.<\/li>\n\n\n\n<li>Always keep your tests up-to-date.&nbsp;<\/li>\n<\/ol>\n\n\n\n<p>Making these policies explicit parts of your software development process and DevOps pipelines will help you get to a state of continuous testing that reliably protects your product\u2019s quality as you ship fast and frequently.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A guide to implementing automated testing in CI\/CD. For agile teams looking to balance speed with product quality.<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-342","post","type-post","status-publish","format-standard","hentry","category-test-automation"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/342","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/comments?post=342"}],"version-history":[{"count":15,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/342\/revisions"}],"predecessor-version":[{"id":2706,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/342\/revisions\/2706"}],"wp:attachment":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/media?parent=342"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/categories?post=342"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/tags?post=342"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}