{"id":353,"date":"2022-01-12T23:08:02","date_gmt":"2022-01-12T23:08:02","guid":{"rendered":"http:\/\/rainforestqa.com\/flaky-tests\/"},"modified":"2025-03-18T13:14:50","modified_gmt":"2025-03-18T13:14:50","slug":"flaky-tests","status":"publish","type":"post","link":"https:\/\/www.rainforestqa.com\/blog\/flaky-tests","title":{"rendered":"A practical guide to reducing the burden of flaky tests"},"content":{"rendered":"\n<p>Flaky tests are automated software tests that sometimes pass and sometimes fail without an obvious reason. Often these tests will work well for a while, then occasionally start to fail. If the test passes on a second or third try without any obvious reason for the failures, the tester typically chalks it up to a glitch in the system and ignores the failed test result.&nbsp;<\/p>\n\n\n\n<p>If the test continues to perform inconsistently, the tester may stop running that test altogether, losing potentially critical test coverage.&nbsp;<\/p>\n\n\n\n<p>Either way, if failed test results are being ignored or tests aren\u2019t being run, real bugs can get missed.&nbsp;<\/p>\n\n\n\n<p>While it may seem like nothing has changed between passing and failing test runs, in reality, there\u2019s always something that changed to cause the failed test result.&nbsp;<\/p>\n\n\n\n<p>This article will help you:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Identify common causes of flaky tests.&nbsp;<\/li>\n\n\n\n<li>Find the best way to handle flaky tests that do arise.<\/li>\n\n\n\n<li>Learn the easiest way to uncover the root cause of a flaky test.<\/li>\n\n\n\n<li>Develop good practices to prevent flaky tests.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p><strong>Note:<\/strong> If you\u2019re a current Rainforest user and you\u2019re looking for troubleshooting tips, <a href=\"https:\/\/help.rainforestqa.com\/docs\/how-to-debug-automation-results\" target=\"_blank\" rel=\"noopener\">see this article<\/a>.<\/p>\n\n\n\n<p><em>It\u2019s easier to find the root cause of automated test failures with Rainforest QA than any code-based testing framework. <a href=\"https:\/\/www.rainforestqa.com\/talk-to-sales\">Talk to us<\/a> to see the difference. <\/em><\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.rainforestqa.com\/blog\/flaky-tests\/#Common_Causes_of_Flaky_Tests\" >Common Causes of Flaky Tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.rainforestqa.com\/blog\/flaky-tests\/#What_to_Do_with_Flaky_Tests\" >What to Do with Flaky Tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.rainforestqa.com\/blog\/flaky-tests\/#The_Easiest_Way_to_Find_the_Root_Cause_of_Flaky_Tests\" >The Easiest Way to Find the Root Cause of Flaky Tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.rainforestqa.com\/blog\/flaky-tests\/#Best_Practices_to_Prevent_Flaky_Tests\" >Best Practices to Prevent Flaky Tests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.rainforestqa.com\/blog\/flaky-tests\/#Easily_Handle_Flaky_Tests_with_Rainforest_QA\" >Easily Handle Flaky Tests with Rainforest QA<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Common_Causes_of_Flaky_Tests\"><\/span>Common Causes of Flaky Tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In an ideal world, you\u2019d be able to test your application in the exact same test environment and return the exact same results every time. While some testing software can get close to this ideal (by using virtual machines with standard configurations instead of real devices, for example), the reality is that it can be very difficult to control every aspect of your test run.&nbsp;<\/p>\n\n\n\n<p>This is especially true for end-to-end (E2E) tests because there are so many moving parts and dependencies. Some teams believe it&#8217;s nearly impossible to mitigate flaky tests with automated E2E testing and therefore skip automating UI tests altogether. But E2E testing plays an essential role in ensuring a high quality user experience. Even if automating your E2E tests means you encounter a few flaky tests, automation can still significantly improve the speed and repeatability of your testing.&nbsp;<\/p>\n\n\n\n<p><strong>Plus,<\/strong> <strong>there are ways to mitigate flaky tests<\/strong>.<\/p>\n\n\n\n<p>The first step is to understand what can cause flaky tests. Even though the term \u2018flaky test\u2019 may suggest it\u2019s always an issue with the test that causes inconsistent results, there are many other factors that could cause inconsistent results.&nbsp;<\/p>\n\n\n\n<p>Three of the most common causes of test flakiness, other than issues with the test itself, are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Test environment issues.<\/strong> This includes issues such as the application loading too slowly (e.g., the test tries to execute an action before the page finished loading), the application crashing or going offline intermittently, network latency, or interactions with third-party APIs failing or running slower than usual.<\/li>\n\n\n\n<li><strong>Non-determinism in app behaviors.<\/strong> Non-determinism is when a feature is designed to display different results or behaviors between runs even when given the same input. For example, you may be trying to test search functionality where the last step of the test is to verify that results appear. To verify this, the test looks for a particular title to appear somewhere on the first page. Perhaps that specific title appears on the first page nine times out of 10, but on the tenth run, it shows up on the second page, causing the test to fail.&nbsp;&nbsp;&nbsp;<\/li>\n\n\n\n<li><strong>Issues with data management.<\/strong> If multiple tests use the same test user account, then running tests at the same time could create collisions. For example, one test might fail to log in because the account is already in use by another test.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>When a flaky test fails, it may be because of an issue in the test environment that won\u2019t carry over into the production environment. However, some issues such as network errors, slow load times, or problems with third-party APIs could carry over and ultimately end up affecting the end user. If you ignore flaky test results, there\u2019s a good chance you\u2019ll be ignoring real problems.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_to_Do_with_Flaky_Tests\"><\/span>What to Do with Flaky Tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>While ignoring failed flaky test results isn\u2019t a good idea, it\u2019s not always practical to spend a lot of time troubleshooting inconsistent tests. There are several options when it comes to handling flaky tests and it may be helpful to use all of them in different situations. However, it\u2019s important to make sure you\u2019re deliberately choosing the best response to maintain your team&#8217;s standards of quality assurance (QA).&nbsp;<\/p>\n\n\n\n<p>Your options for what to do with flaky tests include:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Immediately search for the cause of the failure and fix it.<\/strong> While this is the ideal solution, it simply isn\u2019t practical on a large scale. But it\u2019s always the first option, and may be the only safe option if the test is covering a critical user path.<\/li>\n\n\n\n<li><strong>Re-run all failed tests, and if they pass, move on.<\/strong> This option saves time in the moment, but can cause your test suite to become less reliable over time if you never address the underlying issue. It\u2019s often a good idea to determine specific criteria for when this option can be used. Some examples of situations that might always call for a re-run include timing issues or temporary issues with the test environment that won\u2019t carry over into production. You may also want to specify how many times in a row this option can be used for one test, and quarantine tests that are particularly flaky.&nbsp;<\/li>\n\n\n\n<li><strong>Temporarily disable the test or remove it from the test run group.<\/strong> This may be a good option if release speed is more important than dealing with inconsistencies the test might uncover. It\u2019s important that you make a plan to fix the test in the future and hold your team accountable to that plan.&nbsp;<\/li>\n\n\n\n<li><strong>Move tests that produce inconsistent results into a separate test run group<\/strong>. This creates a clear expectation that only the tests in this group are flaky, restoring confidence in the rest of your test suite. It\u2019s still good to run the tests in the flaky group, because you can discover other bugs if the test fails on a step other than the one known to be flaky. The results of this separate group would be reviewed with each test run, but a failure in this group would not serve as a gating criteria to hold a release.<\/li>\n\n\n\n<li><strong>Delete the test.<\/strong> If the test has never found critical bugs and isn\u2019t covering an important user path, you may want to reconsider whether you actually need the test.<\/li>\n<\/ul>\n\n\n\n<p>Regardless of how you handle flaky tests, it\u2019s important to keep track of which tests produce inconsistent results, how you handled each failed test result, and the reason for the test failure whenever possible. Documenting each flaky test and what you did about it helps you and your team maintain faith in the test suite and develop your own best practices to prevent flakiness. It also helps you notice recurring patterns that could potentially be resolved.&nbsp;<\/p>\n\n\n\n<p>For instance, if your app is loading slowly in the test environment, consider upgrading your test machines and the associated components.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_Easiest_Way_to_Find_the_Root_Cause_of_Flaky_Tests\"><\/span>The Easiest Way to Find the Root Cause of Flaky Tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>To find the root cause of flakiness, you have to determine what\u2019s changing between test runs. But this can be a tedious task without the right tools.&nbsp;<\/p>\n\n\n\n<p>With most open-source and code-based testing tools, you\u2019ll end up sorting through lines of code in order to understand why some tests pass and others fail, which can be very time-consuming. If testing is normally handled by non-technical QA team members, the task of discovering why each test failed will have to be handled by someone outside the QA team \u2014 in most cases, a developer. This means finding the root cause of flaky tests could create a bottleneck in the software development lifecycle.&nbsp;<\/p>\n\n\n\n<p>Rainforest QA solves this problem by making it much easier and faster for anyone to understand why a test failed, providing video replays and detailed test reports for every test.&nbsp;<\/p>\n\n\n\n<p><a href=\"https:\/\/www.rainforestqa.com\/blog\/automated-ui-testing\" target=\"_blank\" rel=\"noopener\">Instead of using code to test code<\/a>, Rainforest QA uses an intuitive visual editor to create test cases. To write or edit any test step, you choose an action (such as \u201cclick\u201d or \u201cfill\u201d), then click-and-drag the mouse to take a screenshot of the element you want to apply the action to.&nbsp;<\/p>\n\n\n\n<p>Looking at the set of steps in the screenshot below, anyone can follow along and understand what\u2019s happening in the test:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61df4c8f0af16d2571381640_rainforest-signup-flow.png\" alt=\"Rainforest Signup Flow Example Steps\"\/><\/figure>\n\n\n\n<p>And if a test fails, the test step that failed during a test run will be highlighted in red along with a brief message describing the failure:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61df4c727960312e7e16cdfd_microsoft-test.png\" alt=\"An example of a failed test in Rainforest QA\"\/><\/figure>\n\n\n\n<p>For failures that have a less obvious cause (as is often the case with flaky tests), you can investigate further with:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Video replays:<\/strong> Rainforest QA records a video of every test, whether it passes or fails. You can compare each test run to quickly see if anything in the app UI changed between runs. Video replays also show you exactly how the failure would\u2019ve appeared to a real user, which can help you decide how important it is to fix the failure.&nbsp;<\/li>\n\n\n\n<li><strong>\u201cInvestigate Action\u201d:<\/strong> Clicking \u201cInvestigate Action\u201d for any step allows you to take a closer look at how the step was performed. You can see the original element the test was searching for, the closest match the test found, what percent of a match it was, and more.&nbsp;<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61b251f89981e91309926115_quality-assurance-metrics-5.gif\" alt=\"A GIF example of a failed action in Rainforest QA\"\/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Behind-the-scenes data.<\/strong> Each video recording also captures HTTP logs, browser logs and settings, network traffic, and more. These details are often the key to identifying failures caused by environmental hiccups.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>If the root cause of the failure is an actual bug, Rainforest QA offers a Jira integration so you can automatically create a ticket for the development team. The ticket includes the failed test steps, a screenshot of the failed test step, HTTP logs, and a link to the full test results and video recording in Rainforest. Rainforest also integrates with Slack and Microsoft Teams, so you can get instant notifications of any test failure.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Best_Practices_to_Prevent_Flaky_Tests\"><\/span>Best Practices to Prevent Flaky Tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Although it\u2019s difficult to completely eliminate flaky tests in automated testing, there are ways to minimize the number of flaky tests you run into:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Configure automatic retries.<\/strong>&nbsp; As we mentioned above, one of the most common ways to handle flaky tests is to rerun them. The automatic retry feature in Rainforest QA automates this task for you so there are fewer interruptions to your workflow. In Rainforest, you can adjust settings so that each failed test is rerun until it passes, or up to the number of retries you define:<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61df6bdd08451a34fddad71a_2022-01-12_11-06-03.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>\u200d<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Execute tests using virtual machines instead of real devices. <\/strong>Rainforest QA uses virtual machines for all test execution because it creates a more controlled test environment. When using real devices, many additional configuration factors could be introduced that could cause test flakiness. Unless you have access to all of the devices used for testing, it could be very difficult to discover what caused the test to fail. <strong><br>\u200d<\/strong><\/li>\n\n\n\n<li><strong>Throttle your test suite.<\/strong> You can throttle your Rainforest test suite so that fewer tests run at the same time. If your test environment or test data is set up to only handle a few users at a time, this could help minimize test failures due to slow load times caused by user concurrency overload.<\/li>\n\n\n\n<li><strong>Adjust test action wait times and timeouts. <\/strong>By default, Rainforest QA puts a two-second delay between test actions to allow an app or web page to finish loading. If the test is unable to locate the element, it will wait an additional two minutes before marking that step as failed. These wait times can be changed in your settings, and you can also add additional wait times to individual test steps.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>Finally, good test data management can help mitigate flaky tests. As we mentioned above, if multiple tests use the same set of user data, then running tests at the same time could create collisions. To avoid these collisions, you have a few options:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create a unique set of users if the order of test runs is causing flakiness.<br>\u200d<\/li>\n\n\n\n<li>Include steps at the beginning of each test to create a new user with every test run. <em>(Rainforest makes this option easy by offering a library of random and unique names, emails, passwords, etc.)<\/em>&nbsp;&nbsp;<br>\u200d<\/li>\n\n\n\n<li>Run tests that use the same user account in asynchronous test runs.<\/li>\n<\/ul>\n\n\n\n<p>If you have multiple tests that use the same user account, you\u2019ll want to use reset protocols. Whether you reset the testing environment before every test or add steps to your tests to revert to a default state, resetting is important for reducing inconsistent test results.&nbsp;<\/p>\n\n\n\n<p>For example, let\u2019s say you have a test that verifies a user can change their username. If the username isn\u2019t subsequently reset to the original username, every other test that uses that username will fail.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Easily_Handle_Flaky_Tests_with_Rainforest_QA\"><\/span>Easily Handle Flaky Tests with Rainforest QA<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>With Rainforest QA, anyone can quickly figure out if a test is flaky or permanently broken, or if the software has a bug. It\u2019s a scalable, all-in-one quality assurance solution that\u2019s appropriate for small teams just getting started with automated testing or QA-mature teams regularly running 500+ automated software tests as part of their CI\/CD pipeline.<\/p>\n\n\n\n<p><a href=\"https:\/\/www.rainforestqa.com\/talk-to-sales\">Talk to us<\/a> about setting up a Rainforest plan that fits your needs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Flaky tests can be frustrating and detrimental to your testing efforts. This post covers what causes flaky tests and what you can do about about it.<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-353","post","type-post","status-publish","format-standard","hentry","category-test-automation"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/353","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/comments?post=353"}],"version-history":[{"count":9,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/353\/revisions"}],"predecessor-version":[{"id":3077,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/posts\/353\/revisions\/3077"}],"wp:attachment":[{"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/media?parent=353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/categories?post=353"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rainforestqa.com\/blog\/wp-json\/wp\/v2\/tags?post=353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}