Test automation can be an effective piece of the QA strategy puzzle for any team looking to optimize their testing process. Done well, it can speed up software testing cycles, improve the efficiency of QA teams, and ultimately increase product quality. But contrary to popular belief, test automation is not a magical solution to achieving cheap, lightning-fast, effortless QA results.
In a poll survey conducted during our webinar on the cost of test automation, 58% of quality owners reported the cost of test automation to be significantly more expensive than initially forecasted. Rainforest CIO Derek Choy explained two major reasons that a team's test automation initiative can be pricier than they expected.
Many QA teams have the impression that automation is relatively low cost and efficient. But in practice, test automation can actually be quite expensive. Derek stressed that in order to forecast and evaluate the true cost of test automation, teams must take three cost areas into consideration: test writing costs, environment maintenance costs, and test maintenance costs. This final area -- automated test maintenance -- is often overlooked, but in the long run represents a large amount of the resources needed for test automation to stay effective.
“One thing people should not take lightly and forget, is the cost of maintaining automated tests.”
Automated tests require technical expertise to continually manage tests and the environments in which they run. Test automation experts are highly sought-after, and often demand competitive salaries. As a result, hiring out a team to continuously keep automated tests aligned with the code can be a significant expense.
Application feature changes are made sprint-over-sprint. When these changes are directly connected to automated tests, the tests can break. Additionally, when application changes are made to features that indirectly connect with automated tests, the tests can break. When evaluating the cost of incorporating automation into your QA strategy, it’s extremely important to consider how frequently automated tests will need updating to stay useful.
Along with the cost of technical talent to write and maintain automated tests and their environments, there’s also the cost of delaying new software releases. A robust automation strategy essentially creates an additional codebase that your team must maintain. This is especially problematic if you have developers maintaining your automation test suite, as the time they use to refactor broken automated tests is time they can't spend creating new features.
Triaging failed tests can also create delays. Automated tests can fail because there is a bug, but they can also fail because the test itself is broken. If your team is in the middle of launching a release and your automated tests fail, you're not likely to continue with the release until the source of the failure is identified, because there might be problems with software quality.
While test automation can be an excellent tool for QA teams to streamline testing, the cost of maintaining automated tests is not trivial. The key to realizing a high return on your investment in test automation is to determine which tests cases are a great fit for automation, and which are best suited for manual testing.
Listen on-demand to Crack The Automation Code: Create a Balanced Test Automation Strategy to learn the best tools and processes to complement your automation strategy.
With this practical guide, understand the difference between manual vs automated testing and when to use each method.
Follow these 5 steps to create an automated testing strategy that answers the who, what, when, why, and how of software test automation.
Testing the visual layer of your app is more reliable than testing the underlying code, especially for automated UI testing.
Learn how Signagelive built 500 tests in record time and dramatically sped up development cycles with automated tests.