Our new research shows AI adoption is high across software testing workflows. But AI isn’t (yet) paying off in ways that matter to software teams using open-source testing frameworks like Selenium, Cypress, and Playwright.
Open-source teams using AI are still spending just as much — if not more — time on painful test writing and maintenance tasks than ones not using AI.
AI has a high adoption rate in open-source testing, but isn’t correlated with more productivity
According to our survey of over 600 software developers in the U.S., Canada, the U.K., and Australia, 74.6% of teams using open source frameworks for test automation are using AI to assist with test writing and/or maintenance.
Test writing and maintenance, of course, are the most time-consuming and painful parts of the ongoing test automation process.
”To be able to maintain automated tests, especially with a small dev team, just takes time.” – Software Engineering Lead
“In the past, we gave up on testing the front end with open source because it was too difficult to maintain and the tests were very often broken.” – Engineering Manager
“I also understand all the pain with developers maintaining the tests and the impact on velocity that it can have.” – Director of Engineering
But open-source teams using AI aren’t saving time on these tedious tasks. In fact, it looks like teams using open-source frameworks who use AI for test creation and/or maintenance actually spend slightly more time on these tasks than teams who don’t use AI.
This might come as a surprise to anyone who’s bought into the promise of AI to improve productivity on software teams.
But it echoes the results of a recent study by Uplevel showing that developers using GitHub Copilot currently experience no clear productivity gains. (And do have to deal with more bugs.)
One possibility could explain these results and undersell AI’s benefits: teams who choose to adopt AI may have complex products that inherently require more work than average to keep automated test suites updated. In which case, maybe AI is saving those teams some from spending even more time on test suite upkeep, and that effect is hiding in the results.
But given the large adoption rate of AI for test creation and maintenance among teams who use open-source automation frameworks, this scenario seems unlikely. It’s more likely that AI just isn’t currently delivering velocity benefits for these recurring test automation tasks.
AI isn’t a complete wash for teams using open source — it helps small teams keep automated test suites up to date
While AI hasn’t created any clear wins for open-source teams in the time they spend on maintenance, AI might still be helpful for small teams working with open source.
When an automated test suite isn’t kept up to date, it can’t reliably do its job: catching bugs and other issues before customers do. But test maintenance is painful and time-consuming — especially with open-source frameworks, as we’ve seen in the data — so some teams struggle to keep up with it.
Test maintenance is a practice that requires clear definitions of ownership and enforced team policies. So, the smallest dev teams — the ones least likely to have formal policies and procedures in place — are understandably the least likely to keep their automated test suites updated. The time required to work with open-source frameworks just exacerbates the problem.
The survey’s results show that small teams using open-source testing frameworks who use AI for test creation and/or maintenance are more likely to successfully keep their test suites updated and reliable.
But as teams grow, there’s not much difference between using AI and not using AI, as the vast majority of teams report keeping their tests up to date.
The implications for your team
So why is AI underdelivering in important areas of test writing and maintenance?
The results don’t necessarily suggest AI is a dud for teams using open-source frameworks. Implementations of AI across open-source testing setups include various off-the-shelf and custom-made solutions. Some solutions are certainly more effective than others, but the results suggest that teams are still trying to find the ones that work. And there are probably areas in which the technology still needs to improve.
But you don’t have to wait for AI to eventually reduce the burden of creating and maintaining tests in open source frameworks.
Our data show that teams using no-code to automate their E2E tests are spending a lot less time on test maintenance tasks. They give themselves more time and resources to dedicate to shipping code.
But not all no- and low-code tools are built the same — some are complex, require training, and have their own “language” to learn. It’s going to be difficult to save time using any tool like that.
To increase velocity and free up your dev team’s time, the key is to adopt a no-code testing tool that’s intuitive, requiring little or no training. It should produce test scripts in plain English, and deeply integrate AI in ways that demonstrably save time on the most annoying of maintenance tasks.
Learn more in our report, The State of Software Test Automation in the Age of AI.
