Why Smart Test‑Script Generators Won’t Fix Automation Testing Woes

Some companies’ automation testing has become a target of criticism from leadership for several reasons.

Hundreds of outsourced staff are hired solely to write and debug automation scripts, creating a massive external team.

The proportion of defects discovered by automation testing is too low compared with the total number of version defects.

In short, the return on investment for automation testing does not meet expectations.

Nevertheless, automation testing cannot be abandoned; manual execution of massive test cases is prohibitively costly, making the improvement of automation efficiency an urgent task.

Some propose the following “solutions”:

Intelligently generate test scripts to lower the barrier of writing and debugging automation scripts.

Intelligently devise test strategies and select test‑case scripts.

Intelligently analyse test reports, automatically creating defect tickets when the tool determines a product issue.

But do these measures really solve the core problems?

The author argues they do not. Simply adding tools does not address the underlying issues and may even reduce efficiency.

Using “intelligent script generation” as an example, the current automation framework at the referenced company is a Ruby‑based “third‑generation” system where test cases are separated from the underlying support scripts. Test designers write clear, structured scripts using ActionWords, while automation engineers maintain the ActionWord libraries.

Writing test scripts in Ruby is a basic skill for a competent test engineer. Ruby is far simpler than C/C++, yet it is still considered a high barrier by some, which is puzzling.

The prevailing practice of having test engineers design cases that are then handed to outsourced teams for translation into scripts is inefficient. The correct approach is for test engineers—or even full‑stack developers—to write the scripts themselves, as exemplified by Microsoft’s Visual Studio team.

Is “intelligent script generation” feasible?

Using a DSL (Domain‑Specific Language) to describe executable test cases is not the same as “intelligent generation”; it still relies on ActionWords and requires knowledge of their syntax and parameters.

A genuine “intelligent generator” would accept a requirements document and output high‑quality, executable scripts. Anything short of that is merely hype.

Repackaging a “model‑to‑code” approach as a revolutionary technology is not new; similar ideas appeared in the 1990s but have never been adopted by major software vendors.

Improving the IDE for script development—better parameter hints, real‑time help, error and refactoring suggestions—mirrors the usability of Excel formulas and is a more practical direction.

Analyzing execution results is labor‑intensive mainly because of poor script quality and mismatches between product specifications and scripts. Enhancing script quality through better management is the ultimate solution; a script error should indicate a product bug, not a test‑script flaw.

Automation testing is a software system whose complexity must be kept low. Adding ever more complex tools to compensate for inefficiencies only worsens the situation.

To address the problems, the author suggests focusing on three areas:

Strengthen management of outsourced teams and control costs versus benefits.

Raise the skill level and output efficiency of test engineers, encouraging them to write and maintain their own scripts.

Implement rigorous quality control and maintenance for the automation suite, including script portability, fault tolerance, readability, and an aging/retirement mechanism to prevent unchecked growth of the script repository.

Automation testing, like farming, requires diligent management; without it, weeds (inefficiencies) proliferate.

Finally, the common belief that automation should directly discover version defects is challenged; many experts argue that automation’s primary role is not defect detection.