Using ChatGPT to Enhance Software Functional Testing: A Practical Exploration

ChatGPT (Chat Generative Pre‑trained Transformer) is an AI‑driven natural language processing tool released by OpenAI on November 30, 2022, reflecting the rapid shift of AI from perception to cognition.

Like many users, the author experimented with ChatGPT by asking a few everyday and technical questions and received meaningful, insightful answers that sparked further curiosity.

Continuing this curiosity, the author, a software test engineer referred to as "A", explored how ChatGPT could assist in designing test ideas for a software testing scenario.

The product manager’s requirement is: to allow users to select important personal data from a mobile contacts list and export it to a USB drive for backup.

Test engineer A took on this task and decided to use ChatGPT to help flesh out a more comprehensive test approach.

Question 1: "For the data in the mobile contacts that can be exported via USB, please provide test ideas for the export function."

A saw that ChatGPT gave seven points, but felt some important aspects—such as verification of exported data correctness—were missing, even though the sixth point about error handling vaguely touched on it.

Realising that unsatisfactory answers can be improved by refining the prompt, A issued a second request.

Question 2: "The data in the mobile contacts can be exported via USB; users can select data and click the ‘Export’ button. Please provide test ideas for the export function."

This time A received a richer set of ideas, including the crucial "data integrity testing", UI testing, export speed, performance with large data sets, and handling of exceptional cases.

Considering that USB drives come in different file system formats (FAT, FAT32, NTFS, etc.), A refined the question to explicitly mention "USB" before asking again.

Question 3: "The data in the mobile contacts can be exported via USB; users can select data and click the ‘Export’ button. Please provide test ideas for the USB export function (asked around 10 am)."

The answer differed: two points about compatibility and performance disappeared, showing how a tiny wording change can affect the coverage.

Later in the afternoon (around 3 pm), A asked the same question again:

Question 4: "The data in the mobile contacts can be exported via USB; users can select data and click the ‘Export’ button. Please provide test ideas for the USB export function (same wording as Question 3, asked in the afternoon)."

This time the response contained only six points, missing four that were present earlier, leading A to wonder why identical queries at different times yield different results.

A concluded that the same question can produce varying answers depending on timing, indicating a degree of randomness in ChatGPT’s responses.

Functional Testing Exploration Summary

1. The more precise the question posed to ChatGPT, the more accurate the answer.

2. While ChatGPT understands text, subtle differences in Chinese wording can lead to different interpretations and results.

3. Identical questions asked at different times may produce random variations, which can feel unreliable.

Overall, ChatGPT can assist in supplementing test ideas and points, making test case design more comprehensive, but its answers are not fully reliable and may exhibit randomness.

Promotional Note: Teacher Qiao Liang is offering a limited‑time discount on the video course "Continuous Deployment Training Camp (Python Edition)". The course covers building‑test‑deploy pipelines, continuous deployment practices, monitoring, gradual roll‑outs, and database schema changes.

(Scan the QR code to subscribe)