Exploring ChatGPT for Smart Contract Vulnerability Detection: Experiments and Analysis

With the rapid development of blockchain technology, smart contracts have become integral to many applications but are prone to vulnerabilities such as re‑entrancy and integer overflow. This paper presents a collaborative research effort by Ant Security and Sun Yat‑sen University that examines the use of ChatGPT for automated smart‑contract vulnerability detection.

Main Technical Findings

ChatGPT can identify several common vulnerabilities (re‑entrancy, integer overflow, access control, timestamp dependence, denial of service, unhandled exceptions, logic errors) and shows superior detection of integer overflow compared with some existing tools.

It can learn new vulnerability patterns from prompts and apply that knowledge in subsequent analyses.

By avoiding compilation, ChatGPT reduces human effort and can handle complex DApp projects with multiple files and external dependencies.

The model offers good time performance, improving audit efficiency.

Limitations include unstable outputs, errors on long contracts, and susceptibility to comment interference.

The authors conducted extensive experiments using the smartbugs‑curated dataset, comparing GPT‑3.5‑turbo and GPT‑4, single‑round versus multi‑round prompting, and benchmarking against tools such as Slither and Mythril. Results show that ChatGPT achieves competitive precision and recall for certain vulnerability classes (especially arithmetic issues) while lagging on others.

Key observations include token limits affecting long contracts, the impact of prompt design on stability, and the need for careful handling of output formats. The paper concludes that ChatGPT, particularly the newer GPT‑4 model, holds promise as an auxiliary tool for smart‑contract security auditing, though further improvements are required for reliable large‑scale deployment.