What Drives Neural Code Intelligence? A Deep Dive into Models, Datasets, and Future Directions

“Programming is the art of telling another human being what one wants the computer to do.” — Donald Knuth

Overview

Neural Code Intelligence (NCI) uses deep learning to understand, generate, and optimize code, bridging natural language and programming languages. The field has rapidly attracted research and industry attention, with over 50 representative models, more than 20 task categories, and over 680 related works.

Evolution of Code Models

1. Neural Language Modeling era

Early attempts used RNN/CNN architectures, incorporating code structure such as abstract syntax trees, data flow, and control flow. Techniques like code2vec and code2seq produced code embeddings that captured semantic and structural information.

2. Code Pre‑trained Models (CodePTMs) era

Following the success of pre‑trained language models in NLP, models such as CodeBERT and CodeT5 adapted Transformer architectures for code, combining structural modeling with large‑scale pre‑training and fine‑tuning.

3. Large Language Model (LLM) era

With the rise of GPT‑3, PaLM, etc., code‑focused LLMs like Codex, CodeGen, and StarCoder emerged, shifting from task‑specific fine‑tuning to prompt‑learning and in‑context learning, expanding applications to reasoning, mathematics, and NLP tasks.

Learning Paradigm Shifts

Model development mirrors NLP: from neural approaches for single tasks, to pre‑training + fine‑tuning for multiple tasks, to prompt‑learning dominant LLMs. This shift also broadens code intelligence from traditional code‑related tasks to numeric reasoning, symbolic solving, and information extraction.

Datasets and Benchmarks

The survey reviews corpora such as CodeSearchNet and The Stack, describing their characteristics. It also summarizes common benchmarks for clone detection, defect detection, code translation/repair, and generation, and compares performance of representative CodeLLMs.

Cross‑Domain Collaboration

Beyond standard code tasks, the paper discusses code‑assisted reasoning, code‑based training for mathematical ability, and using code as an intermediate representation to improve classic NLP tasks like information extraction.

Applications and Future Directions

Potential applications include software engineering (coding assistants, automated development), data‑driven decision making (Text2SQL, data science), agents (robot control, automation), and AI‑for‑Science (molecular generation, automated theorem proving). The authors outline open research problems in models, evaluation, efficiency, and cross‑domain integration.

Resources

The authors maintain a GitHub repository containing curated reading lists, tutorials, blogs, and other resources related to the survey.

GitHub: https://github.com/QiushiSun/NCISurvey