How Code-Structure-Aware Methods Strengthen Large Language Models for Programming

Introduction

Large language models have become core tools for code generation, completion, repair, and migration, but treating code merely as a linear token sequence ignores its strict syntactic and semantic constraints. The thesis argues that reliable code models need to learn not only the next token but also the legal locations, complete units, and semantic rules inherent in program structure.

ADELT: Transpilation Between Deep Learning Frameworks

ADELT tackles code translation across frameworks such as PyTorch, Keras, and MXNet. The problem is split into two sub‑tasks: structural migration of the code skeleton and API keyword mapping. ADELT first normalizes source code, extracts a skeleton with placeholders for API names, and uses few‑shot prompting to translate the skeleton. API keywords are aligned via a shared embedding space learned by PyBERT and a dictionary lookup, decoupling structural conversion from API alignment.

Figure 2: ADELT pipeline—code normalization, skeleton extraction, language‑model‑driven skeleton migration, and API dictionary mapping.

SAFIM: Syntax‑Aware Fill‑in‑the‑Middle Evaluation

Standard code‑completion benchmarks (HumanEval, MBPP) evaluate whole‑function generation and miss fine‑grained structural errors. SAFIM creates three Fill‑in‑the‑Middle tasks that require the model to restore masked code fragments while preserving their role in the AST: algorithm block completion, control‑flow completion (e.g., if, while, for), and API call completion. Experiments on 15 models show that larger parameter counts do not guarantee better structural understanding; pre‑training objectives, data quality, and inclusion of FIM targets significantly affect performance.

Figure 5: Average Pass@1 on SAFIM versus model size, colored by pre‑training paradigm.

AST‑T5: Structure‑Aware Pre‑training for Code Generation and Understanding

Typical T5 pre‑training masks random token spans, which can cut through variable names or statements, misaligning the task with real code editing. AST‑T5 replaces random spans with whole AST sub‑trees (subtree corruption) and segments long inputs along function or class boundaries. This forces the model to reconstruct semantically meaningful units such as complete expressions or statements.

Figure 6: Comparison of random span masking and AST‑aware subtree masking.

Evaluations on code generation, translation, repair, and understanding tasks show consistent gains, especially for translation and repair where preserving structure is critical.

AST‑FIM: Structure‑Aware Fill‑in‑the‑Middle Pre‑training for Code Completion

Standard Fill‑in‑the‑Middle (FIM) masks random character or token sequences, which rarely correspond to realistic edit operations. AST‑FIM masks complete AST sub‑trees, aligning the pre‑training objective with developer edits such as inserting a whole statement block or modifying an API call. The authors also introduce Real‑FIM‑Eval, a benchmark derived from real Git commits that distinguishes Add (code insertion) and Edit (code modification) scenarios.

Figure 8: AST‑FIM masks versus random character FIM masks.

AST‑FIM outperforms random FIM on both SAFIM and Real‑FIM‑Eval across model sizes and languages, while retaining left‑to‑right generation ability. Limitations include reliance on perplexity metrics for Real‑FIM‑Eval and modest gains for pure left‑to‑right generation.

Conclusion and Future Work

The thesis establishes a research pipeline: ADELT proves the benefit of decoupling structural migration from API alignment; SAFIM provides a fine‑grained evaluation that reveals structural weaknesses; AST‑T5 shows how AST‑aware pre‑training improves multiple downstream tasks; and AST‑FIM extends structure awareness to decoder‑only models and realistic edit scenarios. Future directions include richer structural modeling beyond AST (cross‑file dependencies, build configurations, runtime traces) and more authentic evaluation benchmarks that cover multi‑file completion, long‑term code evolution, dependency upgrades, and security or maintainability assessments.