Applying Large Language Models to Software Engineering: Challenges, Cross‑File Editing Issues, Bug‑Fixing Evaluation, and SWE‑Bench Results

The session introduces the topic "Artificial Intelligence in Software Engineering Processes" and outlines four main discussion points: challenges faced by LLMs in real software development, cross‑file editing problems, AI bug‑fixing evaluation methods, and the latest language model capability results up to June 2024.

LLM challenges in software development

Long context handling: Understanding and processing large codebases with extensive files.

Cross‑file editing: Modifying code across multiple files and functions, not just a single location.

Problem description comprehension: Accurately interpreting issue descriptions and translating them into concrete code changes.

Diverse problem types: Dealing with unique characteristics and challenges of each problem.

Adapting to new problems: Solving issues not seen in training data.

Interaction with execution environment: Verifying solutions by running tests.

Generating reliable solutions: Ensuring patches pass all relevant tests.

Large codebase handling: Managing complex dependencies and interactions.

Understanding code style and logic: Producing changes that conform to existing conventions.

Cross‑file editing software‑engineering issues

Modifying functions across multiple files.

Modifying classes across multiple files.

Changing code structure in several files.

Handling dependencies between files.

Fixing bugs that span multiple files.

Adding or modifying features that require changes in several files.

Refactoring code across files to improve readability and maintainability.

Evaluation method and steps

The SWE‑bench benchmark is used to assess language model performance on cross‑file editing tasks. Models receive a problem description and a full code repository, and must produce patches that modify the code. Success is measured by the percentage of problems solved, i.e., patches that apply cleanly and pass all tests.

SWE‑bench contains 2,294 real‑world software‑engineering problems from 12 GitHub repositories. Because full SWE‑bench evaluation is costly, a reduced subset called SWE‑bench Lite (300 instances) is provided, focusing on functional error‑fixing and excluding instances with images, external links, short descriptions, multi‑file edits, large patches, file creation/deletion, or error‑message checks.

Organization‑wide model capability results

Results for both the full SWE‑bench and the Lite version are presented, showing the performance of various large language models on these software‑engineering tasks.