Code Model Evaluation Framework and the CodeFuseEval Benchmark Overview

2023 marked a rapid expansion of large language models, with over 130 Chinese and 138 foreign models released, highlighting the importance of systematic model capability evaluation, especially for code‑focused models where output uncertainty and emergent abilities pose significant challenges.

The evaluation of code models must go beyond traditional system metrics, covering technical abilities such as code generation, contextual understanding, and service‑level aspects like stability and usability. Two main evaluation approaches are used: objective metrics based on benchmark datasets and subjective assessments involving expert reviewers.

Evaluation methods are categorized by execution style (automated, manual, model‑based) and prompting strategy (zero‑shot, few‑shot, chain‑of‑thought). Most code generation assessments currently rely on zero‑shot prompts.

CodeFuseEval is an enterprise‑grade, multi‑task benchmark built on open‑source datasets like HumanEval‑x, MBPP, and DS1000, extending them to cover six programming languages (Java, C++, JavaScript, Python, etc.) and over 6,300 tasks, including code completion, translation, test case generation, and bug fixing.

Example task definition from the benchmark:

{
  "task_id": "Python/177",
  "prompt": "import java.io.*;\nimport java.lang.*;\nimport java.util.*;\nimport java.math.*;\n\nclass ProdSquare {\n    /**\n     * Write a Java function to check whether the given number can be represented by product of two squares or not.\n     */\n    public static Boolean prodSquare(int n) {\n        int a = 1;\n        int b = 1;\n        for (int i = 1; i <= n; i++) {\n            if (a * i < 0) {\n                b = b * i;\n            } else {\n                a = a * i;\n            }\n        }\n        return b == 1;\n    }\n}"
  "canonical_solution": "def prod_Square(n):\n    for i in range(2, n+1):\n        if i*i < n+1:\n            for j in range(2, n+1):\n                if (i*i*j*j) == n:\n                    return True\n    return False",
  "test": ["assert prod_Square(25) == False", "assert prod_Square(30) == False", "assert prod_Square(16) == True"],
  "desc_en": "Write a python function to check whether the given number can be represented by product of two squares or not.",
  "Difficulty": "mbpp",
  "desc_cn": "写一个函数来检查给定的数字是否可以用两个正方形的乘积来表示。"
}

The evaluation framework incorporates an adaptation layer, multi‑language execution containers, and benchmark freshness strategies to ensure reliable, repeatable results across diverse model architectures and deployment environments.

Results are visualised using bar charts for single‑task multi‑model comparisons and radar charts for multi‑task assessments, with metrics such as pass@k, BLEU, CodeBLEU, and semantic similarity scores (bluert).

Future work will continue to expand the benchmark to cover four evaluation dimensions—skill, efficiency, robustness, and stability—and to open new multi‑type code tasks, fostering higher‑quality code generation for enterprise scenarios.

References to related research papers and datasets are provided, and the benchmark source code and datasets are publicly available on GitHub and ModelScope.