CodeFuseEval: An Enterprise‑Level Multi‑Task Benchmark for Evaluating Code Large Models

2023 marked a rapid expansion of large language models (LLMs), with more than 130 domestic and 138 foreign models released. As LLMs become increasingly integrated into conversational products, evaluating their code‑generation capabilities and ensuring outputs are useful, harmless, and truthful has become a critical challenge.

The CodeFuseEval benchmark was built on the experience of evaluating Ant Code Fuse series models and crowdsourced feedback. It aims to provide an enterprise‑grade, multi‑type code evaluation suite that covers a wide range of programming tasks (code completion, generation, translation, test‑case generation, bug fixing, etc.) across several languages (Java, C++, JavaScript, Python, etc.).

Evaluation Content – Beyond traditional NLP metrics, code‑model assessment must consider technical abilities (correctness, semantic accuracy, readability) and service‑level factors (stability, openness, user experience). The benchmark defines multiple dimensions such as skill, efficiency, robustness, and stability.

Evaluation Methods – Both objective (quantitative metrics on benchmark datasets) and subjective (human expert review) approaches are used. Objective metrics include pass@k, BLEU, CodeBLEU, bluert, and other similarity scores. Subjective evaluation involves expert panels assessing code quality, documentation, and safety.

Benchmark Datasets – CodeFuseEval incorporates existing datasets (HumanEval‑x, MBPP, DS1000) and adds proprietary crowdsourced data to cover 6 programming languages and over 6,300 tasks. The dataset is continuously expanded with open‑source and internal data, ensuring coverage of real‑world scenarios.

Framework – An engineering‑focused evaluation pipeline was developed, featuring language‑specific execution containers, automated scoring, and a freshness strategy for benchmark updates. The framework supports zero‑shot, few‑shot, and chain‑of‑thought prompting strategies, with zero‑shot being the dominant approach for code generation.

Sample Code Snippet (evaluation entry)

{
  "task_id": "Python/177",
  "prompt": "import java.io.*;\nimport java.lang.*;\nimport java.util.*;\nimport java.math.*;\n\nclass ProdSquare {\n    /**\n     * Write a Java function to check whether the given number can be represented by product of two squares or not.\n     */\n    public static Boolean prodSquare(int n) {\n        {\n            int a = 1;\n            int b = 1;\n            for (int i = 1; i <= n; i++) {\n                if (a * i < 0) {\n                    b = b * i;\n                } else {\n                    a = a * i;\n                }\n            }\n            return b == 1;\n        }\n    }\n}",
  "canonical_solution": "def prod_Square(n):\n    for i in range(2, n+1):\n        if i*i < n+1:\n            for j in range(2, n+1):\n                if (i*i*j*j) == n:\n                    return True\n    return False",
  "test": ["assert prod_Square(25) == False", "assert prod_Square(30) == False", "assert prod_Square(16) == True"],
  "desc_en": "Write a python function to check whether the given number can be represented by product of two squares or not.",
  "Difficulty": "mbpp",
  "desc_cn": "写一个函数来检查给定的数字是否可以用两个正方形的乘积来表示。"
}

The benchmark also provides visualizations of metric trends, multi‑task radar charts, and per‑model performance tables. All resources, including the dataset, evaluation scripts, and Docker images, are open‑source on GitHub (https://github.com/codefuse-ai/codefuse-evaluation) and ModelScope.

Future Outlook – Code model evaluation will continue to evolve toward richer multi‑dimensional assessments, integrating both technical and service capabilities. Ongoing work includes expanding task types, improving metric reliability, and maintaining an open, reproducible evaluation ecosystem.