Building Test Effectiveness Evaluation Capability for Youku Client

This article shares Youku client's test effectiveness evaluation capability construction experience, focusing on how to assist testers in adjusting direction and strategy. The article discusses how to measure test effectiveness and what pitfalls and key points exist in building an evaluation system.

In the mobile internet wave, To C mobile applications iterate extremely fast, and application gameplay emerges endlessly, thus putting forward higher requirements for application quality assurance: rapid response, clear goals, and effective means. Quality assurance teams need to establish a series of processes, systems, or tools to assist in the implementation of testing, but regardless of what the testing side wants to establish, there is a clear principle throughout: effectiveness (since processes, systems, or tools are used for testing, it is also called test effectiveness).

Different teams define their own rules to measure test effectiveness, which almost cover everything, such as: Bug quantity and trends, test case review, test strategy review, or code coverage data, etc. Evaluating test effectiveness cannot be separated from the development implementation and project context, and testing has never existed independently; testing and development are inseparable.

Combining the current status of Youku's quality assurance system, we prioritize choosing the most direct and data-driven approach to evaluate test effectiveness: code test coverage. The article compares two tools: JaCoCo and Youku's internal coverage collection tool, analyzing their pros and cons in terms of code intrusion, performance loss, suitability for gray release, and statistical range.

The overall architecture and process for obtaining class loading information is described, including obtaining all ClassLoader's ClassTable objects through the Native layer, getting the class name list through ClassTable objects, and determining whether classes are loaded through the Lookup method of ClassTable in libart.so.

Sample results are shown through Youku client testing platform, including summary results, TOP module class coverage comparison results, and module detail comparison data. The technical implementation uses Python3 for data download and analysis, with analysis of 300,000 zip files containing 100,000-120,000 classes taking about 1 hour to complete.

The results and benefits include establishing Android-side main and guest module/class dimension test effectiveness evaluation means, using online code heat analysis mechanism for daily automated testing and gray testing, effectively collecting test effectiveness data for multiple versions of Youku main and guest, and effectively optimizing test coverage for business modules.

Future plans include promoting test effectiveness analysis capabilities to Youku main and guest TOP modules, increasing single-module coverage by at least 10%, full-link automation, combining Youku main and guest technology atlas for coverage visualization output, and more.