Understanding Automated Testing Frameworks and Their Application in Real‑Time Video SDKs

Dev for Dev, short for Developer for Developer, is a collaborative innovation activity launched by Agora and the RTC developer community that brings engineers together through technical sharing, project co‑creation, and interactive discussions to surface valuable technical content.

The author, Sun Yongnan, an engineer from Agora's audio‑video lab, presents an overview of automated testing frameworks, describing them as foundational infrastructure that provides an execution environment for automated test cases or scripts.

Automated testing frameworks offer numerous benefits, helping users develop, execute, and report test cases efficiently; they combine coding standards, test processes, best practices, modular structures, reporting mechanisms, and data injection into a cohesive system.

Popular framework types include:

Module‑based automation testing framework

Repository‑architecture automation testing framework

Data‑driven automation testing framework

Keyword‑driven automation testing framework

Black‑box hybrid automation testing framework

Behavior‑driven automation testing framework

In a black‑box framework, users only need to focus on inputs and expected outputs without concerning themselves with internal program structure.

Testing real‑time video SDKs differs from typical web or app testing: it relies heavily on custom automation tools that simulate network impairments, capture SDK logs, and generate objective test data, thereby reducing manual effort and improving efficiency.

Our current automation solution combines a CI cluster, testing tools, an automation framework, and a data‑platformized approach to support daily testing, release verification, and developer self‑testing.

Key modules include CI cluster + testing tools + automation framework + data platform, which underpin daily, release, and developer‑self testing activities.

Multiple test nodes support the entire video release business line, providing scalable and reliable execution.

Automation testing is not limited to executing tests; its closed‑loop advantages include easy input, comprehensive coverage, strong fault tolerance, and high output aggregation, gradually breaking through precision and efficiency bottlenecks.

To mitigate network‑induced data errors, we employ a funnel‑style re‑run filtering method, as illustrated below:

This iterative evaluation keeps error margins within acceptable limits, reducing the impact of network fluctuations on SDK test data.

After automation runs, we compare metrics across test versions and releases via objective reports, visualizing differences for precise quality assessment.

While current report views satisfy basic comparison needs, further data consolidation and business‑driven insights are required.

There is no official standard for testing frameworks; they are chosen based on concrete business requirements, integrating mainstream or custom solutions to provide quick, problem‑solving value for black‑box users.

FunTester original special collection Performance Testing Topics Java, Groovy, Go, Python Unit & White‑Box Testing FunTester Community Highlights Testing Theory Nuggets API Functional Testing Topics FunTester Video Topics Case Studies: Solutions, Bugs, Crawlers UI Automation Topics Testing Tools Topics -- By FunTester

阅读原文，跳转我的仓库地址