Understanding Test Fragility: Components, Classification, and Diagnosis (Part 2)

This is the second article in the "Test Fragility" series. The first article introduced the four components required to run test cases and possible reasons for test fragility. This article discusses classification techniques and diagnostic methods for those reasons.

Recall the four components that may cause fragility:

Test itself

Test execution framework

Application or System Under Test (SUT) and the services/libraries it depends on

Operating system, hardware, and network that SUT and the framework rely on

Below is a diagram summarising this point.

We now discuss, for each component, the possible causes of instability, classification hints, and mitigation methods.

Component 1 – Test Cases

Test cases themselves can be unstable. This may include test data, execution flow, prerequisite setup, and the initial state of other dependencies.

Component 2 – Test Execution Framework

An unreliable test execution framework may introduce fragility.

Component 3 – Application or SUT and its dependent services/libraries

The application under test (or SUT) can be a source of fragility. Moreover, the application may depend on many services, each of which can introduce its own fragilities.

Component 4 – Underlying OS, hardware, and network

The underlying hardware and operating system may also be sources of test fragility.

Summary

From the many failures observed, test fragility in automation is a considerable challenge. This article outlined the components involved in running tests and the types of fragilities that can arise, serving as a checklist for diagnosing and fixing flaky test cases.