Unlock Faster Automated Testing with a Powerful JSON Assertion Engine

Background

In automated testing, the assertion component—often called the “gatekeeper”—verifies that system output matches expectations, but traditional assertion methods struggle with complex JSON data, leading to low efficiency, difficult maintenance, and poor readability.

Low efficiency: Manual assertion code is labor‑intensive and error‑prone for massive test data.

Maintenance difficulty: Interface changes require extensive manual updates to assertion code.

Poor readability: Verbose assertion code hampers understanding of test cases.

Architecture Overview

The JSON assertion framework consists of several modules:

Assertion engine: Receives test data, applies rules and ignore patterns, performs JSON comparison, and outputs results. It can be invoked directly from automation scripts.

Pre‑processing: Cleans, transforms, and sorts input JSON to ensure consistent format before comparison.

Post‑processing: Formats, filters, and aggregates comparison results into a readable report.

Data management: Handles input/output data, rule sets, and ignore patterns, supporting expected‑result generation and visual data upload.

Core comparison: Deeply compares pre‑processed JSON objects, identifies differences, and produces detailed diff results.

Visualization page: Provides a graphical interface for entering JSON, configuring rules, and viewing comparison outcomes.

Core Entry

KuMatchers serves as the entry point of the assertion toolset, wrapping a rich set of matchers that integrate seamlessly with various automation frameworks.

// Compare with a file in classpath
actual KuMatchers.equalToJsonInClassPath(expectFile)

// Compare with an object
actual KuMatchers.equalToJson(expect)
    .withLabel("Report label")
    .ignorePathsRegexes(["/test1"])
    .withDecimalPlaces(2)
    .markAsAsyncTest()
    .enableStrictMode()
    .setArraySorting([
        "/path1": ["key1","key2"],
        "/path2": ["key1","key2"],
        "/path3": []
    ])

Global configuration controls assertion behavior:

# Auto‑create expected result file
matcher.json.compare.expect.auto.create=true

# Overwrite expected result file (use with caution)
matcher.json.compare.expect.overwrite=false

# Overwrite on success
matcher.json.compare.expect.overwrite.if.success=false

# Overwrite on failure
matcher.json.compare.expect.overwrite.if.fail=false

# Strict mode (new fields cause failure)
matcher.json.compare.strict=false

# Global ignore‑path regex (applies to all tests)
# matcher.json.compare.ignore.regex=/b, /subModels/.*/id

Rule Design

Traditional boolean‑based assertions limit expressiveness. The JSON assertion adopts a visual rule format based on JSON Patch operations (add, remove, replace) and JSON Pointer paths.

add: Insert a value at the specified path.

remove: Delete the value at the specified path.

replace: Replace the value at the specified path with a new one.

Assertion result is false when the generated rule list is non‑empty, otherwise true.

[
  {"op":"add","path":"/0/a","value":1},
  {"op":"remove","path":"/0/a","value":1},
  {"op":"replace","path":"/0/a","toPath":"/0/a","value":2,"toValue":1}
]

Rule advantages include unified visual and engine rules, flexible expression of complex JSON differences, and easy parsing.

Pre‑/Post‑Processing

Two categories of processing rules are provided:

Input‑affecting rules: Decide whether to use processed data when updating expected results.

Result‑only rules: Filter or sort comparison results without altering original data.

Core Algorithm

The comparison core handles object and array diffing.

Object Comparison

Common keys are recursively compared.

Keys present only in the expected object are marked as remove.

Keys present only in the actual object are marked as add.

Array Comparison

Two strategies are available:

Index‑based comparison: Fast but sensitive to element order.

Longest common subsequence (LCS) comparison: Provides clearer diff at higher computational cost.

Performance tests show index‑based comparison is faster for large arrays, while LCS yields more intuitive visual results when order varies.

Implementation uses dynamic programming to compute the LCS and backtrack the diff:

// State transition
def dp(i, j) {
    if (a[i-1] == b[j-1]) {
        return dp(i-1, j-1) + 1
    } else {
        return Math.max(dp(i, j-1), dp(i-1, j))
    }
}

// Backtrack diffs
def backtrackDiffs(a, b, dp) {
    def arrayChanges = []
    def aSize = a.size(), bSize = b.size()
    def aChangeCnt = 0, bChangeCnt = 0

    while (aSize > 0 || bSize > 0) {
        if (aSize > 0 && bSize > 0) {
            if (a[aSize-1] == b[bSize-1]) {
                recordChanges(arrayChanges, aChangeCnt, bChangeCnt, aSize-1, bSize-1)
                aChangeCnt = 0
                bChangeCnt = 0
                aSize--
                bSize--
            } else if (dp[aSize-1][bSize] > dp[aSize][bSize-1]) {
                aChangeCnt++
                aSize--
            } else {
                bChangeCnt++
                bSize--
            }
        } else if (aSize > 0) {
            aChangeCnt++
            aSize--
        } else {
            bChangeCnt++
            bSize--
        }

        if (aSize == 0 && bSize == 0) {
            recordChanges(arrayChanges, aChangeCnt, bChangeCnt, -1, -1)
            aChangeCnt = 0
            bChangeCnt = 0
        }
    }
    return arrayChanges.reverse()
}

Page Visualization

The UI builds on the svelte‑jsoneditor component, adding node highlighting and indexing based on diff rules.

Node highlighting: New nodes – green background; removed nodes – red; modified nodes – yellow.

Node indexing: scrollTo jumps to a specific JSON path; indexing can be filtered by diff type.

Conclusion and Outlook

The JSON assertion engine addresses efficiency, maintainability, and readability challenges in automated testing. Future work includes intelligent rule generation with machine learning, performance scaling via parallelism, broader format support (XML, YAML), and tighter integration with reporting tools.