Day56 Review: Mastering Software Fault Tolerance and Continuous Learning

Overview

Date: 2026‑08‑12 Stage: First‑pass identification and knowledge‑shelf consolidation Study time: 45 minutes (insufficient)

Goal: Distinguish fault tolerance, error detection, and complexity reduction; explain recovery blocks, N‑version programming, and redundancy designs; and complete the Day01‑Day56 first‑pass knowledge‑shelf verification.

1. No New Vocabulary Today

Day55 questions (numeric answers):

系统多久出一次故障？
故障后多久恢复？
可用性和可靠度是多少？
串联、并联怎样计算？

Day56 questions to answer:

怎样尽量避免软件故障？
怎样尽快发现软件故障？
已经发生故障，怎样继续提供正确服务？
第一遍学过的知识，能否在题干中找到对应货架？

2. New Reliability‑Design Shelf

软件可靠性设计
├─ 容错设计：发生故障后，仍设法继续提供正确服务
│   ├─ 恢复块
│   ├─ N版本程序
│   └─ 冗余设计
├─ 检错设计：尽快发现并报告故障
└─ 降低复杂度设计：减少缺陷产生的机会

Strict error‑checking:

Calling detection “automatic recovery” is wrong.

Confusing recovery blocks with N‑version programs is wrong.

Assuming two identical software copies eliminate design defects is wrong.

During review, only memorising Day numbers without linking to problem statements is wrong.

3. Reference Materials

Official tutorial 9.4 – Software reliability design goals and principles.

9.4.1 – Fault‑tolerance techniques: recovery blocks, N‑version programs, redundancy.

9.4.2 – Error‑detection techniques: detection objects, latency, implementation, handling.

9.4.3‑9.4.4 – Complexity reduction, dual‑machine hot‑standby, server clusters.

“Red Book” 4.1 – Summary of fault‑tolerance, detection, complexity reduction, and system‑configuration techniques.

4. Reliability Metrics vs. Reliability Design

Day55 covered measurement:

MTTF, MTTR, MTBF

Availability

Reliability

Series‑parallel system calculations

Day56 focuses on actions:

Fault tolerance

Error detection

Complexity reduction

Dual‑machine hot‑standby

Server clusters

Key distinction:

Reliability metrics describe the system’s current performance (the “result”).

Reliability design describes how to improve the system (the “method”).

Example – a registration platform:

Average failure once every 1 000 hours.

Mean recovery time 10 minutes.

Annual availability 99.98 %.

These numbers are metrics. To improve reliability, one can:

Increase fault detection and alerting.

Prepare standby servers or algorithms.

Automate fail‑over.

Use recovery blocks or N‑version programs.

Reduce software complexity.

5. Three Types of Reliability Design

1. Fault‑tolerance design – When a failure occurs, the system strives to complete the task.

A software module that fails can switch to a backup implementation, an alternative path, or a majority vote to continue providing correct service.

Example – a registration rule calculation returns an exception:

主算法失败
   ↓
改用备用算法重新计算
   ↓
仍返回正确挂号结果

Key points to look for in exam questions:

Continue running after failure.

Backup implementation.

Replace faulty module.

Majority voting.

Dynamic redundancy.

2. Error‑detection design – Detect and report faults as early as possible.

Common techniques:

Check return values for out‑of‑range results.

Detect timeouts.

Validate status flags.

Verify data format, checksum, or business constraints.

On detection, stop, partially stop, or raise an alarm.

Four elements required by the textbook:

Detection object – where and what to detect.

Detection latency – how quickly after a fault it is detected.

Implementation method – how the check is performed.

Handling method – stop, partial stop, alarm, or hand‑off.

Detection only guarantees “knowing there is a problem”; it does not guarantee the problem is automatically solved.

Example – monitoring discovers a timeout on the registration interface, raises an alarm, but automatic fail‑over is needed to achieve fault tolerance.

3. Complexity‑reduction design – Prevent defects from being introduced.

High complexity raises the difficulty of understanding, modifying, testing, and maintaining code, increasing defect likelihood.

Ways to reduce complexity:

Simplify internal module logic.

Shorten long code and call chains.

Optimize data flow.

Remove unnecessary inter‑module dependencies.

Clarify interfaces and responsibility boundaries.

This is a pre‑fault preventive approach. The official tutorial calls it “complexity‑reduction design”.

4. Comparison Table (converted to list)

Lower‑complexity – before a fault occurs – aims to reduce defect creation – does not provide automatic continuation.

Detection – after a fault occurs – aims to know where the anomaly is – usually cannot alone keep the service running.

Fault‑tolerance – after a fault occurs – masks or replaces the fault – goal is to continue service.

5. Recovery Block

Basic structure – a set of operations treated as a fault‑tolerance unit, with multiple functionally equivalent but differently implemented program blocks.

保存恢复点
   ↓
执行主程序块
   ↓
验收测试通过？
   ├─ 是：接受结果并结束
   └─ 否：恢复到原状态
        ↓
        执行备用程序块1
        ↓
        验收测试通过？
           ├─ 是：接受结果
           └─ 否：再恢复并尝试下一个备用块

Key components:

Main program block.

One or more backup program blocks.

Acceptance test.

Recovery point and state rollback.

Medical example – drug‑dosage calculation :

主算法：按标准公式计算剂量；
验收测试：剂量必须在患者体重和年龄对应的安全范围；
若主算法结果未通过：恢复输入状态，执行备用算法；
备用算法通过验收测试后：返回结果。

Advantages:

Runs only one version at a time, saving resources.

If the primary version fails, backup versions are tried sequentially.

Suitable when a clear acceptance condition can be defined.

Risks:

Acceptance test itself may be buggy.

All program blocks might share the same requirement error.

Incomplete state rollback can corrupt backup execution.

Sequential retries increase response time.

6. N‑Version Programming

Basic structure – multiple independent versions receive the same input and run in parallel.

┌→ 版本1 ┐
相同输入───┼→ 版本2 ──→ 多数表决 → 输出结果
          └→ 版本3 ┘

Example – three versions produce doses 20 mg, 20 mg, and 200 mg; majority voting selects 20 mg, masking the erroneous third version.

Why independence matters:

If all versions are written by the same team, use the same flawed algorithm, or copy the same code, they may fail together, making majority voting ineffective.

The textbook recommends different designers, algorithms, programming languages, and testing methods.

Even with independent implementations, a common‑requirement error can cause a “common‑cause failure”.

Advantages:

Parallel execution yields fast decision.

A single faulty version can be outvoted.

Suited for high‑safety, high‑reliability scenarios.

Risks:

High development and verification cost.

Increased resource consumption.

The voting component itself can become a single point of failure.

Cannot automatically overcome common‑requirement errors.

Difficult to compare outputs when they are not directly comparable.

7. Recovery Block vs. N‑Version – Quick Identification

Seeing “acceptance, rollback, sequential backup execution” → Recovery block.

Seeing “multiple versions run in parallel, majority voting” → N‑version programming.

8. Redundancy Design

Hardware redundancy (easy to understand) :

主电源故障 → 备用电源接管；
主服务器故障 → 备用服务器接管；
网络链路故障 → 另一条链路继续传输。

Software redundancy must avoid merely copying the same code.

Two servers running identical software protect against hardware failure, but if the code contains the same defect, both crash on the same input:

实例A 处理异常输入时崩溃；
实例B 收到相同输入后也崩溃。

Therefore, software redundancy should provide:

Different execution paths.

Different algorithms.

Different implementation methods.

Modules that can replace the primary one.

Redundancy only yields theoretical gains when failures are sufficiently independent and the switch‑over mechanism works correctly.

Cost of redundancy includes development, testing, maintenance, extra storage/memory, synchronization, fault‑detection and switch‑over logic, standby capacity, and increased architectural complexity.

9. System‑Configuration Fault Tolerance

Active/Standby (dual‑machine hot‑standby) :

主机 Active：正常处理业务
备机 Standby：监控并准备接管
心跳检测：判断主机是否存活
主机故障：备机被激活并接管资源

Characteristics: simple switch‑over logic; standby may waste resources; shared storage, heartbeat link, and switch‑over software can become risk points.

Dual‑machine mutual‑backup – each machine runs a different application and serves as the other's backup.

服务器A：运行应用甲，备用应用乙
服务器B：运行应用乙，备用应用甲

When one fails, the other takes over, achieving higher resource utilization than pure hot‑standby, but the surviving node bears increased load.

Dual‑machine dual‑active (dual‑active) – both servers are active, handling the same application and sharing load.

请求 → 负载均衡 → 服务器A
                → 服务器B

Provides load sharing and mutual backup; related to Day11’s cluster and load‑balancing concepts.

Server cluster – multiple independent servers present a unified service endpoint.

客户端只看到一个服务入口；
多个节点共同提供服务；
某节点故障后，服务可在其他节点重启或被接管；
可以继续添加节点扩展容量。

Hot‑standby, clustering, and fault tolerance are not synonyms; they can be combined but must specify service entry, state preservation, fault detection, and takeover process.

10. Comprehensive Case – Regional Medical Prescription Review Platform

Requirements:

If a calculation module errors, it must not return a dangerous dose.

Results must be automatically judged safe.

Core service node must recover within 30 seconds after failure.

Faults must trigger timely alerts and retain audit logs.

Three independent versions must not all share the same erroneous algorithm.

Solution sketch:

High‑risk, easily comparable dosage calculation – use N‑version programming with independent teams, algorithms, and languages.

Rule‑based calculations with clear safety ranges – use recovery blocks (save recovery point, run primary, run acceptance test, rollback, try backups).

Run‑time and timeout anomalies – apply error‑detection design (return‑value checks, timeout detection, status flags, data‑format validation, alerting).

Node failure – deploy multiple instances, health checks, automatic instance removal, and traffic redirection.

Common dependencies – apply redundancy and isolate fault domains.

Reduce software complexity – simplify module logic, clarify interfaces, and eliminate unnecessary dependencies.

Key actions:

Parallel N‑version execution with majority voting for dosage calculation.

Recovery block with acceptance test for rule‑based safety checks.

Monitoring and alerting for detection, coupled with automatic fail‑over to maintain service.

Clustered deployment with heartbeat‑driven fail‑over to meet the 30‑second RTO.

Fault‑injection testing to verify detection, isolation, switch‑over, and continued correct service.

11. First‑Pass Knowledge Shelf Overview

The first pass does not require memorising all 56 days; it requires mapping a problem to the correct knowledge shelf.

Business & Software Engineering – Days 15‑21, 46‑48.

Architecture Core – Days 22‑28.

Modern & Specialized Architecture – Days 29‑42.

Distributed & High‑Concurrency – Days 01‑14, 31‑33.

Computer Systems Foundations – Days 43‑49.

Data Architecture – Days 08‑10, 39‑40, 50‑52.

Security – Days 38, 53‑54.

Reliability & Fault Tolerance – Days 11, 55‑56.

12. Quick Classification of Eight Question Types

Q1 – Performance & high concurrency (caching, async, DB tuning, scaling, rate‑limiting).

Q2 – Modifiability (adapter pattern, low coupling, architectural decision cost).

Q3 – Transaction consistency (local/ distributed transactions, 2PC, TCC, Saga, compensation).

Q4 – Idempotency (unique business IDs, constraints, state machines, dedup tables).

Q5 – Logging & alerting (batch vs. stream processing, data lake, data warehouse, Lambda/Kappa).

Q6 – RBAC/TBAC (role‑based, task‑based dynamic authorization, least privilege, audit).

Q7 – High MTTR (monitoring, auto‑switch, recovery drills, MTTR reduction).

Q8 – N‑version programming (independent versions, majority voting, common‑cause risk, voting component risk).

13. First‑Pass Gap‑Checking Method

Read the question stem, decide the shelf.

Without notes, state definition, applicable scenario, and one cost.

If stuck, open the corresponding Day’s small section.

Close notes again and explain the concept with your own example.

Knowledge‑point status tags:

A – can identify, explain, and give examples.

B – can identify but explanation incomplete.

C – only recognize after seeing answer.

D – completely unknown.

Handling:

A – no need to review now.

B – reinforce in the second‑pass case study.

C – schedule a 10‑minute review this week.

D – immediately revisit the Day and fill the shelf.

14. Doodle Recall (core diagram)

软件可靠性设计
               /     |     \
          少犯错   早发现   能继续
             |       |       |
        降低复杂度 检错   容错
                         / | \
                恢复块 N版本 冗余
                 |      |      |
            验收回滚 多数表决 备用替换

Note: identical defects cause simultaneous failure, so redundancy must also guard against common‑cause failures.

15. Three‑Minute Teaching Outline

Explain what problems fault tolerance, detection, and complexity reduction each solve.

List components of a recovery block.

Why N‑version programs require independent designs.

How to quickly distinguish recovery block vs. N‑version in exam questions.

Why two identical software copies cannot reliably mask the same defect.

Differences among hot‑standby, mutual‑backup, and dual‑active configurations.

What the eight first‑pass knowledge shelves contain.

How learning goals shift after Day 57 (from recognition to solution synthesis).

16. Self‑Test Questions (selected)

What are the three main lines of software reliability design? – Fault‑tolerance, error‑detection, complexity‑reduction.

What four elements does error‑detection design consider? – Detection object, detection latency, implementation method, handling method.

Why does detecting a timeout and raising an alarm not equal completing fault tolerance? – Detection only knows the fault; fault tolerance must automatically continue service via retry, fail‑over, or substitution.

What key parts compose a recovery block? – Main block, one or more backup blocks, acceptance test, recovery point/rollback.

How does N‑version programming produce a final result? – Parallel independent versions run, then a majority vote selects the output.

Difference in execution and result judgment between recovery block and N‑version? – Recovery block runs primary then sequential backups with acceptance testing; N‑version runs all versions concurrently and decides by majority voting.

Why can’t identical software copies solve the same defect? – They share the same code and design flaw, so they fail together; redundancy must use different paths/algorithms.

How do hot‑standby, mutual‑backup, and dual‑active work? – Hot‑standby: one active, one standby; Mutual‑backup: each runs a different app and backs up the other; Dual‑active: both run the same app, share load, and back each other up.

Which technique uses rollback and a backup algorithm after a safety‑check failure? – Recovery block.

Which technique runs three independent versions and selects a result agreed by at least two? – N‑version programming; main risk is lack of independence leading to common‑cause failure and voting component becoming a single point of failure.