Designing a Robust Web Crawler Architecture: Insights from Three Iterations

Recent work on web crawler projects revealed a need for a solid architecture despite a small team and ad‑hoc initial designs.

Key characteristics of a good architecture

1. Completeness

The first three versions of the crawler architecture were created with limited preparation and insufficient understanding of existing systems, resulting in an inaccurate, "blind men touching an elephant" design.

2. Standardized, understandable, simple, and barrier‑free

During a weekly meeting, the initial sketch was confusing, prompting the realization that proper UML diagrams (sequence and deployment) are essential for clear communication between product and engineering teams. Standardized UML diagrams made the architecture easy to grasp for all stakeholders. Second‑version sequence diagram Second‑version deployment diagram

3. Clear goals, accuracy, maintainability, and extensibility

The third version refined the architecture to a three‑layer model: service orchestration, asynchronous crawling tasks, and a data service model, clearly separating synchronous and asynchronous phases. Clear goals: The abstract three‑layer design directly addresses the core problems. Accuracy: Feedback from QA confirmed that the diagram accurately reflects the business workflow. Maintainability & extensibility: Emphasizing low‑cost, scalable design, the architecture can evolve step‑by‑step as infrastructure (e.g., Mesos, Marathon, configuration platform) matures. Third‑version sequence diagram Third‑version deployment diagram

Source: https://www.jianshu.com/p/11e8eda8b5a1