Designing a High‑Performance, Scalable Province‑Wide Toll Collection Platform

Business Background

A province’s highway network consists of nearly one hundred segments, each operated by different companies, requiring vehicles to stop at toll stations and change cards when crossing segments, which reduces traffic efficiency. With the maturity of information technology, a unified networked toll system allows drivers to obtain a card at entry and pay only once at exit, eventually enabling nationwide single‑payment tolling.

The new system must centralize transaction logs, high‑resolution checkpoint logs, and images, support transaction splitting, provide video monitoring, and prevent fare evasion. Data that was previously scattered across segment systems is now aggregated at a provincial center for shared queries and statistical reporting.

Key Requirements

The data flow between system layers is illustrated in the diagram below.

There are approximately 100 segment centers, each with several toll stations, totaling about 2,000 entry lanes and 1,000 exit lanes. Vehicles generate entry transaction logs and high‑resolution checkpoint data when entering a toll station, and exit transaction logs when paying at the exit. Equipment status information is periodically sent to the segment center.

Segment data must be promptly aggregated to the provincial platform, which queries entry logs and checkpoint images for verification. The provincial settlement center splits fees based on segment logs and distributes results back to the platform and segment centers. The platform also distributes base rates, device control commands, and operational parameters downstream.

Further analysis revealed the system characteristics:

Most scenarios have low real‑time requirements; only entry‑log queries need sub‑second response.

Concurrent access is modest (under 2,000 users, <500 query concurrency).

Business logic is simple except for complex fee‑splitting rules.

Entry‑log and checkpoint queries require 24/7 high availability to avoid traffic congestion.

Daily data volume exceeds 30 million transaction logs and 20 million checkpoint logs; images (≈200 KB each) impose huge storage demands.

Major Decisions

Design decisions balanced functional and quality requirements with budget, team expertise, network environment, management constraints, and schedule. The tight two‑month schedule and multiple contractors forced careful trade‑offs.

Performance

Database Partitioning by Business Domain

Over fifteen subsystems were defined, each with its own database schema. Shared data (users, organizations, roads, toll stations, dictionaries) reside in a common schema, while specialized business data have dedicated databases. Oracle partitioned tables by date (e.g., 3‑day or 10‑day partitions) keep each partition under 100 million rows, enabling sub‑second queries with proper indexing and fast partition‑drop cleanup.

In‑Memory Database for Fast Queries

To guarantee sub‑second responses for entry‑log queries under higher concurrency, a NoSQL in‑memory store was considered for the most recent three days of data. After evaluating options, MongoDB was preferred over Memcached (no query capability) and Redis (less feature‑rich). Commercial products like GemFire were also tested, offering in‑memory computation and sub‑100 ms query latency at the cost of higher memory and CPU usage.

High‑Configuration Hardware

The project employed IBM servers, EMC storage, Cisco switches, and F5 load balancers. Dedicated storage with RAID10/5 provided redundancy, simplifying disaster recovery.

Reliability and High Availability

Commercial Middleware

Oracle RAC clusters ensured database‑level HA, while IBM WebSphere provided application‑level clustering and soft load balancing for stateless query services.

Message Middleware

RabbitMQ was selected for its concurrency performance and HA features, enabling decoupled communication between the provincial platform and segment centers for control commands and data streaming.

Scalability

The system was divided into multiple loosely coupled subsystems, each potentially deployed in clusters. Inter‑subsystem messaging isolates failures, and a portal with single sign‑on integrates the web components.

Operations

Virtualization

Leveraging VMware, three high‑end servers were virtualized into nearly a hundred VMs, reserving two powerful nodes for Oracle clustering. Virtualization facilitated dynamic resource allocation and simplified management, embodying cloud‑computing principles.

Overall Architecture

Logical Architecture

The stack consists of:

Infrastructure layer: servers, networking, firewalls, backup, virtualization software, OS.

Middleware layer: commercial databases, application servers, in‑memory databases, open‑source message middleware, data extraction tools.

Service component layer: reusable business components and development frameworks.

Business application layer: individual functional subsystems.

Component Dependencies

Kettle extracts logs into the database and publishes to RabbitMQ.

GemFire loads recent logs into memory.

Query services call GemFire client APIs for entry logs.

Fee‑splitting service consumes messages, processes them, and writes results to the business database.

Web subsystems use CAS for single sign‑on and a unified user management system.

Physical Deployment

Conclusion

Just as real‑time strategy games require careful unit composition and tactics, designing system architecture involves strategic decisions about technologies, patterns, and trade‑offs. The chosen combination of partitioned Oracle tables, an in‑memory database, commercial middleware, and virtualization delivered a high‑performance, reliable toll collection platform within the project’s constraints.