How Distributed Indexing Improves Backup Performance and Scalability

In backup software, data indexing is essential for managing, restoring, and retrieving backed‑up information. Traditional centralized indexing stores the index database on the backup management server, forcing every read and write through this single point and creating a severe performance and scalability bottleneck as the volume of backed‑up files increases.

Simpana addresses these issues with a distributed two‑level index design. The first‑level (primary) index records high‑level metadata about backup objects—files, emails, VMs, databases—and includes data object, content, and classification indexes. The second‑level (secondary) index stores detailed per‑file entries, enabling content search and regulatory compliance.

The primary index is managed by the Comm Server, saved in an MS SQL database, and periodically backed up to backup media. It tracks each backup task with fields such as time, computer name, task type, and tape ID. Because it aggregates only one record per task, its size remains modest.

The secondary index resides on each Media Agent in a relational database, cached on the agent’s disk to improve performance. It contains thousands of rows per backup task—for example, backing up a Windows system drive can generate over 40,000 entries. These detailed records are backed up together with the actual data to tape, ensuring consistency.

This distributed (secondary) index architecture also suits multi‑branch environments. Each branch site maintains its own local secondary index, while the central management server only handles the primary index and overall task coordination, dramatically reducing WAN traffic and bandwidth consumption.

Backup and restore workflow : After a user creates a backup job and policy, the Comm Server initiates the task, creates a primary index record, and commands the iDA to scan the source server, generating a file list and primary secondary‑index fields. The Media Server then writes the secondary index to the backup medium and updates the primary index accordingly. During restore, the Comm Server uses the primary index to locate the secondary index on tape, restores it to the Media Agent’s cache, and presents the file list to the user for selection.

Index maintenance : The primary index is backed up daily by a default scheduled task that copies metadata (including the primary index and deduplication database) to disk and tape. If the metadata or primary index becomes corrupted, Simpana provides GUI tools to restore it first from near‑line disk backups, and if necessary from offline tape copies. The secondary index is protected because it is written alongside the backup data; if the secondary index on disk is damaged, the system can recover it using the primary index’s location information.

This distributed indexing approach eliminates the central server bottleneck, improves scalability for large data sets, reduces network load for branch sites, and ensures reliable backup and fast recovery of both metadata and actual data.