Designing a Scalable Instagram Backend: Architecture, Storage, and Timeline Strategies

Introduction

Instagram is a social application that allows users to share photos and videos with optional captions, tags, and location metadata. Posts can be public or limited to followers, and users may set their profiles to private.

Requirements

Functional

Publish photos and videos

Follow and unfollow other users

Like or dislike posts

Search photos and videos by caption or location

Generate a chronological news feed that includes posts from followed users, as well as suggested and promoted content

Non‑functional

Scalability: support millions of users with sufficient compute and storage

Low latency for feed generation

High availability

Persistence: uploaded media must not be lost

Eventual consistency is acceptable for delayed propagation of posts

Reliability: tolerate hardware and software failures

Storage Model

Entities

User : id, name, email, bio, location, creation date, last login, etc.

Follower : one‑way follow relationship (A follows B)

Photo : id, location, caption, creation time, foreign key to owning user

Video : id, location, caption, creation time, foreign key to owning user

Data Model Diagram

SQL vs NoSQL

The data is inherently relational and requires ordered retrieval (chronological feed) and strong durability. Queries such as fetching followers or media by user ID are essential, so a relational database is chosen.

Top‑Level Design

Load Balancer : distributes incoming client requests

Application Servers : host business logic

Relational Database : stores structured user, media, and relationship data

Blob Storage : stores raw photo and video files

Detailed Design

Photo/Video Upload, View, and Search

Clients send an upload request to the load balancer, which forwards it to an application server. The server creates a metadata record in the relational database and stores the binary media in blob storage. On success the client receives a confirmation; on error an appropriate response is returned.

Viewing follows a similar path: the client requests a media item, the server retrieves metadata from the database and the binary data from blob storage, then streams it back. Search requests include keyword filters that are applied to the caption or location fields in the database.

Read operations dominate writes. To improve performance the system separates read‑only services from write services, employs caching for frequently accessed metadata, and uses lazy loading on the client side to fetch only media currently in view, reducing bandwidth and latency.

Timeline Generation

Pull Model : When a user opens the app, the service fetches the list of accounts the user follows, retrieves recent media from each followed account, merges the results, and returns the ordered list. This on‑demand approach can incur high latency for each request.

Push Model : Each user pushes newly created posts to the timelines of their followers at write time, eliminating the need for a full pull at read time.

Hybrid Approach : Users are classified based on follower count.

Push‑based users : hundreds to thousands of followers – their posts are pushed to follower timelines.

Pull‑based users : tens of thousands to millions of followers – timelines are generated on demand (pull).

The timeline service pulls data for pull‑based users and receives pushes for push‑based users.

Timeline Storage

Timeline entries are stored in a key‑value store using userID as the key and a list of post references (photo/video URLs) as the value. When a list approaches the size limit of the key‑value store, the overflow is moved to blob storage and the key points to the blob.

Stories Feature

A “Stories” column can be added to the media table to store an expiration timestamp (24 hours). A scheduled background job periodically deletes entries older than 24 hours, ensuring stories disappear after the intended period.

Evaluation

Scalability : add more application servers and database replicas to handle increased traffic and data volume.

Latency : cache hot metadata and use a CDN for media delivery to reduce response times.

Availability : replicate databases and blob storage across geographic regions; load balancer routes around failed nodes.

Persistence : durable storage with regular backups guarantees that uploaded photos and videos are never lost.

Consistency : relational database provides strong consistency for metadata; eventual consistency is acceptable for timeline propagation.

Reliability : database replication, redundant load‑balancing, and fail‑over mechanisms prevent single points of failure.