Skip to content
Designing a Collaborative Document Editor: WebSocket, OT/CRDT & Snapshot Replay Deep Dive

Designing a Collaborative Document Editor: WebSocket, OT/CRDT & Snapshot Replay Deep Dive

This comprehensive guide covers designing a real-time collaborative document editing system, detailing WebSocket communication, OT/CRDT conflict resolution algorithms, server-side sequencing, snapshot-based storage, offline editing, presence, and performance optimization with runnable Java code examples.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

1. Clarify Requirements Before Drawing Architecture

Experienced engineers align on boundaries first. The article defines a requirements table:

Core Function : Multiple users simultaneously edit same document, see others' content and cursors in real time → Long connections + incremental operations

Document Type : Rich text (paragraphs/headings/lists/tables) → Block/tree data model, not plain string

Collaboration Scale : 50-100 editors + thousands of read-only viewers per doc → Read/write separation, broadcast fan-out throttling

Total Scale : 10M DAU, peak 300K long connections → Stateless gateway, horizontal scaling

Edit Frequency : ~2 ops/sec per user (typing + cursor) → Operation batching required

Consistency : Eventual consistency acceptable, ms-level convergence → Can drop distributed locks and strong consistency

Network : Mobile weak network, disconnect/reconnect, offline editing 5 min → Local queue + incremental catch-up

History : View, rollback, retain 30 days → Snapshot + OpLog, compliance

One-sentence characterization: This is a high-frequency small writes + strong real-time broadcast + strong convergence consistency system. The hard part isn't CRUD or storage, but how to let 10 people type simultaneously without overwriting each other .

2. Overall Architecture: Layered Decoupling

Core idea: client optimistic updates + server centralized sequencing .

Client : Local optimistic rendering, operation collection, cursor sync, offline cache (IndexedDB) — Input must never block

Gateway : Long-connection management, auth, heartbeat, traffic shaping, route by docId — Stateless , scale freely

Collaboration Engine : Conflict merge, global sequencing, operation broadcast, in-memory document state — Stateful , single-doc serial

Message Layer : Cross-node broadcast + async OpLog persistence — Buffer spikes, slow DB writes don't block editing

Storage : Hot data in Redis, OpLog + incremental snapshots on disk — Append-only, sequential I/O

Three design trade-offs interviewers love: Gateway stateless, engine stateful — all ops for same docId routed via consistent hash to same thread on same node , serializing processing. This downgrades distributed consistency to a single-machine in-memory problem . Real-time path separated from persistence path — broadcast via WebSocket immediately, persistence via MQ asynchronously. DB jitter doesn't affect editing experience. Stateful down, stateless up — scalability via stateless gateway, consistency via stateful collaboration service.

3. Complete Edit Flow: Three Critical Sequence Details

Client optimistic apply : render locally first, wait for ACK later, otherwise perceived latency explodes.

Server authoritative sequencing : version numbers issued by server only; clients must not auto-increment — this is the convergence prerequisite.

Catch-up via active pull, not passive wait : client detects version lag, calls pullOpLog(fromVersion) for incremental ops; if too far behind (e.g., >500 ops), fetch full snapshot and reset.

Message Protocol

/** WebSocket Message Protocol */
@Data @Builder
public class WebSocketMessage {
    private MessageType type; // JOIN/LEAVE/OPERATION/BROADCAST/SYNC/PING/PONG/ERROR
    private String documentId;
    private String clientId;
    private CRDTOperation operation;
    private int clientVersion;
    private int serverVersion;
    private long timestamp;
    private boolean compressed;
    private byte[] payload;
}
public enum MessageType { JOIN, LEAVE, OPERATION, BROADCAST, SYNC_REQUEST, SYNC_RESPONSE, PING, PONG, ERROR }

Real edit operation example:

{
  "type": "insert",
  "opId": "u789-1712345678901-3",
  "docId": "doc-abc123",
  "clientId": "u789",
  "baseVersion": 42,
  "serverSeq": 12345,
  "position": 12,
  "content": "x",
  "timestamp": 1712345678901
}

Five Hard Connection-Layer Designs

Connection Identity : Bind connectionId + clientId + docId, no IP-based routing

Heartbeat : Ping every 30s, disconnect if no pong for 60-90s, cleanup resources

Message Sequence : Server generates global monotonic serverSeq; client detects gaps and requests retransmission

Reconnection : Reconnect with lastServerSeq, server pushes only incremental ops after that

Idempotency : Each op carries globally unique opId = clientId + timestamp + seq; duplicates discarded

Heartbeat implementation (Spring WebSocket):

@Component
public class HeartbeatHandler extends TextWebSocketHandler {
    private static final int HEARTBEAT_INTERVAL = 30000;
    private static final int TIMEOUT = 60000;
    @Override
    public void afterConnectionEstablished(WebSocketSession session) {
        ScheduledFuture<?> future = scheduler.scheduleAtFixedRate(() -> {
            if (session.isOpen()) {
                try { session.sendMessage(new TextMessage("{\"type\":\"ping\"}")); }
                catch (IOException e) { session.close(CloseStatus.SERVER_ERROR); }
            }
        }, HEARTBEAT_INTERVAL, HEARTBEAT_INTERVAL, TimeUnit.MILLISECONDS);
        session.getAttributes().put("heartbeat", future);
    }
    @Override
    protected void handleTextMessage(WebSocketSession session, TextMessage message) {
        JsonObject json = JsonParser.parseString(message.getPayload()).getAsJsonObject();
        if ("pong".equals(json.get("type").getAsString())) {
            session.getAttributes().put("lastPong", System.currentTimeMillis());
        }
    }
}

Bandwidth saver: operation merging. User types "hello" → not 5 ops, but locally batch 50ms into one INSERT(pos=10, text="hello"). Cuts typing QPS from 10 to 2, bandwidth down 60%+. Server also transforms + broadcasts batch ops once.

4. Core Conflict Resolution: OT vs CRDT Deep Dive

4.1 What Conflicts Look Like

Initial doc "abc", A and B edit concurrently on same version:

Initial: "abc" v=0
A: Insert(1,"X") → expect "aXbc" ┐ concurrent submit
B: Delete(2,1)   → expect "ac"   ┘
❌ Without transform, naive execution diverges.
✅ OT transform:
  transform(B, A): B.pos=2 > A.pos=1 → pos+1 → Delete(3,1)
  transform(A, B): A.pos=1 < B.delete range → pos unchanged
  Both ends converge to "aXb".

Core contradiction: two ops based on same old version, direct execution by arrival order shifts position coordinate system, causing divergence. OT and CRDT are two schools solving this.

4.2 OT vs CRDT Comparison

Core Idea : OT — Central server transforms concurrent ops, adjusts offsets. CRDT — Data structure has commutativity, associativity, idempotence; merge is naturally conflict-free .

Consistency Guarantee : OT — Relies on server total order (Single Source of Truth) + CP1/CP2 math properties. CRDT — Lamport timestamps + ID partial order auto-sort, mathematically proven convergence.

Central Server : OT — Strong dependency. CRDT — Can be decentralized, supports P2P.

Offline Editing : OT — Weak, needs replay/compensation of many historical ops. CRDT — Naturally friendly: local execute first, auto-merge on reconnect.

Implementation Complexity : OT — 🔴 Extreme, transform function combinatorial explosion, CP2 verification very hard. CRDT — 🟡 Medium, fixed rules but tombstone GC complexity.

Metadata Size : OT — 🟢 Small, only ops stored. CRDT — 🔴 Large, each char carries unique ID, memory bloat 3-5x.

Undo : OT — Needs inverse op + replay. CRDT — Relatively natural.

Representative Products : OT — Google Docs, Tencent Docs, Feishu. CRDT — Figma, Notion (partial), Yjs, Automerge.

Best Fit : OT — Linear text, online-first, central server. CRDT — Strong offline, desktop, decentralized, structured data.

4.3 Which to Choose? Two Talking Points

Talking Point 1 (Centralized Product): "I choose OT + server-centric sequencing . Our scenario has stable network, linear text; OT ops are small, low storage cost, history replay for audit, permissions and persistence easier. CRDT's offline advantage covered by 'local queue + reconnect replay'. Google Wave hit many OT pits, but Google Docs and Tencent Docs prove this path is solid for online docs."
Talking Point 2 (Offline-first / Small Team): "I prefer CRDT . OT transform functions are too hard to get right, CP2 verification drives you crazy. CRDT has mature open-source like Yjs, Automerge; offline native, faster to ship. Java can bind y-crdt or implement lightweight RGA. Trade-off: metadata bloat and tombstone GC complexity."
Real bonus points: articulate migration triggers: "Main online path uses OT central sequencing; if later we need fully offline desktop, whiteboard/graphics structured data, or decentralized P2P, switch to Yjs-style CRDT — the pivot is whether client-side full autonomy is required ."

4.4 OT Transform Function: Interview Must-Write Code

Real systems use [retain, insert, delete] triple sequences; here atomic Insert/Delete shows transform logic. Complete version handling devil details like insert-into-delete-range and overlapping deletes :

public sealed interface Op permits Insert, Delete { int pos(); }
public record Insert(int pos, String text, String siteId) implements Op {}
public record Delete(int pos, int len, String siteId) implements Op {}

public final class OT {
    /** Transform local to coordinate system after remote applied.
     * Goal: apply(apply(doc, remote), local') == apply(apply(doc, local), remote')
     * @param localIsLower tie-break for same-position insert (siteId lexicographic ensures all clients consistent) */
    public static Op transform(Op local, Op remote, boolean localIsLower) {
        if (remote instanceof Insert ri) {
            int rlen = ri.text().length();
            if (local instanceof Insert li) {
                // Same-position insert: smaller siteId goes left, guarantees convergence 🎯
                boolean shift = ri.pos() < li.pos() || (ri.pos() == li.pos() && localIsLower);
                return shift ? new Insert(li.pos() + rlen, li.text(), li.siteId()) : li;
            }
            Delete ld = (Delete) local;
            if (ri.pos() <= ld.pos()) {
                return new Delete(ld.pos() + rlen, ld.len(), ld.siteId()); // insert before delete → shift right
            }
            if (ri.pos() >= ld.pos() + ld.len()) {
                return ld; // insert after delete → unaffected
            }
            // ⚡ Insert falls inside delete range: one Delete must split into two
            return new SplitDelete(ld.pos(), ri.pos() - ld.pos(),
                    ld.len() - (ri.pos() - ld.pos()), ld.siteId());
        }
        Delete rd = (Delete) remote;
        int rEnd = rd.pos() + rd.len();
        if (local instanceof Insert li) {
            if (rEnd <= li.pos()) return new Insert(li.pos() - rd.len(), li.text(), li.siteId());
            if (rd.pos() >= li.pos()) return li;
            return new Insert(rd.pos(), li.text(), li.siteId()); // insert point deleted → attach to delete start
        }
        Delete ld = (Delete) local;
        int lEnd = ld.pos() + ld.len();
        if (rd.pos() >= lEnd) return ld; // completely right, unaffected
        if (rEnd <= ld.pos()) return new Delete(ld.pos() - rd.len(), ld.len(), ld.siteId()); // completely left, shift left
        // ⚡ Delete ranges overlap: subtract already-deleted portion; len exhausted → noop
        int overlap = Math.min(rEnd, lEnd) - Math.max(rd.pos(), ld.pos());
        return new Delete(Math.min(ld.pos(), rd.pos()), ld.len() - overlap, ld.siteId());
    }
}
Highlight explanations (must say in interview): Same-position insert tie-break is convergence linchpin. Two users type at same position simultaneously; must use globally consistent comparator ( siteId lexicographic) to decide order, otherwise client A gets XY , client B gets YX → immediate divergence. Two easiest-to-miss pits: ① Insert inside delete range → one Delete must split into two; ② Overlapping deletes → subtract, if length exhausted degrade to noop (two users delete same char). transform must satisfy TP1 property (transformed ops commute): T(T(e, a), b') = T(T(e, b), a') . Real engineering: one op transforms against a batch of concurrent ops sequentially — this cascade is rebase — Git users instantly understand.

How to verify OT correctness? Two hard methods:

Property-based fuzz testing : random op sequences + random concurrent interleavings, assert all clients converge, guard transform(a,b) ≡ transform(b,a) convergence.

TP2 transform property verification :

transform(transform(a,b), transform(c,b)) == transform(transform(a,c), transform(b,c))

.

Pass both, weird bugs unlikely. Production also handles: rich-text formatting, Emoji/Chinese/Unicode surrogate pairs ( position by code point/grapheme, not Java char ), cursor sync. Complexity control: only support atomic Insert/Delete, decompose complex ops into atomic sequences.

4.5 CRDT Three Core Pieces

If CRDT path, must explain: globally unique IDs, tombstone deletion, deterministic merge rules .

Piece 1: LSEQ Character Model

Each character carries globally unique ID and total-order position string:

public class CrdtCharacter {
    private final String id; // globally unique = siteId + sequence
    private final char value;
    private final String position; // total-order position string (LSEQ algorithm)
    private final boolean tombstone; // tombstone: delete marks, not physical delete, ensures merge consistency
    /** Core merge rule: sort by position lexicographically, tie-break by ID.
     * Any node execution yields identical result, naturally conflict-free. */
    public static int compare(CrdtCharacter a, CrdtCharacter b) {
        int posCompare = a.position.compareTo(b.position);
        return posCompare != 0 ? posCompare : a.id.compareTo(b.id);
    }
    /** Generate new unique position between two chars.
     * Production uses LSEQ/Logoot optimization to avoid unbounded position strings. */
    public static String generatePosition(String prev, String next, String siteId) {
        return prev + siteId.hashCode() % 10;
    }
}

Piece 2: RGA (Replicated Growable Array) Insert Logic

Positioning by neighbor IDs, not numeric offsets:

public class RGANode {
    private final String id; // globally unique: [lamportClock, clientId]
    private final String leftId; // left neighbor ID
    private final String rightId; // right neighbor ID
    private char value;
    private boolean isDeleted; // tombstone: mark only, never hard delete
    private long timestamp;
}

public class RGAArray {
    // Core: insert by neighbor ID, not numeric position
    public void insert(String leftId, String newId, char value, long timestamp, String clientId) {
        RGANode leftNode = findNode(leftId);
        RGANode rightNode = leftNode.getNext();
        // Key: concurrent insert at same position sorting rule
        while (rightNode != null && isRightNodeShouldComeFirst(rightNode, timestamp, clientId)) {
            leftNode = rightNode;
            rightNode = rightNode.getNext();
        }
        RGANode newNode = new RGANode(newId, leftId, rightNode != null ? rightNode.getId() : null, value);
        insertBetween(leftNode, newNode, rightNode);
    }
    /** Concurrent insert sorting rule:
     * 1. Compare timestamp, larger timestamp comes first
     * 2. Timestamp equal compare clientId, larger ID comes first
     * All nodes execute same rule → final order identical! */
    private boolean isRightNodeShouldComeFirst(RGANode rightNode, long newTs, String newClientId) {
        if (rightNode.getTimestamp() != newTs) return rightNode.getTimestamp() > newTs;
        return rightNode.getClientId().compareTo(newClientId) > 0;
    }
}

Piece 3: Document Engine Merge & GC

Vector clocks decide version priority, tombstone nodes periodically collected:

@Component
public class CRDTDocumentEngine {
    /** Apply op to document. Delete only soft-delete (tombstone), ensures merge consistency regardless of order. */
    public DocumentState applyOperation(DocumentState doc, CRDTOperation op) {
        List<CharacterNode> characters = doc.getCharacters();
        switch (op.getType()) {
            case INSERT:
                CharacterNode newNode = CharacterNode.builder()
                    .id(op.getId()) // op unique ID: "clientId-logicalClock"
                    .content(op.getContent())
                    .timestamp(op.getTimestamp())
                    .clientId(op.getClientId())
                    .build();
                int insertPos = findInsertPosition(characters, op.getPosition(), op.getId());
                characters.add(insertPos, newNode);
                break;
            case DELETE:
                CharacterNode target = findNodeById(characters, op.getContent());
                if (target != null) target.setDeleted(true); // soft delete, idempotent: repeat delete no side-effect
                break;
            case RETAIN: break;
        }
        gcDeletedNodes(characters); // periodic tombstone cleanup
        return DocumentState.builder()
            .characters(characters)
            .version(doc.getVersion() + 1)
            .lastOperationId(op.getId())
            .build();
    }
    /** CRDT merge function - key! Vector clock decides version priority,
     * merge result identical regardless of operation arrival order. */
    public DocumentState merge(DocumentState local, DocumentState remote) {
        if (isNewer(local.getVersionVector(), remote.getVersionVector())) {
            return mergeWithPriority(local, remote);
        }
        return mergeWithPriority(remote, local);
    }
    /** Vector clock compare: remote has newer component → "newer" */
    private boolean isNewer(Map<String, Long> localClock, Map<String, Long> remoteClock) {
        boolean hasNewer = false;
        for (Map.Entry<String, Long> entry : remoteClock.entrySet()) {
            Long localTime = localClock.getOrDefault(entry.getKey(), 0L);
            if (entry.getValue() > localTime) hasNewer = true;
            else if (entry.getValue() < localTime) return false; // local has newer component
        }
        return hasNewer;
    }
    /** Garbage collection: remove deleted nodes to free memory.
     * Production uses delayed deletion to avoid GC jitter. */
    private void gcDeletedNodes(List<CharacterNode> characters) {
        Iterator<CharacterNode> it = characters.iterator();
        while (it.hasNext()) {
            CharacterNode node = it.next();
            if (node.isDeleted() && System.currentTimeMillis() - node.getTimestamp() > GC_THRESHOLD) {
                it.remove();
            }
        }
    }
}
⚠️ CRDT costs must be clear: 100K-char document metadata may bloat to 5-10MB (each char carries siteId + clock ), so tombstone GC and string compression storage are mandatory, not optional.

5. Server Core Implementation: Sequencing, Transform, Broadcast, Persistence

5.1 Serial Sequencing: One "Single-Thread Track" Per Document

Real systems don't synchronized globally per request. Correct pattern: Document-level Actor — same doc ops go into single-threaded queue, different docs naturally parallel.

/** Document-level Actor: each docId bound to single-thread Executor
 * —— this is the whole secret of "serial sequencing" 🔑 */
public class DocumentActor {
    private final String docId;
    private final Executor eventLoop; // sharded by docId hash to single-thread executor
    private final OpRepository opRepository;
    private final Broadcaster broadcaster;
    private final Set<String> processedKeys = new HashSet<>();
    private long version;
    public CompletableFuture<Ack> submit(Op clientOp) {
        CompletableFuture<Ack> future = new CompletableFuture<>();
        eventLoop.execute(() -> {
            try { future.complete(commit(clientOp)); }
            catch (Exception e) { future.completeExceptionally(e); }
        });
        return future;
    }
    private Ack commit(Op clientOp) {
        String idempotentKey = clientOp.clientId() + ":" + clientOp.clientSeq();
        // 1. Idempotency: prevent duplicate apply from network retries
        if (processedKeys.contains(idempotentKey)) return new Ack(version, true);
        // 2. Client version cannot be "ahead"; ahead means missing server ops
        if (clientOp.baseVersion() > version) throw new IllegalStateException("missing server ops");
        // 3. Client based on old version: fetch server ops after baseVersion, transform sequentially
        //    THIS IS OT'S CORE STEP!
        Op transformedOp = clientOp;
        List<Op> conflictOps = opRepository.findAfter(docId, clientOp.baseVersion(), version);
        for (Op remoteOp : conflictOps) transformedOp = OtTransformer.transform(transformedOp, remoteOp);
        // 4. Noop op directly ACK (e.g., two users deleted same char)
        if (transformedOp.isNoop()) return new Ack(version, false);
        // 5. Generate new server version (server issues versions, clients must not auto-increment)
        version++;
        // 6. Persist first, then broadcast (WAL), ensures recoverability & audit
        opRepository.append(docId, version, transformedOp);
        processedKeys.add(idempotentKey);
        // 7. Broadcast to other clients
        broadcaster.publish(docId, new ServerOp(version, transformedOp));
        return new Ack(version, false);
    }
    public record Ack(long version, boolean duplicated) {}
}

Executor sharding is one hash line:

/** Each docId binds a single-thread Executor — space for correctness */
private ExecutorService docExecutor(String docId) {
    return docExecutors[Math.abs(docId.hashCode() % docExecutors.length)];
}

Production adds document-context local cache (Caffeine, expireAfterAccess(1h), max 100K doc sessions) so hot docs don't rebuild in-memory state each time; async snapshot triggered at version multiples of 100 (next section).

Five interview bonus points in this code: doc-level single-thread avoids complex locks → clientId+clientSeq idempotency → baseVersion conflict detection → server issues versions → persist-then-broadcast. Interviewer can pick any to drill.

5.2 Gateway & Broadcast: Netty Zero-Copy + Room Model

Gateway does three things: receive ops, local broadcast, cross-node forward. With Netty, zero-copy broadcast is a show-off optimization:

@ChannelHandler.Sharable @Component
public class DocCollabHandler extends SimpleChannelInboundHandler<TextWebSocketFrame> {
    // docId -> ChannelGroup (production combines with Redis Pub/Sub for cross-node broadcast)
    private final ConcurrentHashMap<String, Set<Channel>> docGroup = new ConcurrentHashMap<>();
    private final ObjectMapper mapper = new ObjectMapper();
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, TextWebSocketFrame frame) throws Exception {
        // 1. Parse client-reported operation
        CollabOperation op = mapper.readValue(frame.text(), CollabOperation.class);
        String docId = op.getDocId();
        // 2. Collaboration engine executes local merge, produces final consistent op
        CrdtMergeResult result = CollabEngine.merge(docId, op);
        // 3. Zero-copy broadcast to all other clients in same doc
        Set<Channel> channels = docGroup.get(docId);
        if (channels != null && !channels.isEmpty()) {
            ByteBuf msg = Unpooled.wrappedBuffer(mapper.writeValueAsBytes(result.getFinalOp()));
            for (Channel ch : channels) {
                if (ch != ctx.channel()) ch.writeAndFlush(new TextWebSocketFrame(msg.retain()));
            }
            msg.release();
        }
        // 4. Async write OpLog, periodic snapshots
        OpLogService.asyncSave(result.getFinalOp());
    }
}
msg.retain() / msg.release() reference counting lets one message serialize once, write to whole room — thousand-user room doesn't serialize thousand times. sendText always async, never block IO thread ; single connection send failure isolated, one user's network lag can't drag down whole room.

5.3 Cross-Node Broadcast: Users on Different Gateways

300K connections spread across dozens of gateway nodes; sessions are node-local. A on gateway 1, B on gateway 3 — how to communicate? Two mainstream approaches:

Approach 1: Redis Pub/Sub (low latency preferred). Each gateway subscribes only to channels for docs it has users in, avoiding full broadcast storm:

@Service @RequiredArgsConstructor
public class CollabGateway {
    /** docId -> (userId -> Session); gateway only holds rooms it serves */
    private final ConcurrentHashMap<String, ConcurrentHashMap<String, WebSocketSession>> rooms = new ConcurrentHashMap<>();
    private final RedisTemplate<String, String> redis;
    private final StringRedisTemplate stringRedis;
    /** ① Client sends an operation */
    public void onMessage(WebSocketSession session, EditOp op) {
        if (!permissionService.canEdit(op.getDocId(), op.getUserId())) {
            send(session, Result.fail("No edit permission 🚫")); return;
        }
        // Assign global monotonic seq (Redis INCR, naturally atomic, avoids multi-replica reorder)
        long seq = stringRedis.opsForValue().increment("doc:seq:" + op.getDocId());
        op.setSeq(seq);
        // ★ Write OpLog first (WAL), then broadcast — guarantees no message loss
        opLogProducer.sendAsync(op); // Kafka async persistence
        // Local room direct broadcast (saves one network hop)
        broadcastLocal(op.getDocId(), op);
        // Cross-node broadcast: publish to Redis channel keyed by docId
        redis.convertAndSend("collab:room:" + op.getDocId(), JsonUtil.toJson(op));
    }
    /** ② Subscribe other gateways' broadcasts, forward to local room members */
    @RedisListener(channel = "collab:room:*")
    public void onRemoteOp(String channel, String payload) {
        String docId = channel.substring("collab:room:".length());
        EditOp op = JsonUtil.fromJson(payload, EditOp.class);
        broadcastLocal(docId, op);
    }
    private void broadcastLocal(String docId, EditOp op) {
        ConcurrentHashMap<String, WebSocketSession> members = rooms.get(docId);
        if (members == null) return;
        String frame = JsonUtil.toJson(op);
        members.values().parallelStream().forEach(s -> {
            try { if (s.isOpen()) s.sendMessage(new TextMessage(frame)); }
            catch (IOException ignored) { } // single-point exception isolation, doesn't affect others ✅
        });
    }
    /** ③ Heartbeat + disconnect cleanup */
    @Scheduled(fixedRate = 30_000)
    public void heartbeat() {
        rooms.forEach((docId, members) -> members.values().removeIf(s -> !s.isOpen()));
    }
}

Approach 2: Kafka Distribution (broadcast + persistence share one pipe, better ordering, slightly higher latency):

@Component
public class CrossNodeMessageBroker {
    /** Broadcast op: send to Kafka, each WS node consumes and distributes */
    public void broadcastOperation(String documentId, CRDTOperation op) {
        kafkaTemplate.send("document-operations", documentId, JsonUtils.toJson(op));
    }
    /** Consume Kafka message and forward to local WebSocket clients */
    @KafkaListener(topics = "document-operations")
    public void handleOperation(String message) {
        CRDTOperation op = JsonUtils.fromJson(message, CRDTOperation.class);
        Set<String> connectedClients = documentSessionManager.getConnectedClients(op.getDocumentId());
        for (String clientId : connectedClients) {
            WebSocketSession session = sessionRegistry.getSession(clientId);
            if (session != null && session.isOpen()) {
                session.sendMessage(new TextMessage(JsonUtils.toJson(
                    WebSocketMessage.builder()
                        .type(MessageType.BROADCAST)
                        .operation(op)
                        .build())));
            }
        }
    }
}

Redis Pub/Sub : Latency — Millisecond, lower; Message Persistence — None, fire-and-forget; Best For — Real-time broadcast fan-out

Kafka : Latency — Slightly higher but controllable; Message Persistence — Yes, replayable; Best For — OpLog + broadcast unified

Common production combo: Redis Pub/Sub for real-time broadcast, Kafka for OpLog persistence — each plays to its strength. Also Redis INCR for seq is lightweight and atomic; if hotspot worry, shard by docId across Redis instances, or let doc primary generate seq locally then async report, trading strictness for throughput.

5.4 Client "Unacked Queue + Buffer Replay": Optimistic UI Three-Part Structure

Server can be fancy, but client must catch. Classic OT client structure separates pending (sent, awaiting ACK) and buffer (local input not yet sent) — Google's ot.js does exactly this:

public class OtClient {
    private final Deque<Op> pending = new ArrayDeque<>(); // sent, unacked
    private final Deque<Op> buffer  = new ArrayDeque<>(); // local input not yet sent
    private long version;
    /** Local input: push to buffer, merge into batch every 50ms then send (saves 90% requests) */
    public void localEdit(Op op) {
        buffer.add(op);
        renderLocally(op); // optimistic render, input never blocks
    }
    /** Receive server broadcast */
    public void onRemote(OpEvent e) {
        pending.removeIf(o -> o.clientSeq() == e.clientSeq()); // own op acknowledged
        // 🔑 Key: transform remote op past local pending, then transform pending past remote op
        for (Op remote : e.ops()) {
            Deque<Op> newPending = new ArrayDeque<>();
            for (Op p : pending) {
                remote = OT.transform(remote, p, tieBreak(p));
                newPending.add(OT.transform(p, remote, !tieBreak(p)));
            }
            pending = newPending;
            applyToDocument(remote);
        }
        version = e.version();
        resendBuffer(); // after catch-up, immediately flush buffer
    }
    public void onAck(long v) { version = v; resendBuffer(); }
}

6. Storage & Versioning: Snapshot + OpLog, Golden Pair

Absolutely never write full document on every save — a few dozen KB doc, 10K editors blows up DB. Standard answer: classic Snapshot + OpLog (Git-style):

1. Find latest snapshot (index O(logN))
2. Replay OpLog after snapshot
3. Get current document content + version

Example: snapshot version = 1000, op log 1001~1080
  Final doc = snapshot + replay(1001~1080)

Simplified storage model, five tables: doc_meta: Doc title, creator, update time doc_member: Member permissions doc_op: Incremental edit ops (append-only, immutable, natural audit support) doc_snapshot: Periodic snapshots doc_ack: Client submit idempotency records

Kafka consumer for aggregation persistence + periodic snapshots:

@Service
public class VersionService {
    private static final int SNAPSHOT_INTERVAL = 200; // snapshot every 200 ops
    /** Scheduled aggregation persistence (consume Kafka) */
    @KafkaListener(topics = "collab-oplog")
    public void persist(EditOp op) {
        opLogMapper.insert(op); // append write, sequential IO, very fast
        if (op.getSeq() % SNAPSHOT_INTERVAL == 0) {
            DocState state = replay(op.getDocId(), op.getSeq());
            snapshotMapper.save(new Snapshot(op.getDocId(), op.getSeq(),
                JsonUtil.toJson(state), now()));
        }
    }
    /** Load arbitrary version: nearest snapshot + incremental replay */
    public DocState load(String docId, long targetSeq) {
        Snapshot snap = snapshotMapper.findNearestLess(docId, targetSeq); // index O(logN)
        DocState state = snap == null ? DocState.empty()
            : JsonUtil.fromJson(snap.getContent(), DocState.class);
        for (EditOp op : opLogMapper.findRange(docId, snap.getSeq() + 1, targetSeq)) {
            state.apply(op); // replay, like Git replay
        }
        return state;
    }
}
✅ Performance gain: Append-write + periodic snapshots turn "full write" into "sequential IO"; restore arbitrary version time from O(full) to O(snapshot + 200 incremental), second-level rollback, storage cost ~70% less than full retention.

Prevent OpLog Bloat: Snapshot Compaction

OpLog grows forever, needs background compaction:

@Scheduled(fixedDelay = 30_000)
public void compact() {
    for (String docId : hotDocs()) {
        DocSession s = sessions.get(docId);
        // Trigger: accumulated ops > 1000 OR time since last snapshot > 5min
        if (s.opsSinceSnapshot() < 1000 && s.sinceSnapshotMs() < 300_000) continue;
        Snapshot snap = Snapshot.of(s.version(), s.text());
        objectStore.put("snap/" + docId + "/" + s.version(), snap);
        opLog.truncateBefore(docId, s.version()); // old ops deletable, but keep recent 100 for undo/audit
        s.markSnapshot();
    }
}

Hot/cold separation: hot ops in Redis/MySQL, cold historical ops sink to object storage.

Incremental Sync: Don't Chase If Too Far Behind

When client syncs with version, server must make "downgrade decision":

/** Incremental sync service: only send changed ops, avoid full sync */
public List<CRDTOperation> getIncrementalOperations(
        String documentId, String clientId, int clientVersion) {
    int serverVersion = getServerVersion(documentId);
    // Client far behind (offline half day), replaying one-by-one not worth it → send full snapshot
    if (serverVersion - clientVersion > MAX_DELTA_SIZE) {
        DocumentState fullState = documentService.getDocument(documentId);
        return List.of(CRDTOperation.builder()
            .type(OperationType.FULL_SYNC)
            .content(JsonUtils.toJson(fullState))
            .build());
    }
    // Normal: return only incremental ops
    return operationRepository.findByDocumentIdAndVersionAfter(documentId, clientVersion);
}
Same for OT: "transform chain too long" uses same fallback: limit max 500 sequential transforms, beyond that send full snapshot. Catch up if possible, otherwise reset — practical distributed systems fallback philosophy.

7. Reconnection & Offline Editing: No Loss, No Duplication, Mergeable

This is often missed by candidates but always probed by interviewers.

Four key designs:

Local unacked queue : user types → local apply immediately → into pendingOps → send server → ACK removes → timeout retry. Queue persisted to IndexedDB, survives page refresh.

Idempotent retry : each op carries clientId + clientSeq; server sees duplicate key, returns success without reapplying (Redis Set / Bloom filter deduplication).

Reconnect incremental pull : client sends lastVersion, server returns "snapshot + ops after that version", no full refresh.

Offline merge strategy : CRDT naturally supports offline merge; OT treats offline ops as a patch, does three-way merge, highlights truly conflicting regions for manual resolution. Server failure → client continues editing locally, on reconnect auto-merges all offline ops — baseline for degraded experience.

Offline operation queue reference:

@Service
public class OfflineOperationQueue {
    private final Queue<CRDTOperation> localQueue = new ConcurrentLinkedQueue<>();
    /** Add op to local queue and persist */
    public void enqueueOperation(CRDTOperation op) {
        op.setStatus(OperationStatus.PENDING);
        localQueue.offer(op);
        saveToLocalDB(op); // persist to IndexedDB
    }
    /** Try batch sync every 5 seconds */
    @Scheduled(fixedDelay = 5000)
    public void syncToServer() {
        if (!networkService.isConnected()) return;
        List<CRDTOperation> batch = new ArrayList<>();
        localQueue.drainTo(batch);
        if (!batch.isEmpty()) {
            try {
                documentService.batchApplyOperations(batch);
                batch.forEach(this::removeFromLocalDB); // success clears local
            } catch (Exception e) {
                batch.forEach(localQueue::offer); // failure re-queues
                log.error("Sync failed, ops will retry", e);
            }
        }
    }
}

8. Cursors & Presence: Never On Main Path

Two users editing simultaneously see each other's cursors/selections. Looks simple, traps are in volume — mousemove fires dozens per second, mishandled pushes real edit messages out.

Who's editing : Redis Hash(docId → {userId: heartbeat}) + TTL 30s — Heartbeat renewal, natural expiry, avoids zombie state

Cursor position : No persistence , independent presence channel, 300ms debounce + rate limit 10/sec — Cursor is "volatile data", loss acceptable

Selection highlight : Cursor message carries range — Priority lower than text ops

Three iron rules:

Cursor not via OT, not in version , independent Awareness channel, last-write-wins + TTL sufficient.

Cursor messages and text messages must separate channels — message QoS tiering: high-frequency volatile data must not crowd edit main path bandwidth.

Cursor position also needs transformation : someone inserts 5 chars before you, your cursor shifts right 5, else two users typing at same position cursors jump. Local ops render first, remote ops adjust cursor offset separately; only broadcast to "same room and in viewport" people, reducing useless pushes.

9. Permissions & Security: Trust Nothing From Client

Collaborative systems assume client untrustworthy. RBAC four-level + bitmask suggested:

User → Role(Owner / Editor / Commenter / Viewer) → Document
      └─ Permission bitmask: 0b1111 = read | write | comment | share

Server-side verification checklist:

Doc-level permission : View, edit, comment, manage; gateway entry + collaboration service dual interception

Operation-level privilege escalation : Read-only users cannot submit edits; every op double-checked canEdit(docId, uid), prevents gateway bypass

Document state : Admin lock, read-only mode

Operation legality : Position bounds, message size limit

Content compliance : Risk control, sensitive words, operation audit (OpLog append-only naturally supports)

One-liner: Client only renders and submits intent; server is ultimate authority. Sensitive ops (share/delete) go through separate strong-consistency interface (MySQL transaction).

Operation-level permission via AOP unified interception:

@Service
public class DocumentPermissionService {
    public boolean canEdit(String documentId, String userId) {
        DocumentPermission perm = permissionRepository.findByDocumentIdAndUserId(documentId, userId);
        if (perm == null) return false; // default no permission
        return perm.getRole() == Role.EDITOR || perm.getRole() == Role.ADMIN;
    }
}

@Aspect @Component
public class PermissionAspect {
    @Before("@annotation(RequireEdit)")
    public void checkEditPermission(JoinPoint joinPoint) {
        Object[] args = joinPoint.getArgs();
        String documentId = (String) args[0];
        String userId = (String) args[1];
        CRDTOperation op = (CRDTOperation) args[2];
        // Comment ops allowed for comment-permission users
        if (op.getType() == OperationType.COMMENT && permissionService.canComment(documentId, userId)) return;
        if (!permissionService.canEdit(documentId, userId)) {
            throw new PermissionDeniedException("You lack edit permission for this document");
        }
    }
}

Anti-malicious flooding: gateway connection-level rate limit (token bucket, e.g., 20 op/s) + message size cap + single-room user cap, excess downgraded to read-only.

10. Large Documents & Performance: Don't Operate on Big Strings

100K, 1M char docs — naive impl freezes. Optimization combo:

Data structure : Piece Table (VS Code style) or Rope , insert O(1), avoid large string frequent copies.

Block loading : doc split by Block (paragraph / 4KB chunk), only sync changed blocks + viewport content, lazy load on scroll.

Virtual scrolling : 100K chars render only viewport 100 lines; server can also send only viewport content with total height for scrollbar.

CRDT-specific : string compression storage + periodic tombstone GC cleans unreferenced deleted nodes, contains 3-5x metadata bloat.

Broadcast throttling : 50ms op merge, push diff not full doc; read-only spectators on separate replica + CDN/polling fallback , 99% read traffic peeled off hot path.

Pre-launch metrics (reference):

End-to-end latency : < 100ms — User imperceptible (WebSocket measured 35-50ms)

Single-doc concurrency : 50-100 editors no pressure — Beyond limit upgrade to read-only + comment mode; 1K+ needs special optimization

Op delivery rate : 99.999% — Must never lose user input

Consistency : Eventual, all replicas converge within 1s — Forbid two users seeing different content

History versions : At least 30 days, any point rollback — Compliance

11. Difficulty Cheat Sheet (Memorize, Recite in Interview)

Concurrent edits don't converge — Multi-client op order inconsistent → OT transform + server global version sequencing + siteId tie-break , or CRDT

Transform function combinatorial explosion — Many op pair combinations → Only support atomic Insert/Delete, decompose complex ops; property-based fuzz testing guards correctness

Ops arrive out of order — Network doesn't guarantee order → docId consistent hash routes to single collaboration node single-thread sequencing

Message loss/duplication — Network jitter, retries → Version + ACK retransmission + opId idempotency + OpLog fallback + reconnect incremental pull

Cross-node message fan-out — WS sessions node-local → Redis Pub/Sub room channel broadcast , subscribe only docs with users

Broadcast storm — Large room fan-out volume → 50ms op merge, push diff only, spectators on separate replica

OpLog storage bloat — One record per keystroke → Snapshot + OpLog periodic checkpoint compression, hot/cold separation

Transform chain too long — Long offline → many historical ops → Limit max 500 sequential transforms, beyond send full snapshot

Cursor message flood — mousemove high frequency → Presence independent channel + rate limit 10/s + 300ms debounce, never persist

Same-position cursor jump — Cursors affect each other → Cursor independent sync + position transform, local ops priority

Offline edit conflicts — Long offline, many changes → Local queue + reconnect two-way rebase; OT three-way merge + conflict highlight manual resolution

Mistake irreversible — Someone deletes half doc → OpLog full retention + arbitrary version rollback + diff compare + key node manual version tags

Large doc lag — Full diff / big string → Piece Table + Block chunking + virtual scrolling + incremental ops

Undo semantics confusion — Whose op to undo in collaboration → Inverse op stack + transform before apply; only undo own ops

Long-connection HA — Gateway node failure → Stateless gateway + heartbeat + reconnect drift + session graceful migration

Permission escalation — Bypass gateway submit op → Gateway auth + collaboration service per-op double-check + operation audit

Unicode position chaos — Emoji / Chinese surrogate pairs → Position by code point / grapheme, not char Hot doc thousands spectators — Read traffic crushes main path → Read/write separation: editors on main path, spectators on read-only replica + CDN

Undo extra note: undo stack stores inverse ops (Insert ↔ Delete), pop transforms against all subsequent ops before applying — undo "intent" not "text" , and collaborative undo of others' ops is counter-intuitive, only own ops allowed.

12. Interviewer High-Frequency Follow-ups (With Short Answers)

Q1: Why not DB row locks / distributed locks for conflicts? Locks suit short transactions. User edits are long, unpredictable; holding lock tens of seconds kills throughput; offline can't hold locks. Collaborative editing essence is "merge" not "mutex" , hence op transform + sequencing.

Q2: Two users insert at same position, how guarantee same order everywhere? Use (version, siteId) deterministic tie-break: server-sequenced first applies, same version tie-break by siteId lexicographic. All clients use same rule → inevitable convergence.

Q3: Server crashes, will in-flight edits be lost? Op written to OpLog (Redis AOF / Kafka persistence) before broadcast; in-memory state rebuildable from "latest snapshot + OpLog". RPO depends on flush policy — async per op + 100ms forced fsync → loss window < 100ms. Write MQ first then ACK, or at least async OpLog + snapshots — baseline.

Q4: Why not just send full text every time? 100K doc, insert one char: full text 200KB, op only dozens bytes, 3-4 orders magnitude difference. Full text is last-write-wins, simultaneous typing swallows each other's content .

Q5: How verify OT implementation correct? Property testing (random op sequences + random concurrent interleavings, assert all clients converge) + TP2 transform property verification. Pass both → basically stable.

Q6: If switch to CRDT, what changes? Swap to Yjs-style RGA: each char carries (siteId, clock) unique ID, delete uses tombstone. Benefits: true offline, P2P; costs: memory bloat (100K chars ~5-10MB metadata) and tombstone GC complexity. Server may not even understand content, just relay.

Q7: Reconnection handling? Client sends lastVersion, server supplements missing ops, local rebase; lag beyond threshold → fetch snapshot.

Q8: Prevent malicious user flooding? Gateway connection-level rate limit (token bucket 20 op/s) + message size cap + single-room user cap, excess downgraded to read-only.

13. From MVP to Production: Don't Chase Perfect Collaboration Upfront

If interviewer asks "limited resources, how to land", don't dump full stack above — shows you only stack buzzwords. Phase it:

Phased rollout
Phased rollout

Guarantee no content loss first, then multi-user, then real-time, finally offline. Each step independently deployable, independently rollbackable — that's engineering mindset.

Post-launch engineering essentials:

Monitoring metrics : op submit QPS, broadcast latency P99, ACK timeout rate, conflict transform count, single-doc online users, OpLog backlog, snapshot replay time — full-chain tracing, auto-alert on anomaly.

Canary strategy : by document dimension (ordinary docs → internal docs → high-activity team docs → full), not by user, else same doc might hit two logic versions.

Stress test focus : single-doc 100 concurrent editors, single gateway 100K long connections, merge capacity under high-frequency input, incremental pull during reconnect storms.

Advanced directions : multi-region nearby access, Local-First eliminating perceived latency, AI smart conflict resolution, track-changes review mode.

Closing Summary

Three lines capture the skeleton:

Conflict line : OT/CRDT makes concurrent edits converge — this is the eye, transform function and tie-break rules must be writable from memory;

Communication line : stateless gateway + room routing + Redis/Kafka broadcast — supports hundreds of thousands long connections;

Storage line : OpLog + Snapshot — backs version rollback, offline recovery, audit.

One-sentence system summary: Use "central sequencing" to downgrade distributed concurrency to single-machine serial, use "OT/CRDT" to guarantee multi-client convergence, use "Snapshot + OpLog" to bottom-out no data loss. The rest — idempotency, retry, chunking, auth — are engineering grunt work.

Two landing tips: 1) Don't write algorithms from scratch , prefer mature open-source like Yjs, Automerge for secondary dev, focus energy on connection stability, permissions, version rollback — the business experience; 2) Always have fallbacks — no matter how clever algorithm, history version rollback must exist; user data lost, nothing else matters.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Distributed SystemsJavasystem designWebSocketconflict resolutionversion controlsnapshotreal-time synchronizationCRDTcollaborative editingoperational transformationOT
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.