How Extended NSQ Message Formats Enable Load Testing, Isolation, and Filtering

Extended NSQ Message Format Design

To support more use cases, Youzan middleware adds an extensible message format to NSQ, introducing three new fields (shown in green in the diagram):

Version of extension content – 1 byte to identify the extension type (e.g., 0x01 for JSON).

Length of extension content – 2 bytes indicating the byte length.

Extension content – a variable‑length binary string.

These additions allow NSQ to carry extra information without altering the original message structure, enabling new functionalities.

Use Case: Load‑Testing

During load‑testing, producers tag messages as test data in the extension header. NSQ delivers both production and test messages through the same topic. Consumers inspect the extension field to decide whether to process or discard test traffic, allowing testing without changing topic configurations. The drawback is that production and test data share a topic, making isolation complex if errors occur.

Use Case: Service Isolation (QA Environments)

In QA, two application versions coexist: a stable version and a new‑feature version. The extended header carries a tag that indicates the target environment. NSQ routes messages based on three rules:

If no consumer matches the tag, the message goes to the stable QA consumer.

If a consumer with a matching tag exists, the message is delivered to that consumer.

If the message lacks a tag, it defaults to the stable QA consumer.

This routing enables isolated testing of new features without disrupting the stable environment.

Use Case: Message Filtering

NSQ delivers messages to channels in a multicast fashion, with consumers competing for each message. By setting a filter_key in the extension header, producers can control which consumers receive a message:

Messages without a filter_key are delivered only to channels lacking a filter_key.

Messages with a filter_key are delivered to matching channels, while channels without a filter_key also receive them.

Channels whose filter_key does not match are considered to have already consumed the message and receive nothing.

This mechanism can be implemented on the server, the client, or both.

NSQ Migration Proxy Tool

The migrate‑proxy enables seamless migration from open‑source NSQ to Youzan’s custom NSQ version. It acts as a proxy, merging lookup results from both clusters and routing read/write traffic based on r/w parameters. The migration proceeds in three stages:

Stage 1: Lookup responses contain nodes from both clusters; consumers connect to both, while producers continue using the open‑source cluster.

Stage 2: Lookup for producers returns only the target cluster, shifting production to the new cluster while consumers still consume from both.

Stage 3: After confirming all messages in the old cluster are consumed, lookup for consumers returns only the target cluster, completing the migration.

Figures illustrating each stage are included in the original article.

Connectors

Spark connector : A Spark consumer reads NSQ messages and processes them via Spark Streaming.

Flume sink : An Apache Flume sink extension sends events to NSQ, persisting failed events locally for retry and allowing users to configure NSQ as a downstream target.

Future Plans

The team plans to add several features:

Flow control : Token‑bucket based rate limiting to protect the cluster from high‑traffic topics.

Batch subscription : Consuming and acknowledging groups of messages at once to improve throughput in high‑latency networks.

Enhanced security auditing : Independent authentication for topic production and channel operations, plus unified login for nsqadmin.

Distributed transaction coordinator : A framework built on NSQ to address consistency across microservices.

Advanced content‑based filtering : A more complex filtering system built atop the extension header to avoid tightly coupling business logic with NSQ core code.

Conclusion

The article presented Youzan’s extended NSQ message format, demonstrated its capabilities through three practical scenarios, described migration tooling and Spark/Flume integrations, and outlined upcoming enhancements to further broaden NSQ’s applicability.