Understanding Kafka Schema Registry Compatibility Types and Schema Evolution

In this article we explore schema evolution and compatibility types in Kafka using the Schema Registry, showing how a solid grasp of compatibility prevents breaking producer or consumer contracts over time.

We use a real‑world example from a "Hadoop Developer in the Real World" course where Meetup.com RSVP data is streamed to Kafka. The Avro schema for an RSVP record is:

{ "namespace": "com.hirw.kafkaschemaregistry.producer", "type": "record", "name": "Rsvp", "fields": [ { "name": "rsvp_id", "type": "long" }, { "name": "group_name", "type": "string" }, { "name": "event_id", "type": "string" }, { "name": "event_name", "type": "string" }, { "name": "member_id", "type": "int" }, { "name": "member_name", "type": "string" } ] }

The producer (a Spring Kafka project) writes messages using this schema and Avro serialization; the consumer reads them with the same schema.

If the producer removes a required field such as member_id , consumers will encounter deserialization errors like org.apache.avro.AvroTypeException: ... missing required field member_id , illustrating why compatibility matters.

Kafka offers several compatibility types. BACKWARD allows new schemas to be read by consumers using the old schema; deleting a required field is considered backward compatible only if consumers can still read old data, but it may still break existing clients.

Adding a field without a default value is not backward compatible because consumers expecting the field will fail; however, adding a field with a default value is allowed.

BACKWARD_TRANSITIVE checks the new schema against all previous versions, ensuring full backward compatibility across the schema history.

FORWARD ensures that data produced with the current schema can be read by consumers using the new schema; this prevents breaking changes when removing required fields, but adding non‑default fields remains disallowed.

FORWARD_TRANSITIVE extends the forward check to all registered versions.

FULL requires both backward and forward compatibility, allowing only additions or deletions of fields that have default values. FULL_TRANSITIVE applies the same rule across the entire version chain.

NONE disables all compatibility checks, permitting any change (generally unsafe for production).

To change a topic's compatibility type, send a PUT request to the Schema Registry with the desired compatibility (e.g., FORWARD ). After the change, attempts to delete a required field like event_id will be rejected, confirming the new compatibility enforcement.

Finally, a POST request can be used to register a new schema version. An example payload that adds a venue_name field with a default value is:

{"schema":"{\"type\":\"record\",\"name\":\"Rsvp\",\"namespace\":\"com.hirw.kafkaschemaregistry.producer\",\"fields\":[{\"name\":\"rsvp_id\",\"type\":\"long\"},{\"name\":\"group_name\",\"type\":\"string\"},{\"name\":\"event_name\",\"type\":\"string\"},{\"name\":\"member_name\",\"type\":\"string\"},{\"name\":\"venue_name\",\"type\":\"string\",\"default\":\"Not Available\"}] }"}

This demonstrates how the Schema Registry enforces compatibility rules to protect consumers while allowing safe schema evolution.