Optimizing Real-Time Kafka Writes in Spark Streaming Using a Broadcasted KafkaProducer

In many Spark Streaming projects, developers write each RDD partition's data to Kafka by creating a new KafkaProducer inside the partition loop, which leads to severe performance degradation.

The article first shows the naive implementation where kafkaStreams.foreachRDD creates a producer for every partition, setting properties such as group.id, acks, retries, bootstrap servers, and serializers.

To avoid this overhead, a lazily‑initialized wrapper class broadcastKafkaProducer is introduced; it defines a lazy val producer = createproducer() and provides send methods that return a Future[RecordMetadata].

Next, the wrapper is instantiated and broadcast to all executors using Spark's Broadcast mechanism, with the same producer configuration as before.

Finally, the streaming job uses the broadcasted producer inside foreachRDD and foreachPartition to send records, eliminating per‑partition producer creation and achieving dozens‑fold speed improvements.