Unlock Seamless Object Serialization & Checkpoint Recovery in Spark with Neutrino

Extending Neutrino: SerializableProvider and Object Lifecycle

Neutrino provides a low‑level API called SerializableProvider that combines Serializable and Provider interfaces, allowing seamless transmission of non‑inheritable (final) classes—similar to the Provider concept in Spring or Guice.

When binding a final class such as EventProcessor, the usual proxy generation fails because a proxy cannot inherit from a final class. Using SerializableProvider solves this problem.

Example click‑event handling code (illustrated in the image) demonstrates the compilation error that occurs when a final class is bound with the default proxy approach.

Switching to SerializableProvider eliminates the error, as shown in the alternative binding method.

Binding the final class EventProcessor together with its SerializableProvider enables injection of the provider and seamless object creation.

The provider carries the dependency‑graph node ID, allowing Neutrino to recreate the object on the worker side using the same graph.

Handling Mutable State and Checkpoint Recovery

Neutrino’s serialization works well for immutable objects, but mutable state modified on the driver (e.g., cache‑level configuration) must also be transferred to workers. One approach is to bind such objects with SerializableProvider or as plain Serializable instances.

In a streaming job, the cache level is read from a configuration center each interval and stored in HttpEventEnrichment. Because the enrichment object’s state changes after injection, it cannot use the standard withSerializableProxy binding; instead it is passed as a regular Java object implementing Serializable.

The corresponding EnrichmentModule code (shown below) demonstrates this binding.

When checkpointing is enabled, handlers used only on the driver still need to be serializable because they participate in the RDD DAG reconstruction.

Neutrino automatically recreates such objects during checkpoint recovery by rebuilding the dependency graph on each JVM.

Injecting Core Spark Objects

Neutrino also supports injection of key Spark components (e.g., SparkSession, SparkContext, StreamingContext) for use on the driver side.

After binding, a DStream<ClickEvent> can be obtained directly from the injector.

Neutrino also supports parent/child injector hierarchies, mirroring Guice and Spring capabilities.

Overall, Neutrino’s extended APIs enable passing arbitrary object types—including final classes—manage mutable state across driver and worker, and simplify checkpoint recovery in Spark streaming applications.