How Neutrino Solves Dependency Injection Challenges in Spark Jobs

Dependency injection (DI) is a common object‑oriented design pattern that decouples a module from the concrete implementations of the components it depends on.

In the traditional DI model the container creates and injects required dependencies, turning a tightly‑coupled relationship into a loosely‑coupled one.

Large projects often have deep, complex dependency hierarchies. The following code diagram illustrates a class

Upper1

that depends on

Medium1

and

Medium2

, which in turn depend on other classes, forming a directed acyclic graph.

The container (e.g., Spring, Guice) registers these relationships and can instantiate objects on demand by traversing the graph.

Neutrino is an open‑source framework created by Russell Bie at Hulu’s Content Discovery team to address DI problems on the Spark platform.

The framework emerged while building a near‑real‑time model‑training platform on Spark streaming, where algorithms and recommendation scenarios needed to be cleanly separated. Over three years, the codebase evolved to handle Spark’s distributed nature, was patented, and eventually open‑sourced at https://github.com/disneystreaming/neutrino .

Neutrino focuses on serializing DI objects and their direct and indirect dependencies on Spark. Built on Guice, it automatically serializes objects between the driver and workers, and extends the container’s scope management to workers.

Standard Java DI frameworks assume a single JVM. In Spark, the driver JVM coordinates many worker JVMs, making it necessary to pass objects from driver to workers. This requires serializing the entire object graph, which can be cumbersome, especially for non‑serializable resources such as network or database connections.

Consider an event‑enrichment scenario where a click event must be enriched with product details fetched via an HTTP API. The enrichment logic is defined by an

EventEnrichment

interface and bound in Guice on the driver. The resulting

HttpEventEnrichment

instance must be sent to workers, but its

HttpClient

dependency is not serializable.

Using Neutrino, the developer binds the enrichment module as usual; Neutrino generates a serializable provider that creates the real

HttpEventEnrichment

on each worker, handling the non‑serializable

HttpClient

via a static reference.

The provider carries only a small payload containing the node ID from the dependency graph. Because the same graph exists on workers, the proxy uses the ID to reconstruct the full object and its dependencies locally, eliminating the need to serialize the entire object graph.

This approach also enables scoped lifecycles across workers; for example, a singleton binding ensures the same instance is reused on a worker after the first creation.

Modules themselves must be serializable, which is easier than serializing every object. After binding the enrichment module, the injector is created as usual.

A limitation of the current proxy mechanism is that the generated proxy inherits from the original interface, so the bound class must be inheritable.

This article introduced the difficulties of applying DI to Spark jobs and demonstrated how Neutrino resolves them. The next article will dive deeper into advanced features such as arbitrary object transmission, checkpoint recovery, and lifecycle control.