Resolving Spark Task Not Serializable Errors: Causes, Code Examples, and Best Practices

When writing Spark programs, using external variables or functions inside operators like map and filter can trigger the Task not serializable exception. Although referencing external data is often necessary, the referenced class must be fully serializable, otherwise Spark cannot ship the closure to executors.

Example 1 – Member variable reference

class MyTest1(conf: String) extends Serializable {
  val list = List("a.com", "www.b.com", "a.cn", "a.com.cn", "a.org")
  private val sparkConf = new SparkConf().setAppName("AppName")
  private val sc = new SparkContext(sparkConf)
  val rdd = sc.parallelize(list)
  private val rootDomain = conf
  def getResult(): Array[(String)] = {
    val result = rdd.filter(item => item.contains(rootDomain))
    result.take(result.count().toInt)
  }
}

Running this code produces the following error because SparkContext (and later SparkConf ) cannot be serialized:

Exception in thread "main" org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
    ...
Caused by: java.io.NotSerializableException: org.apache.spark.SparkContext
    - field (class "com.ntci.test.MyTest1", name: "sc", type: "class org.apache.spark.SparkContext")

Marking the non‑serializable members with @transient resolves the issue:

class MyTest1(conf: String) extends Serializable {
  val list = List("a.com", "www.b.com", "a.cn", "a.com.cn", "a.org")
  @transient private val sparkConf = new SparkConf().setAppName("AppName")
  @transient private val sc = new SparkContext(sparkConf)
  val rdd = sc.parallelize(list)
  private val rootDomain = conf
  def getResult(): Array[(String)] = {
    val result = rdd.filter(item => item.contains(rootDomain))
    result.take(result.count().toInt)
  }
}

Example 2 – Member function reference

class MyTest1(conf: String) extends Serializable {
  val list = List("a.com", "www.b.com", "a.cn", "a.com.cn", "a.org")
  private val sparkConf = new SparkConf().setAppName("AppName")
  private val sc = new SparkContext(sparkConf)
  val rdd = sc.parallelize(list)
  def getResult(): Array[(String)] = {
    val rootDomain = conf
    val result = rdd.filter(item => item.contains(rootDomain))
                     .map(item => addWWW(item))
    result.take(result.count().toInt)
  }
  def addWWW(str: String): String = {
    if (str.startsWith("www.")) str else "www." + str
  }
}

Again the program fails unless sparkConf and sc are marked @transient . Moving addWWW to a Scala object (static‑like) eliminates the need to serialize the enclosing class:

def getResult(): Array[(String)] = {
  val rootDomain = conf
  val result = rdd.filter(item => item.contains(rootDomain))
                   .map(item => UtilTool.addWWW(item))
  result.take(result.count().toInt)
}

object UtilTool {
  def addWWW(str: String): String = {
    if (str.startsWith("www.")) str else "www." + str
  }
}

Full‑class serialization verification

If the extends Serializable clause is removed after applying @transient , Spark throws a NotSerializableException for the whole class, confirming that any closure referencing a class member forces the entire class to be serializable.

Practical recommendations

Avoid directly referencing class member variables or functions inside Spark closures whenever possible; instead define needed values locally or in a companion object .

If such references are unavoidable, ensure the enclosing class implements Serializable and mark non‑serializable members with @transient .

Consider extracting independent logic into static‑like objects or small serializable helper classes to reduce serialization overhead.

By following these guidelines, Spark applications can prevent the common "Task not serializable" error and achieve more reliable distributed processing.