Monthly Archives: September 2015

Spark: Exception in thread “streaming-job-executor-2” java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror

  Following exception gets thrown if your uber/fat jar used by “bin/spark-submit” is missing some libraries. Note that due to the delayed execution the line that actually causes the exception could be somewhere before … Exception in thread “streaming-job-executor-2” java.lang.NoSuchMethodError: … Continue reading

Posted in Uncategorized | Leave a comment

Spark: java.io.InvalidClassException: scala.Option; local class incompatible: stream classdesc serialVersionUID local class serialVersionUID

The following exception is thrown when launching spark program with “sbt run”. Instead use “bin/spark-submit” and the exception will go away.   Exception Caused by: java.io.InvalidClassException: scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388 … Continue reading

Posted in Uncategorized | Leave a comment

com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name ‘id’ (in class org.apache.spark.rdd.RDDOperationScope)

When running a basic Spark Streaming program the following exception was thrown. Turns out, there is a conflict between jackson version; which could be solved by adding to your build.sbt the following: dependencyOverrides ++= Set( “com.fasterxml.jackson.core” % “jackson-databind” % “2.4.4” … Continue reading

Posted in Uncategorized | Leave a comment

Spark Exception toDF

val sqlContext = new SQLContext(sc) import sqlContext.implicits._ val df = rdd.toDF() Throws exception (below). Solution It seems that some of the classes are not included when running “sbt package” and then invoking spark with “spark-submit”.  This could be confirmed by running … Continue reading

Posted in Uncategorized | Leave a comment

SparkException: ExecutorLostFailure (executor driver lost)

In attempt to simplify things; I was running a self-contained (standalone) spark by using “sbt run” (as was suggested by the earlier versions of the docs).   However, it was quite unstable with spark throwing cryptic errors (see example below). … Continue reading

Posted in Uncategorized | 2 Comments