Spark Streaming : NoSuchMethodError

When running a simple spark streaming app; I was getting the NoSuchMethodError  exception (shown below).

Initially, I thought it was caused by converting RDD to DataFrame (needed by ML pipeline); but it wasn’t.  In the end it turned out that I had different versions of scala on the driver-side and the spark server.  As of Sep.2015,  precompiled version are for scala 2.10; moreover to use the scripts to deploy spark to EC has dependency on scala 2.10.3.  My driver was using 2.11.7.  After recompiling w/ 2.10 the error went away.

 

 

 

Convert RDD to DataFrame

// Get the singleton instance of SQLContext
val sqlContext = SQLContextSingleton.getInstance(sc)
import sqlContext.implicits._

val df = rdd.toDF()

Exception

15/10/01 17:47:33 INFO DAGScheduler: Job 2 finished: foreachRDD at CustomReceiver.scala:61, took 0.000030 s
Exception in thread "streaming-job-executor-1" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
 at CustomReceiver$$anonfun$main$1.apply(CustomReceiver.scala:75)
 at CustomReceiver$$anonfun$main$1.apply(CustomReceiver.scala:61)
 at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
 at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:631)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:42)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:40)
 at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:399)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:40)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
 at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
 at scala.util.Try$.apply(Try.scala:161)
 at org.apache.spark.streaming.scheduler.Job.run(Job.scala:34)
 at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:207)
 at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
 at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:207)
 at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
 at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:206)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

 

About Neil Rubens

see http://ActiveIntelligence.org

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*