Monthly Archives: June 2015

ScalaDoc: UnOrdered List

There is a listing of wiki-like markup that ScalaDoc supports.  Unfortunately the example of unordered lists was somewhat confusing: Unordered list item: $ – item becomes a list item ($ signifies the left margin of a line (it doesn’t appear … Continue reading

Posted in Uncategorized | Leave a comment

ScalaDoc: package level documentation is missing

Follow the official guide. Add to `build.sbt`: scalacOptions in (Compile,doc) ++= Seq(“-groups”, “-implicits”) Execute: sbt doc

Posted in Uncategorized | Leave a comment

value toDF is not a member of org.apache.spark.rdd.RDD

Problem value toDF is not a member of org.apache.spark.rdd.RDD Solution Need to add: val sqlContext = new SQLContext(sc) import sqlContext.implicits._     [error] ^ [error] one error found [error] (compile:compileIncremental) Compilation failed

Posted in Uncategorized | 1 Comment

Spark-streaming: java.lang.ClassNotFoundException: org.apache.spark…

Problem java.lang.ClassNotFoundException: org.apache.spark… Solution Deploying: Generate an uber JAR with all the dependencies (including the dependency included in sbt and their transitive dependencies) and then deploy the application. This is further explained in the Deploying section. https://spark.apache.org/docs/latest/streaming-programming-guide.html

Posted in Uncategorized | Leave a comment

Fixing’s Spark: java.io.IOException: java.lang.ClassNotFoundException: scala.collection.mutable.WrappedArray$ofRef

Problem When running apache spark was getting the following exception: java.io.IOException: java.lang.ClassNotFoundException: scala.collection.mutable.WrappedArray$ofRef Solution   Adding to build.sbt fork := true Fixed the problem.  It seems that mixing scala 2.10 and 2.11 causes this error.  Here is a related reference.

Posted in Uncategorized | Leave a comment

Building Apache Spark with SBT for scala 2.11

./build/sbt -Dscala-2.11 -DskipTests  publish-local

Posted in Uncategorized | Leave a comment