Monthly Archives: December 2015

Rename DataFrame Column

val df = rdd.toDF().withColumnRenamed(“from_col”, “to_col”)   apache spark rdd

Posted in Uncategorized | Leave a comment

Spark DataFrame Row containing Nested Case Class

nested DF: http://stackoverflow.com/questions/30501300/is-spark-dataframe-nested-structure-limited-for-selection http://stackoverflow.com/questions/30008127/how-to-read-a-nested-collection-in-spark http://xinhstechblog.blogspot.jp/2015/06/reading-json-data-in-spark-dataframes.html

Posted in Uncategorized | Leave a comment

Spark insert / append a record to RDD / DataFrame ( S3 )

In many circumstances, one might want to add data to Spark; e.g. when receiving/processing records via Spark Streaming.  Spark is changing rather quickly; and so are the ways to accomplish the above task (probably things will change again once 1.6 … Continue reading

Posted in Uncategorized | 2 Comments