Monthly Archives: October 2015

Spark: Column label must be of type DoubleType but was actually StringType

[error] Exception in thread “main” java.lang.IllegalArgumentException: requirement failed: Column label must be of type DoubleType but was actually StringType. Should use StringIndexer http://spark.apache.org/docs/latest/ml-features.html#stringindexer   keywords: spark ml mllib dataframe df apache spark double int to string label output

Posted in Uncategorized | Leave a comment

EC2 Spark: Permission denied (publickey)

I had an error (Permission denied (publickey)) while trying to launch an EC2 Spark cluster with spark-ec2.  Turns out; you need to have a key pair for each zone (in my case I was using the keypair from a different … Continue reading

Posted in Uncategorized | 2 Comments

Spark AWS EC2: AWS was not able to validate the provided access credentials: You are not authorized to perform this operation

For some reason when executing spark-ec2; I kept getting the exception below. I did do the “chmod 600 my.pem” on the permission files; but that didn’t fix the error. Workaround was to execute spark-ec2 with sudo This gets one step further … Continue reading

Posted in Uncategorized | 1 Comment

AWS Lambda Java: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token

For more details see: https://github.com/neil-rubens/aws-lambda-scala-api-gateway With a new support for Java; Lambda has become a very good candidate for implementing many of the microservices (and providing a RestFul interface for them via API Gateway).  However the documentation on how to connect … Continue reading

Posted in Uncategorized | Leave a comment

AWS Lambda API Gateway: authentication error

Problem Connecting Amazon’s API Gateway with Lambda function kept resulting in failure; with error requesting API Key, access token, etc. Solution There seems to be a bad integration between Lambda’s “add API Endpoint” and API Gateway.  Instead I went to … Continue reading

Posted in Uncategorized | 2 Comments

Spark Streaming : NoSuchMethodError

When running a simple spark streaming app; I was getting the NoSuchMethodError  exception (shown below). Initially, I thought it was caused by converting RDD to DataFrame (needed by ML pipeline); but it wasn’t.  In the end it turned out that … Continue reading

Posted in Uncategorized | Leave a comment