Spark AWS EC2: AWS was not able to validate the provided access credentials: You are not authorized to perform this operation

For some reason when executing spark-ec2; I kept getting the exception below.

I did do the “chmod 600 my.pem” on the permission files; but that didn’t fix the error.

Workaround was to execute spark-ec2 with sudo

This gets one step further but then another Error (You are not authorized to perform this operation) occurs.  After some debugging it seems that “AWS_ACCESS_KEY_ID” and “AWS_SECRET_ACCESS_KEY” were not properly propagated within the spark-ec2.py (probably due to the way boto gets the credentials).

I’ve modified the script (to specify AWS profile for credentials; which is a best practice anyways) and asked for it to be merged in: https://github.com/apache/spark/pull/9018

 

Keywords: EC2 IAM spark ec2 scripts

Exception

<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID></RequestID></Response>
Traceback (most recent call last):
 File "/opt/spark/ec2/spark_ec2.py", line 1513, in <module>
 main()
 File "/opt/spark/ec2/spark_ec2.py", line 1505, in main
 real_main()
 File "/opt/spark/ec2/spark_ec2.py", line 1334, in real_main
 (master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
 File "/opt/spark/ec2/spark_ec2.py", line 486, in launch_cluster
 master_group = get_or_make_group(conn, cluster_name + "-master", opts.vpc_id)
 File "/opt/spark/ec2/spark_ec2.py", line 347, in get_or_make_group
 groups = conn.get_all_security_groups()
 File "/opt/spark-1.5.0-bin-hadoop2.6/ec2/lib/boto-2.34.0/boto/ec2/connection.py", line 2969, in get_all_security_groups
 [('item', SecurityGroup)], verb='POST')
 File "/opt/spark-1.5.0-bin-hadoop2.6/ec2/lib/boto-2.34.0/boto/connection.py", line 1182, in get_list
 raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID></RequestID></Response>
<Response><Errors><Error><Code>UnauthorizedOperation</Code><Message>You are not authorized to perform this operation.</Message></Error></Errors><RequestID></RequestID></Response>
Traceback (most recent call last):
 File "/opt/spark/ec2/spark_ec2.py", line 1520, in <module>
 main()
 File "/opt/spark/ec2/spark_ec2.py", line 1512, in main
 real_main()
 File "/opt/spark/ec2/spark_ec2.py", line 1341, in real_main

 

 

 

About Neil Rubens

see http://ActiveIntelligence.org
This entry was posted in Uncategorized. Bookmark the permalink.

One Response to Spark AWS EC2: AWS was not able to validate the provided access credentials: You are not authorized to perform this operation

  1. Kishlaya kumar says:

    Try to sync your local machine system time with current time
    on Linux you can use command
    sudo ntpdate us.pool.ntp.org

Leave a Reply

Your email address will not be published. Required fields are marked *


*