Using s3cmd to access S3 buckets from EC2 instances with IAM Role authentication
s3cmd is an open-source command line tool for uploading, retrieving and managing data in Amazon S3 and other providers that implement the S3 protocol (e.g. Google Cloud Storage, DreamHost DreamObjects, etc). It is popular tool with a variety of applications, including backup scripts.
This post covers using s3cmd within an EC2 instance, with authentication to S3 managed via IAM Roles (IAM = Identity and Access Management).
If your project doesn’t have dependencies on scripts and/or pre-existing code that require s3cmd, and you don’t mind locking your project deeper into AWS’ ecosystem, a good alternative is Amazon’s own AWS CLI.
Configuring an IAM Role
When an EC2 instance is launched with an associated IAM role, access keys and secret keys do not need to be stored in config files on the instance itself.
After such an instance is launched, AWS-aware services and scripts on the it can access AWS resources subject to the permissions defined in the IAM role’s associated IAM role policy. Authentication is seamlessly handled via the AWS SDK.
s3cmd versions 1.5.0-alpha2 and above support IAM roles and authentication via the AWS SDK.
IAM roles
Log in to your AWS account and access the AWS IAM Dashboard (also referred to as the IAM Management Console) to manage role-based permissions.
Create an IAM role for your instances if you haven’t already.
IAM role policy
In the IAM Management Console, ensure that your IAM role has an attached policy that provides access to any S3 resource(s) that you want your instance(s) to access via s3cmd. It’s advisable to ensure that role policies are restricted to the minimum amount of scope possible.
Within the IAM Management Console:
- Choose Roles and select the role that your EC2 instances belong to (or will belong to)
- Review your existing policies to confirm what your instances can currently access
- If necessary, use the Create Role Policy button to add a policy with the Policy Generator
For example, suppose you had a an s3 bucket called example-log-bucket and you wanted to configure your EC2 instances to send access logs to it.
Following the order of the Policy Generator’s input fields, you’d create: an Allow policy for the Amazon S3 service regarding the actions s3:ListBucket, s3:PutObject, and s3:PutObjectAcl, and apply them to the ARN arn:aws:s3:::example-log-bucket.
To learn more about how to specify ARNs that identify your various AWS services, see the Amazon docs: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3
Note that you can use wildcards in ARNs when defining a Role Policy.
Installing and configuring s3cmd on the EC2 instance(s)
s3cmd versions 1.5.0-alpha2 and above support IAM roles.
In earlier versions, an access key and secret key had to be specified in a .s3cfg file in the home directory of the user running the s3cmd command, or in environment variables.
Option A: Install s3cmd with the Ubuntu/Debian apt package manager
If you’re running Ubuntu 16.04 or later (test box at the time of writing) the package version is sufficiently up-to-date (>1.5.0). Install s3cmd with apt-get:
sudo apt-get install s3cmd
Option B: Install s3cmd with the python pip package manager
Pip is an alternative to Debian/Ubuntu apt-get, especially if you are using a different distro or like to use python/pip.
sudo pip install s3cmd
Check the version number with s3cmd --version. If you discover that its <1.5.0, you can specify a particular version for pip to install as follows, substituting in your desired version number (e.g. 1.5.0-alpha3) in place of VERSION_NUMBER below:
pip install s3cmd==VERSION_NUMBER
Configure s3cmd on the EC2 instance(s)
Once an EC2 instance is launched with a suitable IAM Role and associated IAM Role Policy, its easy to get started with s3cmd. The following example is set in the context of the currently logged in user.
Open/create the file ~/.s3cfg using a text editor:
nano ~/.s3cfg
Populate the file with the following content. Rather than filling in hardcoded credentials, leave each of the fields blank (use the example as is; do not specify any values):
[default]
access_key =
secret_key =
security_token =
Save the file and exit the editor.
Later versions of s3cmd that support IAM Roles will auto-magically detect the appropriate authentication values.
s3cmd will now be able to interact with any AWS resource(s) that it is permitted to access, subject to s3cmd’s capabilities and the IAM Role’s active Role Policies.
Use s3cmd
s3cmd has over 60 command line options. Check out the documentation: