Category: System Administrator

Editing WordPress’ wp-config.php with wp-cli and adding variables with sed

The WordPress CLI (command line interface) is a huge step for enabling developers and devops/sysadmin folks manage their WordPress installations. It’s awesome for writing scripts to automate key tasks.

This post covers editing the WordPress configuration file wp-config.php with the WP-CLI’s wp config command, as well as using the sed command to address a key missing feature of WP-CLI: the ability to add new config variables.

There are plenty of reasons you might want to edit wp-config.php via a script or directly via the command line. For example, developers might appreciate a bash script that sets WP_DEBUG to true, and a devops person might want to create an automated deploy process that ensures key SMTP settings are in place.

The rest of this post will assume your WP-CLI command can be invoked with wp. Depending on how you installed it, the command might be available to you as wp-cli or wp-cli.phar. If you are running wp-cli’s phar file directly, substitute php wp-cli.phar in place of wp in the examples.

Editing config variables with wp-cli’s config command

WP-CLI supports modifying config variables in wp-config.php via wp config.

This is a great feature, albeit with the noted catch that wp config only works for a given variable if that variable is already defined in wp-config.php. I’ll show you how to work around that and add variables with sed in the next section.

The following example uses sudo to run wp config as the _www web server user, the default web server user on MacOS. On Ubuntu and many other linux distros, this user is likely www-data:

sudo -u _www wp config set FS_METHOD 'direct'
sudo -u _www wp config set DISABLE_WP_CRON true
sudo -u _www wp config set WP_DEBUG true
sudo -u _www wp config set WP_DEBUG_LOG true

These are some of the most popular config options that developers and admins want to modify.

Adding config variables to wp-config.php using sed

There are a number of command-line utilities on Linux and Unix-like systems that can edit text files. One of the most popular is sed, the quintessential stream editor. Unix admins have been working with streams forever, long before NodeJS made it cool :).

sed is pre-installed on most systems and can be used directly in the Terminal or inside a bash (or other shell) script.

The following example uses sed to add config variables to wp-config.php right before the well-known “That’s all, stop editing!” comment line found in the file.

This snippet works on MacOS and elsewhere. MacOS and OSX, as well as their related family in the BSD/unix world, generally bundle a classic POSIX-compliant version of the sed command which is more limited vs. the more common and more popular GNU sed that ships with major linux distributions like Ubuntu. If you’re on linux, delete the double quotes '' immediately following the -i flag to be compatible with GNU sed.

Editing wp-config.php with sed:

sed -i '' '/\/\* That.s all, stop editing! Happy blogging. \*\// i\
// FX_SCRIPT FS_METHOD \
define( "FS_METHOD", "direct" ); \
\
// FX_SCRIPT WP_DEBUG \
define( "WP_DEBUG", true ); \
define( "WP_DEBUG_LOG", true ); \
\
// FX_SCRIPT DISABLE_WP_CRON \
define( "DISABLE_WP_CRON", true ); \
\
' wp-config.php

The -i option tells sed to edit the file in-place i.e. modify the file directly. Otherwise, sed lives up to its name and streams output to stdout.

The MacOS version of sed requires a backup file to be specified as the first argument whenever the -i option is used. You can pass empty quotes '' to specify no backup file as demonstrated in the example.

The linux version of sed does not require a backup filename to be specified. You can simply delete the '' arguments as noted above.

The way that single and double quotes are used is very important for getting this command to work. Getting them right is one of the trickiest parts about using sed.

Also note how backslashes are used at the end of each line. This is required to make the command portable and universal: classic sed (BSD/Unix/MacOS) does not recognize \n as a placeholder for the newline character while GNU sed (linux) does. The backslashes enable a trick to use actual newlines instead of placeholders.

Finally, my example adds comment lines that start with // FX_SCRIPT before each change (get it? FX = firxworx, the name of this blog!). I do this to make it easy to search with grep and/or visually look for changes in wp-config.php files that were made by my scripts. You may wish to follow a similar practice. This makes it easier to write other scripts that might find and comment out or delete these entries at a later time.

Set up MacOS’ built-in Apache + PHP as a LAMP/WordPress Dev Environment

This guide covers configuring MacOS High Sierra, Mojave, and beyond as a local development server for LAMP-stack projects. This is a high-performance option for running projects locally on a Mac and is useful for PHP and WordPress development.

At the end of this guide, you will be able to create new folders within a special virtual hosts directory such as ‘example.com/’ with the sub-folder ‘public_html/’ and the ‘http://example.com.test’ will automatically be available with apache serving files out of the ‘public_html/’ folder.

Apple already bundles many of the necessary pieces with MacOS including apache and PHP. This guide covers configuring them to support WordPress, as well as installing other dependencies such as mysql/mariadb, and useful tools like sequel-pro and wp-cli.

Alternatives

Self-contained solutions for running a LAMP environment on a Mac can be found in products like MAMP and the free XAMPP. They double up what’s already on your Mac but do have some features that you might find appealing.

Virtual Machines are another option. Tools such as Vagrant are great and extremely useful for more complex projects where customized server configuration(s) are required. However, spinning up an entire virtual server with its own CPU, RAM and disk allocation simply to run WordPress is a huge tax on system resources. It’s often a better idea to reserve VM tools for better-suited applications.

A lighter-weight alternative to full VM’s is container virtualization such as with Docker. This brings its own configuration and deployment challenges that may introduce unnecessary effort if Docker doesn’t play a greater role in a given project.

Pre-checks

Confirm your system’s apache version with the command apachectl -v, and its PHP version with the command php -v.

MacOS Sierra and beyond should be running apache 2.4+ and PHP 7+.

Apache configuration

Apple’s default configuration requires several changes to create a suitable development environment.

Backup httpd.conf

First, make a backup copy of apache’s httpd.conf file:

sudo cp /etc/apache2/httpd.conf /etc/apache2/httpd.conf.pre-dev-env

Apache httpd.conf configuration

Open httpd.conf using a text editor such as nano:

sudo nano /etc/apache2/httpd.conf

Each of the following lines are found at different locations within the file. Find each one of them and ensure they are uncommented — delete any preceding # character so they start with a letter instead:

#LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so
#LoadModule rewrite_module libexec/apache2/mod_rewrite.so
#LoadModule php7_module libexec/apache2/libphp7.so
#Include /private/etc/apache2/extra/httpd-vhosts.conf

If you’re using the nano editor, its “where” Control+W feature can help you locate each line. It works similar to the “find” command (Control+F) found in most web browsers and word processors.

When you’re done, use Control+O to save (output) your changes and Control+X to exit nano.

The above changes have:

  • enabled the module mod_vhost_alias to enable dynamic virtual hosts;
  • enabled the module mod_rewrite which WordPress uses to rewrite URL paths;
  • enabled PHP7 support so WordPress’ php scripts can execute; and
  • included an extra conf file where we will define a dynamic apache virtualhost for your projects.

Apache dynamic virtual host configuration

In the previous step we referenced a httpd-vhosts.conf file in httpd.conf.

Create a backup of the original file:

sudo cp /private/etc/apache2/extra/httpd-vhosts.conf /private/etc/apache2/extra/httpd-vhosts.conf.pre-dev-env

Now edit that file:

sudo nano /private/etc/apache2/extra/httpd-vhosts.conf

This file is populated by default with a couple of example virtual host definitions: blocks beginning with <VirtualHost *:80> and ending with </VirtualHost>.

Comment out (i.e. prefix each line with #) or delete the two example virtual host definitions.

Next, add your own dynamic virtual host definition to the file.

I’ve decided to use /usr/local/var/www as the base folder to serve projects from. You can modify this to another path if you’d like.

Add the following block:

# Custom configuration for local dev environment

<Directory "/usr/local/var/www">
  Options Indexes MultiViews FollowSymLinks
  AllowOverride All
  Require all granted
</Directory>

<Virtualhost _default_:80>

  # Dynamic virtual hosts for local development
  UseCanonicalName Off
  VirtualDocumentRoot "/usr/local/var/www/%-2+/public_html"

  # Show 404's in the apache error log
  LogLevel info

</Virtualhost>

Save (Control+O) and exit (Control+X) the nano editor.

Explaining the Dynamic Virtual Host definition

In httpd-vhosts.conf as modified above, the <Directory> block is required for Apache 2.4+ and tells Apache that it can serve files from the /usr/local/var/www folder. This explicitly overrides the rigorous security-focused default specified in httpd.conf that tells Apache it can’t serve up any file on the hard disk (/).

The <VirtualHost> block defines a _default_ VirtualHost that listens on port 80.

The VirtualDocumentRoot directive tells Apache where to look for the dynamic VirtualHost’s files. The %-2+ placeholder stands in for “the penultimate (second to last) and all preceding parts” of the server name in the request. Therefore, per the directive, when handling a request for http://example.com.test, Apache will look for that site’s files in /usr/local/var/www/example.com/public_html.

I like to have my project folder names in dev environments mirror my likely production values, such as example.com/ or hypothetical.app/. You may wish to name your project folders differently. You may prefer to use the %0 placeholder for the entire server name (example.com.test), or %1 for the first-part only (example).

To learn more about these directives, refer to the docs:

Create the required paths

Finally, ensure the paths that you specified for your project folders exist. In my case:

mkdir -pv /usr/local/var/www

If you have any project build folders, you can add them (or better yet: create symlinks to them) here!

To give you an idea of how this works:

In /usr/local/var/www if you were to create the folder helloworld.com/ and sub-folder helloworld.com/public_html, and put a basic index.php containing hello world! inside it, at the end of this tutorial Apache will serve it up when you visit http://helloworld.com.test on your machine.

Smoke-check (config check)

As a smoke-check to make sure you haven’t made any typos or introduced any bugs, run the following command to check the syntax of your apache config files:

sudo apachectl configtest

If you followed all of the above steps carefully, it should output “Syntax OK”.

If everything checks out, restart apache so your latest conf changes can take effect:

sudo apachectl restart

Configuring DNS for local domain names

Now we will solve the problem of directing local requests for http://example.com.test to Apache.

One option is to manually add an entry to /private/etc/hosts for every one of your projects. For example, you might add the line 127.0.0.1 myproject.test to enable the local URL ‘http://myproject.test’.

We are going to use a more dynamic approach using a DNS server so that all requests for any URL at the *.test domain will resolve to localhost.

Any DNS server could be employed for this job. We will use dnsmasq, a perfect choice for this type of lightweight local routing.

Follow the steps in my guide Using dnsmasq on MacOS to setup a local domain for development to complete this step.

Customizing PHP settings (php.ini)

MacOS’ PHP uses a default php.ini file based on /private/etc/php.ini.default.

To customize your PHP environment, if a php.ini file doesn’t already exist at /private/etc/php.ini, copy the default template to create a main php.ini file:

sudo cp /private/etc/php.ini.default /private/etc/php.ini

Make any changes you wish to php.ini and restart apache to reload all configuration files:

sudo apachectl restart

If you were to run phpinfo() in a php file from the web server, you should now see that the Loaded Configuration File property now has the value /etc/php.ini.

A very common tweak to the default PHP configuration is to allow larger file upload sizes. The post_max_size and upload_max_filesize properties are only a few megs by default. These limits can be raised as you see fit.

Many developers also tweak the max_execution_time, max_input_time, and memory_limit settings depending on their project.

Always remember to restart apache after making changes to your PHP configuration.

Installing and configuring mysql server

WordPress and many other PHP apps require a MySQL or compatible database such as MariaDB to store its data.

I prefer to use mariadb, which is available via brew. Run the following command to install it for your user:

brew install mariadb

This installation method is recommended in the official docs:

Following installation, manually start the server by running:

mysql.server start

Running mysql as a service

By default you will need to manually start and stop your mysql/mariadb server.

If you want the server to start when you login to your Mac, use Homebrew’s services feature to have it automatically launch in the background:

brew services start mariadb

Homebrew integrates with MacOS’ launchctl to make this happen the correct way as supported by Apple.

Test logging into mysql via the cli client

Once your server has started, you can test logging in as the root user with the Terminal command:

mysql -u root

Issue the quit command to exit the MySQL prompt and return to your Terminal session.

Set a root password and improve security

Even though a default mysql/mariadb configuration only accepts connections from localhost, it is still wise to run the quick interactive mysql_secure_installation command-line-interface script to set a root password and a few other security-minded options.

Troubleshooting: resolve possible socket incompatibility

There may be a slight configuration mismatch between the defaults of brew’s mysql/mariadb packages and the defaults of MacOS’ built-in PHP mysqli extension. It may be fixed in newer packages.

If try to get a LAMP app such as WordPress to connect to mysql on localhost and encounter a “Connection Refused” error even when the database settings are confirmed correct (as defined in wp-config.php for WordPress), then you may need to make a small fix. You can confirm the issue by trying to connect to 127.0.0.1 instead: if it works but localhost doesn’t then it is a socket-related issue. 

For localhost (socket) connections to the database, brew’s package version tends to use the socket file /tmp/mysql.sock. MacOS’ built-in Apache+PHP mysqli extension assumes the socket file is at /var/mysql/mysql.sock (i.e. the default setting of its mysqli.default_socket property). You can verify the value of this property by running phpinfo(); in a php script and finding it in the list.

To resolve, one could edit php.ini or another conf file. A simple solution is to create a symlink at /private/var/mysql/mysql.sock that points to /tmp/mysql.sock so everything works out:

sudo mkdir -p /var/mysql
sudo ln -s /tmp/mysql.sock /private/var/mysql/mysql.sock

On MacOS /var, a classic folder for unix/linux systems, is symlinked to /private/var so to avoid a symlink-on-symlink type situation, the above uses the real path. 

Note that only socket connections (i.e. database connections made to ‘localhost’) would be impacted by this incompatibility. Connections to ‘127.0.0.1’ force connecting over TCP due to the IP address, bypassing any socket concerns.

Install a database management GUI

If you’d like to use a GUI to help manage your mysql databases, one popular choice is sequel-pro. This serves the same purpose as the popular phpmyadmin webapp except it runs as a desktop app on your Mac. To install it:

brew cask install sequel-pro

You can now launch it from your Applications folder.

Install wp-cli

WordPress’ Command-Line-Tools are a huge timesaver for managing both local and remote WordPress installs, and they are essential for writing scripts that automate WordPress deployment, updates, etc.

Install wp-cli for your user with the command:

brew install wp-cli

Confirm your install by running wp --version in Terminal.

Assuming your project folder includes a public_html/ folder that contains your WordPress install, and that you have created a database called “DB_NAME”, you can download and setup WordPress with a handful of wp-cli commands:

cd /Users/username/web-projects/my-project.com/public_html
wp core download
wp config create --dbname=DB_NAME --dbuser=DB_USERNAME --dbpass=DB_PASSWORD --dbhost=localhost
wp core install --url=my-project.com --title="My Project" --admin_name="example_admin" --admin_password="example_password" --admin_email=you@example.com

Check out the command reference and docs/handbook at https://wp-cli.org/

Serving up your project

To make a project available at http://example.com.test you need to:

  • create a folder example.com/ in your chosen base directory (e.g. /usr/local/var/www)
  • create the public_html subfolder: example.com/public_html/
  • copy project files to public_html/ (manually or as an automated task e.g. with gulp)

OR

  • create a symlink from your project location to the chosen base directory

For the symlink option:

ln -s ~/some/project/folder/example.com /usr/local/var/www/example.com

Be sure to replicate the requirement for a public_html/ sub-folder.

When creating symlinks from elsewhere on your system to /usr/local/var/www be careful of permissions issues!

Managing permissions

MacOS’ Apache is running as the user _www by default. That user must have at least read permissions for any folder that you symlink to in order to serve up your project files.

It’s important to note that ~/Documents and any folders created within are not accessible to any group or other users. Apple thankfully assumes that you don’t want other users on a system to be able to access your private documents! As a result, Apache’s _www user will not be able to follow symlinks into the Documents folder belonging to your user.

If you have any web projects in ~/Documents you could save them somewhere else where Apache can read them and symlink to them from there, or you could simply copy your project files over to /usr/local/var/www.

Depending on your desired setup, you may find it convenient to change the group ownership of /usr/local/var/www to _www and set the groupID bit (setgid):

sudo chgrp _www /usr/local/var/www
sudo chmod g+s /usr/local/var/www

The setgid bit ensures that any new files or folders created underneath www/ will inherit the group ID of the directory vs. the default behaviour of having it set to the primary group ID of the user that created the file. This measure can help ensure that Apache’s _www user can access the contents of the www/ folder.

Finally, remember that our particular Apache VirtualHost configuration requires a public_html/ sub-directory under the my-project.com folder for web-servable files. To illustrate: an index.php file in ~/web-projects/my-project.com/public_html will be served up to a local web browser that requests the URL http://my-project.com.test. This works because there is a symlink in /usr/local/var/www named my-project.com that points to the folder ~/web-projects/my-project.com.

Working on a project

In the future, if you haven’t setup apache and mysql server to run automatically as services, you will need to start them:

sudo apachectl start
mysql.server start

To stop or restart apache, use the following commands:

sudo apachectl stop
sudo apachectl restart

You can stop mysql with:

mysql.server stop

Remember: if you make any changes to your apache config, such as adding a new VirtualHost, you will need to restart or reload apache for your changes to take effect.

Finally remember that the settings covered in this guide will serve up your projects to any clients on your network. If you want to hide them from everyone on your local coffee shop’s shared wifi, turn off the servers, harden your firewall (via System Preferences), and/or tweak your Apache conf to only serve to localhost.

Upgrading MacOS

When you upgrade MacOS, the installer will create versions of your conf files with the suffix ~previous before replacing with its own.

You can also create your own backup copies to be safe.

Following an OS update, if the Apache files are changed, you can compare the new versions with your ~previous versions using the diff command to make sure there’s no major updates (very unlikely) then restore your customized versions. After restoring your files, remember to restart apache for the settings to take effect: sudo apachectl restart.

The files customized by following this guide are:

  • /etc/apache2/httpd.conf
  • /private/etc/apache2/extra/httpd-vhosts.conf

Script to Deploy WordPress

Check out this gist for a bash script to quickly deploy a fresh WordPress install:

https://gist.github.com/firxworx/bb9b7f8d71915f9d4e7f8d3a8a531b26

Using dnsmasq on MacOS to setup a local domain for development

This guide covers using dnsmasq as a local DNS server on MacOS to resolve all URL’s with the .test domain name to localhost (127.0.0.1).

This can be very useful to developers wishing to setup an efficient local development and test environment on their machine. For example, if http://myproject.com.test resolved to localhost, a local apache, nginx, or other server could respond to the request.

While any DNS server could be employed for this job, dnsmasq is a perfect choice for this type of lightweight local routing.

Regarding the .test domain, note that this is a recommended choice per the IETF RFC’s (e.g. RFC-2606, RFC-6761). While it’s possible to configure dnsmasq for any arbitrary domain name, only .test, .example, .invalid, and .localhost are reserved for non-Internet use. A popular choice used to be .dev until Google purchased the top-level-domain (TLD) and changed how Chrome handles .dev URL’s.

Install and configure dnsmasq

To get started, use brew to install dnsmasq for your user:

brew install dnsmasq

At the time of writing, the brew package installer creates a sample conf file with all entries commented out. It can be found at ./etc/dnsmasq.conf under brew’s default install folder.

Execute following commands in sequence to configure dnsmasq to resolve *.test requests to 127.0.0.1:

echo "" >> $(brew --prefix)/etc/dnsmasq.conf
echo "# Local development server (custom setting)" >> $(brew --prefix)/etc/dnsmasq.conf
echo 'address=/.test/127.0.0.1' >> $(brew --prefix)/etc/dnsmasq.conf
echo 'port=53' >> $(brew --prefix)/etc/dnsmasq.conf

The above commands simply append the required lines to the dnsmasq.conf configuration file.

Next, use brew’s services feature to start dnsmasq as a service. Use sudo to ensure that it is started when your Mac boots, otherwise it will only start after you login:

sudo brew services start dnsmasq

Add a resolver

Add a resolver for .test TLD’s:

sudo mkdir -pv /etc/resolver
sudo bash -c 'echo "nameserver 127.0.0.1" > /etc/resolver/test'

This tells your system to use localhost (and therefore dnsmasq) as the DNS resolver for the .test domain.

Test your configuration

Flush your DNS cache:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

Check your configuration by running:

scutil --dns

Inspect the output. One of the entries should specify that for domain test the system will use the nameserver 127.0.0.1. This means that dnsmasq will now be consulted as the nameserver for any .test URLs.

Finally, test the full stack by trying to ping random *.test URL’s such as example.test or myproject.test and confirming that they resolve to 127.0.0.1. For example:

ping -c2 example.test

Provides the output:

PING example.test (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.022 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.055 ms

OR (in cases where your Mac’s firewall settings are configured to not respond to ping):

PING example.test (127.0.0.1): 56 data bytes
Request timeout for icmp_seq 0
...

The important part to confirm is that example.test resolved to (127.0.0.1).

Include the client IP in apache logs for servers behind a load balancer

When servers are behind a load balancer, Apache’s default configuration produces logs that show the load balancer’s IP address instead of the IP of the remote client that initiated the request. Furthermore, if multiple server’s logs are consolidated into one, it can be difficult to determine which server created a given log entry.

This post covers how to improve on Apache’s default logging situation such that:

  • every web server behind the load balancer includes a unique identifier for itself in its log entries, and that;
  • access log entries include the client’s remote IP address as found in the X-Forwarded-For header set by the load balancer.

This setup is more helpful for troubleshooting, configuring monitoring and alerts, etc. and it helps to maximize the value of 3rd-party log aggregation and analysis services like Papertrail or Loggly).

The example commands in this post are applicable to Ubuntu/Debian however they are easily adapted to other environments.

Enable required modules

Start by ensuring that the required Apache modules: env and remoteip, are enabled:

a2enmod env
a2enmod remoteip 
service apache2 restart

Identify each web server

Add a SetEnv directive in the site/app’s apache conf to instruct the env module to set a new environment variable with a value that uniquely identifies each server behind the load balancer.

The example below is a snippet from an Apache VirtualHost’s conf file. It defines a variable called APP_LB_WORKER with the value ‘unique_identifier’. If you were using a devops automation tool such as Ansible, you could use the template module and use a handy variable such as {{ ansible_host }} in place of the example’s hard-coded ‘unique_identifier’ value.

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
</VirtualHost>

Configure the Apache RemoteIP module

Create a file named /etc/apache2/conf-available/remoteip.conf and use it to set: the RemoteIPHeader to ‘X-Forwarded-For’, any appropriate RemoteIPInternalProxy or other RemoteIP directives, and to define a new LogFormat that includes both the environment variable that contains the server’s identifier and the client’s remote IP as sourced from the X-Forwarded-For header.

The RemoteIPInternalProxy directive tells the RemoteIP module which IP address(es) or IP address blocks it can trust to provide a valid RemoteIPHeader that contains the client’s IP.

The following example’s RemoteIPInternalProxy value is representative of an environment where the load balancer’s internal network IP address belongs to a public subnet with the CIDR block 10.0.1.0/24. Choose an appropriate value (or values) for your environment.

A full list of configuration directives for RemoteIP can be found in the Apache docs: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html.

The following example names the new LogFormat as “loadbalance_combined”. You can choose any name you like that isn’t already in use.

/etc/apache2/conf-available/remoteip.conf:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.1.0/24

LogFormat "%a %v %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{APP_LB_WORKER}e\"" loadbalance_combined

Finally, enable the conf:

a2enconf /etc/apache2/conf-available/remoteip.conf

The Apache LogFormat is extensively customizable. Learn more about the placeholders and options from here: https://httpd.apache.org/docs/2.4/logs.html. Remote log aggregation service Loggly also has an excellent overview: https://www.loggly.com/ultimate-guide/apache-logging-basics/.

Tell Apache to use the new LogFormat

Next, tell Apache to use the custom LogFormat that we named loadbalance_combined by editing your site/app’s conf file. The following example builds upon the previous example of a VirtualHost conf:

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
    CustomLog /var/log/app_name/access.log loadbalance_combined
...
</VirtualHost>

The following example is a more elaborate case that uses the tee command to send the log entry to both an access.log file and to the /usr/bin/logger command to include it in the syslog, only if an environment variable named “dontlog” is not set. An example use-case for an env variable like “dontlog” is to set it (e.g. via the SetEnvIf directive) for any requests that correspond to a “health check” from the load balancer to the web server. This helps keep logs clean and clutter-free.

CustomLog "|$/usr/bin/tee -a /var/log/app_name/access.log | /usr/bin/logger -t apache2 -p local6.notice" loadbalance_combined env=!dontlog

Restart Apache

Confirm the validity of your confs using apachectl configtest and restart apache for the new configuration to take effect:

service apache2 restart

Creating certificates and keys for OpenVPN server with EasyRSA on MacOS

This guide covers how to create certificates and keys for OpenVPN server and clients using the EasyRSA tool on MacOS.

The instructions are very similar for most flavours of linux such as Ubuntu once the correct packages are installed (e.g. on Ubuntu: apt-get install openvpn easy-rsa).

If privacy and security are of the utmost concern, generate all certificates and keys on a “clean” machine and verify the signatures of each download.

Step 1: Resolve MacOS Dependencies

This guide assumes that you’re running MacOS Sierra or later.

XCode and Command Line Tools

Ensure that you have installed the XCode Command Line Tools.

To check, the command xcode-select -p outputs a file path beginning with /Applications/Xcode.app/ if they are already installed.

If Command Line Tools is not installed, open the Terminal app and enter xcode-select --install to trigger the installation app.

Another way to trigger the installation app is to attempt to use a command line developer tool such as the GNU C compiler gcc (e.g. gcc --version). If the tools are not installed, you will be greeted by a graphical MacOS installation prompt instead of the expected Terminal output from gcc. You don’t necessarily need the full XCode so you can click the “install” button for just the command line tools.

Work your way through the installer and follow Apple’s steps until you can start working with the necessary commands. The CLI commands will become available to you after you agree to all of Apple’s terms and conditions.

If you experience troubles with the next step, assuming that it is the result of some future change by Apple, it may be beneficial to install the full XCode in addition to the CLI tools. It’s available for free on the App Store, but take note that it’s a hefty multi-gigabyte download.

OpenSSL

EasyRSA requires a late-version of the open-source openssl library.

Apple bundles its own crypto libraries in MacOS but these are generally out of date. At the time of writing, the openssl command bundled with MacOS is not likely compatible with EasyRSA and will produce errors if you try to use it (note: the binary is at /usr/bin/openssl).

A newer EasyRSA-compatible version of OpenSSL is easy to install with the brew package manager (https://brew.sh/). Installing via brew will not clobber or harm the Apple version that’s already on your system. If you need to install brew, go to the project’s website and follow the simple instructions on the landing page.

Assuming you have brew installed, open a Terminal and run the command:

brew install openssl

Brew will download and install openssl to its default package install directory of /usr/local/Cellar.

The package will be installed in “keg only” mode: brew will not create a symlink for its openssl in /usr/local/bin or anywhere else in your $PATH. You will not have a conflicting openssl command, and Apple’s binary will remain intact.

To get EasyRSA to use the openssl binary installed by the brew package, you will need to know its path. Run brew’s package info command and examine the output:

brew info openssl

In my example, I could see that openssl resolved to /usr/local/Cellar/openssl/1.0.2n. In your case, this may be a different path due to a more recent version being available in the future. Next, inspect this folder to locate the binary and determine the full path to it. In my example case, the full path to the binary was:

/usr/local/Cellar/openssl/1.0.2n/bin/openssl

Note down the correct path to the openssl binary for your case. When configuring EasyRSA in the next step, you will need to specify this path in an EASYRSA_OPENSSL variable.

Step 2: Download EasyRSA

Go to https://github.com/OpenVPN/easy-rsa/releases and download the latest .tgz version for your Mac.

Save the file to a folder that you wish to work from (your certificates and keys will be generated here) and unpack it using the Archive utility (double click on it in Finder).

Note that the easy-rsa tools were written with traditional linux/unix-type environments in mind and therefore assume that all paths to the scripts have no spaces in them.

Going forward I will assume the path of your unpacked EasyRSA folder is: ~/vpn/easyrsa. The ‘~’ character is a shortcut to your home folder that works in Terminal, i.e. on a Mac its a placeholder for /Users/your_username and on a typical linux environment /home/username.

Step 3: Configure EasyRSA

Assuming that the path of your unpacked EasyRSA folder is: ~/vpn/easyrsa, open Terminal and navigate to the unpacked folder:

cd ~/vpn/easyrsa

Copy the vars.example “starter” configuration file to vars:

cp vars.example vars

Now customize the initial “starter” configuration file’s settings in vars to reflect your own.

Open it in a text editor and look for the following lines. Uncomment them (i.e. delete the preceding # character) and fill them in with your appropriate values. Specify something for each field below:

#set_var EASYRSA_REQ_COUNTRY   "US"
#set_var EASYRSA_REQ_PROVINCE  "California"
#set_var EASYRSA_REQ_CITY  "San Francisco"
#set_var EASYRSA_REQ_ORG   "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL "me@example.net"
#set_var EASYRSA_REQ_OU        "My Organizational Unit"

Look for the following field and uncomment it:

#set_var EASYRSA_KEY_SIZE        2048

We’ll be using a 2048-bit key (the current default) for this example so the value will not be changed.

A larger key size is more secure but will result in longer connection + wait times over the VPN. At the time of writing in late 2017, its generally believed that a 2048-bit key is sufficient for most usage scenarios. A 4096-bit key is believed to provide additional privacy vs. more powerful state-sponsored actors.

EasyRSA by default uses the openssl binary found in the $PATH. Find the following line, uncomment it, and update the value with the path to the brew-installed openssl binary from Step 1. For example, in my case, the following line:

#set_var EASYRSA_OPENSSL   "openssl"

became:

set_var EASYRSA_OPENSSL "/usr/local/Cellar/openssl/1.0.2n/bin/openssl"

Step 4: Generate Certificate Authority (CA)

Navigate into your easyrsa/ folder. For example:

cd ~/vpn/easyrsa

Initialize the PKI (public key infrastructure) with the easyrsa script. This will create a pki/ subfolder:

./easyrsa init-pki

Create the CA (certificate authority):

./easyrsa build-ca nopass

You will be prompted to input a Common Name. Input the name server and hit ENTER.

The generated CA certificate can now be found at pki/ca.crt.

Step 5: Generate Server Certificate + Key + DH Parameters

Assuming you’re still inside your easyrsa/ folder from the previous step, generate your server certificate and key:

./easyrsa build-server-full server nopass

The generated server certificate can now be found at: pki/issued/server.crt

The generated server key can now be found at: pki/private/server.key

Now generate the Diffie-Hellman (DH) parameters for key exchange. This process can take several minutes depending on your system:

./easyrsa gen-dh

The generated DH parameters can be found at: pki/dh.pem.

You now have all of the files necessary to configure an OpenVPN server.

Step 6: Generate client credentials

You should generate a unique set of credentials for each and every client that will connect to your VPN. You can repeat this step for any client that you need to create credentials for.

All clients in your setup should have a unique name. Change exampleclient in the following to something descriptive that you will recognize and be able to associate with the user/client:

./easyrsa build-client-full exampleclient nopass

The generated client certificate: pki/issued/exampleclient.crt

The generated client key can be found at: pki/private/exampleclient.key

When distributing credentials to a client, they will need at least these 3 files:

  • A client certificate (e.g. pki/issued/exampleclient.crt)
  • The corresponding client key (e.g. pki/private/exampleclient.key)
  • A copy of the CA certificate (pki/ca.crt)

These client credentials can be loaded into a VPN app like Tunnelblick or Viscosity along with client configuration information that corresponds to your VPN server’s specific settings.

Understanding client config files

Client configuration information is usually provided in the form of an additional file: a plaintext config file with the .ovpn extension. Both Tunnelblick and Viscosity recognize the .ovpn extension and file format.

Later versions of openvpn support specifying all of the client configuration information, client certificate, client key, and CA certificate as demarcated blocks within the config file itself, so that clients only need to be provided with a single .ovpn file.

Security reminder

It is good to practice to try and keep all .ovpn, certificate, and key files as safe as possible with exposure to as few eyes/hands/hard-disks/clouds/etc as possible. Distribute them as securely as you can to your clients/users.

Next steps

Now you need a working openvpn server and a client that wishes to connect to your VPN!

I hope this guide was helpful to you. For all the server tutorials out there, as far as I know this is one of the few comprehensive guides out there for creating all required certificates and keys on MacOS.

Router as OpenVPN server

If your openvpn server is your router, you can now login to it’s admin control panel and input the server-related certificate + key + DH parameters that you created above.

Before you activate the VPN server, ensure that your router’s firmware is up-to-date and that you have set a long and reasonably secure password for the admin user.

Running your own server

If you are planning to setup your own openvpn server, there are numerous other resources available online to guide you through the server installation and configuration process for a variety of different operating systems.

You will find that you need all the keys and certificates that you created by following this guide.

These resources will generally include guidance for crafting .ovpn client configuration files to include specific settings that correspond to your server’s particular setup, so that clients can successfully connect.

Send emails from MacOS Terminal or scripts using Rackspace’s SMTP server

This guide will help configure MacOS so it can send emails from the command-line (Terminal) and shell scripts via an SMTP server. This is useful for enabling scripts, scheduled jobs, etc. to send email notifications.

The example in this post work for Rackspace’s SMTP server. The guide is adaptable to other SMTP servers and email providers, while acknowledging that you may need to adjust certain details. Email providers can vary in how they require users to authenticate and interface with their SMTP servers.

There are lot of examples online for using gmail’s SMTP servers, but since I couldn’t find any complete examples for Rackspace, I decided to write this post!

Configuring postfix on MacOS

MacOS/OSX comes bundled with the postfix mail server. We will configure that to use Rackspace’s SMTP server to send mail.

Prerequisites

To use Rackspace’s SMTP, its a given that you must be a Rackspace customer. Create an email address that you wish to be the “sender” of any notifications. Note this email account’s credentials as you’ll need to include them in your postfix configuration.

I recommend that you use a dedicated email address for sending automated notifications (e.g. no-reply@.., notify@.., etc), rather than an email address with an inbox that is important to your business. Within Rackspace’s control panel, you have the option to configure an auto-responder for your automated notification address so that anyone that replies to a notification or sends anything else to it will receive a canned reply in response.

Step1: Edit main postfix configuration file

Make a copy of your original postfix conf file:

sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.bak

Open postfix’s config file for editing:

sudo nano /etc/postfix/main.cf

In an untouched postfix configuration on MacOS, the following 3x separate lines have the following values:

  • mydomain_fallback = localhost
  • mail_owner = _postfix
  • setgid_group = _postdrop.

If you have previously edited your postfix settings, you may wish to search for each item and reset these back to their default values (first confirm that you aren’t about to break anything!).

Add the following to the very end of the open postfix config file:

relayhost = smtp.emailsrvr.com:587
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =
smtp_use_tls=yes
smtp_tls_security_level=encrypt

These setting are specific to Rackspace’s requirements. If you are using another SMTP server, you will need to specify different values.

Save and exit the editor.

Step 2: create the sasl_passwd file for postfix to use

Create the /etc/postfix/sasl_passwd file referenced in main.cf with the following line of text, substituting the email@example.com and password placeholders with your own:

smtp.emailsrvr.com:587 email@example.com:password

Next, use the postmap tool to create a lookup table from the sasl_passwd file:

sudo postmap /etc/postfix/sasl_passwd

This will create the file sasl_passwd.db

Step 3: Restart postfix

sudo postfix reload

You might see some warnings, but they should be benign. We don’t need a full mail system, just the ability to send smtp emails. Move on to the next step to test your configuration!

Step 4: Send a test email

The following command sends a test email to your-email@example.com. Give it a shot by substituting your email address:

echo "testing testing test email" | mail -s test your-email@example.com

Note this command does not produce any output when it runs successfully.

Next, check for the test email in your inbox (if its not there, be sure to also check the spam folder). You may want to add your notification email address to your recipient inbox’s address book to ensure better deliverability.

Troubleshooting tips

If you need to troubleshoot, it can be helpful to check the mail queue:

sudo mailq

You can clear all queued emails with the following command:

sudo postsuper -d ALL

On OSX, postfix logged to the /var/log/mail.log file. On MacOS, this log file doesn’t exist.

For troubleshooting with MacOS, you can view the log in real-time with the command:

sudo log stream --predicate  '(process == "smtpd") || (process == "smtp")' --info

You could have this running in one Terminal tab, and then try to send a test email in another tab, to see what messages you receive.

Acknowledgements

Thanks to this article from <developerfiles.com> which has instructions for using gmail, as well as some commenters on various forums that provided tips to adapt SMTP settings for Rackspace. Rackspace’s documentation was also helpful in realizing a working configuration.

Stripping leading and trailing slashes from paths in Ansible

This post covers how to use ansible’s regex_replace filter to strip leading and/or trailing slashes from file paths and URL fragments.

The final example demonstrates how to generate a valid filename from a file or URL path by removing leading and trailing slashes, and replacing any remaining slashes with underscores. This could be useful for a variety of applications from backup scripts to web scraping.

Stripping slashes

Regexes and jinja2 expressions in ansible can be a pain in the ass, especially when it comes to escaping the right thing.

The key to the following examples is a double-escape of the forward slash character. They have been tested on ansible v.2.3.1.0.

Remove leading slashes

{{ variable_name | regex_replace('^\\/', '') }}

Remove trailing slashes

{{ variable_name | regex_replace('\\/$', '') }}

Stripping both leading and trailing slashes

This example makes use of the | (OR) to combine the previous two examples into one regex:

{{ variable_name | regex_replace('^\\/|\\/$', '')

Here’s a quick debug task that demonstrates the above in action:

- debug:
    msg: "Look Ma! No leading or trailing slashes! -- {{ item | regex_replace('^\\/|\\/$', '') }}"
  with_items:
    - "/home/and/lots/of/folders/trailing_and_leading/"
    - "no_leading/but/there/is/trailing/"
    - "/leading/but/no_trailing"
    - "no_leading/and/no_trailing"

Creating a valid filename from a path

To create a valid filename from a path, we need to remove leading and trailing slashes, then replace any remaining slashes with underscores. This sort of thing can be useful for naming backup files, data obtained from URL scraping, etc.

This is accomplished by adding a second regex_replace to the previous example that replaces all slashes with underscores, e.g. regex_replace('\\/', '_').

The combined regex that creates a valid filename from given a path:

{{ variable_name | regex_replace('^\\/|\\/$', '') | regex_replace('\\/', '_') }}

If variable_name held a value like “/var/www” or “/var/www/” it would result in: “var_www”.

In case you want to replace the slashes in the path with a character other than an underscore, you can adjust the examples to use any valid filename character instead of a slash (e.g. the ‘^’ character).

Working around a regex_replace bug in the shell module

In my working version of ansible (2.3.x) the regex_replace filter is ignored (!) when it is applied to variables in a tasks using the shell module. The value of unfiltered variables are substituted into the shell command fine, and using any other filter works fine too.

Workaround: employ the set_fact module to build a new fact (variable) based on the original variable, applying the regex_replace filter here as required. Reference the new fact in the shell module to take advantage of the pre-filtered values.

In the following example, assume that the hypothetical {{ list_of_paths }} variable contains a list of strings containing file/dir/URL paths. The set_fact module builds the new {{ paths }} fact such that it contains a “pi” item corresponding to every item in the original list. Each of these items has a “stripped” property containing the filtered value and a “path” property containing the original unfiltered value.

- name: workaround example
  set_fact: 
    paths: "{{ paths|default([]) + [{'pi': {'path': item, 'stripped': item|regex_replace('^\\/|\\/$', '')|regex_replace('\\/', '_')}}] }}"
      with_items: "{{ list_of_paths }}"

When looping over {{ paths }} in a shell task (e.g. via with_items), the filtered slash-free values for items can be referenced via {{ item.stripped }}. The original unfiltered path can be referenced via {{ item.path }}.

Install and secure phpmyadmin: require access via an SSH tunnel

Phpmyadmin is a popular database administration and query tool for MySQL and MariaDB. This guide covers how to install it, plus add an additional layer of security that only permits access via an encrypted ssh tunnel (also known as ssh port forwarding). This guide covers Debian and Ubuntu systems running apache2.

This particular approach is easy to automate and improves on other examples that I’ve come across online. Its the only tutorial that I know of that does things The Debian Way, in this case meaning that phpmyadmin’s customized configuration is applied without editing any of its packaged-managed conf files. This ensures that future package updates do not risk overwriting your security-minded configuration.

Why extra security measures are recommended for phpmyadmin

Phpmyadmin is a popular choice for attackers because it is widely deployed, easy to find, easy to attack, and it provides convenient access to potentially highly sensitive and/or valuable data.

A default package install of phpmyadmin on a public-facing web server is wide open to the Internet. It is often easily found with the default directory pattern “/phpmyadmin”.

A quick look at almost any website’s logs will reveal an endless barrage of attempts to locate phpmyadmin, probing a wide range of likely URL patterns and alternate names. In cases where phpmyadmin is discovered, its practically certain the logs will subsequently reveal attackers attempting to brute force and/or exploit their way in.

Another significant concern with phpmyadmin is that it does not come with ssl configured out-of-the-box. Login credentials and any data that’s accessed can be picked up by anyone that cares to listen in anywhere between your computer and the database server. This is especially a concern for anyone who accesses phpmyadmin via an untrusted network, such as public wifi.

Part 1: install phpmyadmin

First, make sure that you have mysql or mariadb installed and configured, and have taken basic steps to secure the setup. The mysql_secure_installation command is bundled with the mysql + mariadb packages and helps cover the basics.

Install the phpmyadmin package:

sudo apt-get update
sudo apt-get install phpmyadmin

The configuration script will prompt with a number of questions. When asked if you would like to automatically configure phpmyadmin for your web server, choose ‘apache2’.

Once the process is complete, you should be able to access phpmyadmin in a web browser by appending “/phpmyadmin” to your server’s URL, e.g.:

http://your-server.com/phpmyadmin

Part 2: Secure phpmyadmin

The following assumes that both apache2 and phpmyadmin were installed via the package manager.

Step 1: create a new conf file

Open your favourite editor and create a file called phpmyadmin-secure.conf in /etc/apache2/conf-available, e.g.:

sudo nano /etc/apache2/conf-available/phpmyadmin-secure.conf

For apache2.4, add the following content:

# ensure that phpmyadmin can only be accessed via localhost 
# remote users must use an ssh tunnel for access
<Directory /usr/share/phpmyadmin>
    Require local
</Directory>

You can double check your apache version with the command: apache2 -v.

For the older apache 2.2.x you need to use an older syntax inside the <Directory> block. If you do not need IPv6 support, delete the last directive in the following block (the entire line containing ::1):

<Directory /usr/share/phpmyadmin>
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1
    Allow from ::1
</Directory>

These directives tell apache that anything in the package install location /usr/share/phpmyadmin should only be accessible by localhost (127.0.0.1). Every other IP address, i.e. anything else on the internet, should be forbidden.

Save and exit (control+x in nano).

Step 2: Enable the conf file

Use a2enconf to enable the conf file:

cd /etc/apache2/conf-available
sudo a2enconf phpmyadmin-secure.conf

This command will create a symlink in apache’s conf-enabled/ directory that points to the conf file in the conf-available/ directory.

Note that the a2disconf command also exists and is invoked in a similar manner to a2enconf. It performs the reverse operation and disables the conf file by removing it from conf-enabled/ after it performs some checks.

Step 3: Reload apache and test

Apache must be reloaded or restarted for the updated configuration to take effect:

sudo service apache2 reload

After reloading/restarting apache, try to access phpmyadmin at its URL, e.g.:

http://your-server.com/phpmyadmin

You should see an error 403 / forbidden.

Tip: when making changes to apache’s conf files, you can check that the syntax is valid prior to reloading/restarting the server with the command apachectl configtest. Its a good idea to test your configuration before applying it because Apache will refuse to start if it has an invalid config, taking your site(s) offline until you can start it with a valid one.

Tip: if you want to hide the fact that phpmyadmin is even installed on your server at all, you can use apache’s mod_rewrite and/or RedirectMatch directives to produce a 404 ‘not found’ instead of a 403 response code.

Part 3: Access phpmyadmin via an ssh tunnel (ssh port forwarding)

To access your phpmyadmin install, connect to your server with ssh, using its tunnel options.

On your local machine, pick an arbitrary unused port and open a terminal (or an ssh client such as puTTy if your machine is a Windows PC). I use 5050 in the example below, but you can replace this with any available port number. Connect to the remote server via ssh:

ssh -L 5050:localhost:80 user@host.example.com

The -L option is key to establishing the tunnel. It tells ssh to forward any connections made to your local machine’s port 5050 to the remote host’s port 80 for as long as the ssh session is active.

Any web server (apache in our case) listening on port 80 will perceive that connections via the tunnel are coming from localhost and not from over a network / the internet.

Once you have a successful connection the server, visit the following URL in your browser to confirm you can access phpmyadmin:

http://localhost:5050/phpmyadmin

All traffic between you and the remote host (including credentials and data) will be encrypted via ssh for as long as your session remains open.

All done!

You can now use phpmyadmin with the knowledge that this powerful and useful but also risky tool is not nearly as exposed to the open internet as it would be with a default install, and that you have taken a small but significant step in the endless battle to protect your credentials and data.

Troubleshooting tips

  • If you can’t access phpmyadmin, one possible gotcha to consider is if your website or app has mod_rewrite redirection rules in place (e.g. to force users to use https, to redirect certain url patterns, etc). These rules may be interfering when you try to access phpmyadmin via the tunnel. If you need to carve out an exception for localhost, but not other hosts, you could add a Rewrite Condition such as: RewriteCond %{HTTP_HOST} !^localhost to exempt it from your other rewrite conditions.

Using s3cmd to access S3 buckets from EC2 instances with IAM Role authentication

s3cmd is an open-source command line tool for uploading, retrieving and managing data in Amazon S3 and other providers that implement the S3 protocol (e.g. Google Cloud Storage, DreamHost DreamObjects, etc). It is popular tool with a variety of applications, including backup scripts.

This post covers using s3cmd within an EC2 instance, with authentication to S3 managed via IAM Roles (IAM = Identity and Access Management).

If your project doesn’t have dependencies on scripts and/or pre-existing code that require s3cmd, and you don’t mind locking your project deeper into AWS’ ecosystem, a good alternative is Amazon’s own AWS CLI.

Configuring an IAM Role

When an EC2 instance is launched with an associated IAM role, access keys and secret keys do not need to be stored in config files on the instance itself.

After such an instance is launched, AWS-aware services and scripts on the it can access AWS resources subject to the permissions defined in the IAM role’s associated IAM role policy. Authentication is seamlessly handled via the AWS SDK.

s3cmd versions 1.5.0-alpha2 and above support IAM roles and authentication via the AWS SDK.

IAM roles

Log in to your AWS account and access the AWS IAM Dashboard (also referred to as the IAM Management Console) to manage role-based permissions.

Create an IAM role for your instances if you haven’t already.

IAM role policy

In the IAM Management Console, ensure that your IAM role has an attached policy that provides access to any S3 resource(s) that you want your instance(s) to access via s3cmd. It’s advisable to ensure that role policies are restricted to the minimum amount of scope possible.

Within the IAM Management Console:

  • Choose Roles and select the role that your EC2 instances belong to (or will belong to)
  • Review your existing policies to confirm what your instances can currently access
  • If necessary, use the Create Role Policy button to add a policy with the Policy Generator

For example, suppose you had a an s3 bucket called example-log-bucket and you wanted to configure your EC2 instances to send access logs to it.

Following the order of the Policy Generator’s input fields, you’d create: an Allow policy for the Amazon S3 service regarding the actions s3:ListBucket, s3:PutObject, and s3:PutObjectAcl, and apply them to the ARN arn:aws:s3:::example-log-bucket.

To learn more about how to specify ARNs that identify your various AWS services, see the Amazon docs: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3

Note that you can use wildcards in ARNs when defining a Role Policy.

Installing and configuring s3cmd on the EC2 instance(s)

s3cmd versions 1.5.0-alpha2 and above support IAM roles.

In earlier versions, an access key and secret key had to be specified in a .s3cfg file in the home directory of the user running the s3cmd command, or in environment variables.

Option A: Install s3cmd with the Ubuntu/Debian apt package manager

If you’re running Ubuntu 16.04 or later (test box at the time of writing) the package version is sufficiently up-to-date (>1.5.0). Install s3cmd with apt-get:

sudo apt-get install s3cmd

Option B: Install s3cmd with the python pip package manager

Pip is an alternative to Debian/Ubuntu apt-get, especially if you are using a different distro or like to use python/pip.

sudo pip install s3cmd

Check the version number with s3cmd --version. If you discover that its <1.5.0, you can specify a particular version for pip to install as follows, substituting in your desired version number (e.g. 1.5.0-alpha3) in place of VERSION_NUMBER below:

pip install s3cmd==VERSION_NUMBER

Configure s3cmd on the EC2 instance(s)

Once an EC2 instance is launched with a suitable IAM Role and associated IAM Role Policy, its easy to get started with s3cmd. The following example is set in the context of the currently logged in user.

Open/create the file ~/.s3cfg using a text editor:

nano ~/.s3cfg

Populate the file with the following content. Rather than filling in hardcoded credentials, leave each of the fields blank (use the example as is; do not specify any values):

[default]
access_key =
secret_key = 
security_token =

Save the file and exit the editor.

Later versions of s3cmd that support IAM Roles will auto-magically detect the appropriate authentication values.

s3cmd will now be able to interact with any AWS resource(s) that it is permitted to access, subject to s3cmd’s capabilities and the IAM Role’s active Role Policies.

Use s3cmd

s3cmd has over 60 command line options. Check out the documentation:

http://s3tools.org/usage