Troubleshooting the fast.ai AWS setup scripts (setup_p2.sh)

Fast.ai offers a well-regarded free online course on Deep Learning that I thought I’d check out.

It seems that a lot of people struggle getting the fast.ai setup scripts running. Complaints and requests for help are on reddit, in forums, etc. This doesn’t surprise me because the scripts are not very robust. On top of that, AWS has a learning curve so troubleshooting following a script failure can be a challenge.

Hopefully this post helps other people that have hit snags. It is based on my experience on MacOS, however should be very compatible for those running Linux or Windows with Cygwin.

Understanding the setup script’s behaviour

It leaves a mess when it fails

If running the setup script fails, which is possible for a number of reasons, it will potentially have created a number of AWS resources in your account and a local copy of an SSH key at ~/.ssh/aws-key-fast-ai.pem. It does not clean up after itself in failure cases.

The setup script doesn’t check for existing fast-ai tagged infrastructure, so subsequent runs can create additional VPC’s and related resources on AWS, especially as you attempt to resolve the reason(s) it failed. The setup script might generate fast-ai-remove.sh and fast-ai-commands.txt but it overwrites these each time its run with only its current values, potentially leaving “orphan” infrastructure.

Thankfully all AWS resources are created with the same “fast-ai” tags so they are easy to spot within the AWS Console.

It makes unrealistic assumptions

The setup script assumes your aws config’s defaults specify a default region in one of its three supported regions: us-west-2, eu-west-1, and us-east-1.

I’m not sure why the authors assumed that a global tech crowd interested machine learning would be unlikely to have worked with AWS in the past and thus no existing aws configuration that might conflict.

The commands in the script do not use the --region argument to specify an explicit region so they will use whatever your default is. If your default happens to be one of the three supported ones, but you don’t have a sufficient InstanceLimit or there’s another problem, more issues could follow.

Troubleshooting

If you encountered an error after running the script, prior to re-running the script, take note of the following checks when attempting to resolve:

Check 1: Ensure you have an InstanceLimit > 0

Most AWS users will have a default InstanceLimit of 0 on P2 instances. You may need to apply for an increase and get it approved (this is covered in the fast.ai setup video).

If a first run of the script gave you something like the following, there was an issue with your InstanceLimit:

Error: *An error occurred (InstanceLimitExceeded) when calling the RunInstances operation: You have requested more instances (1) than your current instance limit of 0 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.* 

InstantLimits are specific to a given resource in a given region. Take note of which region your InstanceLimit increase request was for and verify that it was granted in the same region.

Check 2: Ensure the right region

Verify your current default aws region by running: aws configure get region. The script assumes this is one of three supported regions: us-west-2, eu-west-1, or us-east-1.

The script also assumes that you have an InstanceLimit > 0 for P2 instances in whichever region you would like to use (or T2 instances if you are using setup_t2.sh).

To get things running quickly, I personally found it easiest to make the script happy and temporarily set my aws default to a supported region in ~/.aws/config, i.e.:

[default]
region=us-west-2

Another option is to modify the scripts and add an explicit --region argument to every aws command that will override the default region. If you have multiple aws profiles defined as named profiles, and the profile that you wish to use for fast.ai specifies a default region, you can use the --profile PROFILENAME argument instead.

For example, the following hypothetical aws config file (~/.aws/config) specifies a profile called “fastai”. A --profile fastai argument could then be added to every aws command in the setup script:

[default]
region=ca-central-1

[profile fastai]
region=us-west-2

Check 3: Delete cruft from previous failed runs

This check is what inspired me to write this post!

Delete AWS resources

Review any resources were created in your AWS Console, and delete any VPC’s (and any dependencies) that were spun up. They can be identified because they were created with the “fast-ai” tag which is shown in any tables of resources in the AWS Console.

Cruft resources will have been created in any region that the setup script was working with (i.e. whatever your default region was at the time you ran it).

If you’ve found cruft, start by trying to delete the VPC itself, as this generally will delete most if not all dependencies. If this fails because of a dependency issue, you will need to find and delete those dependencies first.

IMPORTANT: AWS creates a default VPC and related dependencies (subnets, etc.) in every region available to your account. Do NOT delete any region’s default VPC. Only delete resources tagged with “fast-ai”.

Delete SSH keys

Check to see if ~/.ssh/aws-key-fast-ai.pem was created, and if so, delete it before running the script again.

The setup script has logic that checks for this pem file. We do not want the script to find the file on a fresh run.

After a successful run

After the setup script ran successfully, I got output similar to:

{
    "Return": true
}
Waiting for instance start...

All done. Find all you need to connect in the fast-ai-commands.txt file and to remove the stack call fast-ai-remove.sh
Connect to your instance: ssh -i /Users/username/.ssh/aws-key-fast-ai.pem ubuntu@ec2-XX-YY-ZZ-XXX.us-west-2.compute.amazonaws.com

Reference fast-ai-commands.txt for information about your VPC and EC2 instance. An ssh command to connect is in the file, and you can find your “InstanceUrl”.

I suggest picking up the video from here and following along from the point where you connect to your new instance. It guides you through checking the video card with the nvidia-smi command and running jupyter: http://course.fast.ai/lessons/aws.html

Starting and stopping your instance

The fast-ai-commands.txt file outlines the commands to start and stop your instance after the setup has completed successfully, e.g.:

aws ec2 start-instances --instance-ids i-0XXXX
aws ec2 stop-instances --instance-ids i-0XXXX

Its important to stop instances when you are finished using them so that you don’t get charged hourly fees for their continued operation. P2 instances run about $0.90/hr at the time of writing.

How to write custom wp-cli commands for next-level WordPress automation

wp-cli is a powerful tool for WordPress admins and developers to control their sites from the command-line and via scripts.

This is an introductory guide to writing custom wp-cli commands using a plugin, to enable you to expand its features and tailor its capabilities to your projects.

Custom commands are invoked like any other:

wp custom_command example_custom_subcommand

Why wp-cli custom functions are useful

Custom commands can help automate the management of more complex sites, enhance one’s development workflow, and enable greater control over other themes and plugins.

Practical real-world example

On a past project, I implemented wp-cli commands that enabled me to quickly load 100’s of pages of multilingual content in English, French, and Chinese, plus the translation associations between them all, to local and remote WP installs. The internationalization plugin in play, the popular and notorious pain-in-the-ass WPML, had no wp-cli support (and still doesn’t!). It otherwise would’ve required the clicking of bajillions of checkboxes every time copy/translation decks or certain features were revised.

Bare-bones plugin implementation

The following assumes that the wp-cli command is present (and accessible via wp), and that the example plugin has been correctly installed and activated on the target WordPress.

This code creates a basic plugin class called ExamplePluginWPCLI that is only loaded when the WP_CLI constant is defined:

if ( defined( 'WP_CLI' ) && WP_CLI ) {

    class ExamplePluginWPCLI {

        public function __construct() {

                // example constructor called when plugin loads

        }

        public function exposed_function() {

                // give output
                WP_CLI::success( 'hello from exposed_function() !' );

        }

        public function exposed_function_with_args( $args, $assoc_args ) {

                // process arguments 

                // do cool stuff

                // give output
                WP_CLI::success( 'hello from exposed_function_with_args() !' );

        }

    }

    WP_CLI::add_command( 'firx', 'ExamplePluginWPCLI' );

}

Adding a custom command to wp-cli

Consider the line WP_CLI::add_command( 'firx', 'ExamplePluginWPCLI' ); from the above example.

The command’s name, firx, is given as the first argument. You can choose any name for custom commands that aren’t already reserved by an existing command, provided that it doesn’t contain any special characters. It is wise to pick a unique name that minimizes the risk of conflicts with any other plugin or theme that might also add commands to wp-cli.

Defining new wp-cli commands in a class like ExamplePluginWPCLI confers a special advantage over defining them using only standalone php functions or closures: all public methods in classes passed to WP_CLI::add_command() are automatically registered with wp-cli as sub-commands.

Executing a custom command

The class’ public function exposed_function() can be called via wp-cli as follows:

wp firx exposed_function

The class’ public function exposed_function_with_args() can be called via wp-cli as follows. This particular function accepts command line arguments that will get passed into it via its $args and $assoc_args variables as appropriate:

wp firx exposed_function_with_args --make-tacos=beef --supersize

The constructor __construct() is optional and is included as an example. This function is called when an instance of the plugin class is loaded and it can be used to define class variables and perform any necessary setup tasks.

Output: success, failure, warnings, and more

Sending information back to the command line

wp-cli has a number of functions for outputting information back to the command line. The most commonly used are:

// works similar to a 'return' and exits after displaying the message to STDOUT with 'SUCCESS' prefix 
WP_CLI::success( 'Message' )

// works similar to a 'return' and exits after displaying the message to STDERR with 'ERROR' prefix
WP_CLI::error( 'Message' )

// display a line of text to STDERR with no prefix 
WP_CLI::log( 'Message' ) 

// display a line of text when the --debug flag is used with your command
WP_CLI::debug( 'Message' ) 

The success() and error() functions generally serve as a return equivalent for wp-cli functions. If either of these functions are called with a single argument containing a message, the script will exit after the message is displayed.

Formatting output as tables, json, etc

wp-cli has a useful helper function called format_items() that makes it a lot easier and cleaner to output detailed information.

Available options to output json, csv, count, and yaml are brilliant for enabling command output to be easily used by scripts and/or digested by web services.

The function accepts 3 arguments:

WP-CLI::format_items( $format, $items, $fields )
  • $format – string that accepts any of: ‘table’, ‘json’, ‘csv’, ‘yaml’, ‘ids’, ‘count’
  • $items – Array of items to output (must be consistently structured)
  • $fields – Array or string containing a csv list to designate as the field names (or table headers) of the items

For example, the following code:

$fields = array ( 'name', 'rank' ); 
$items = array (
    array (
        'name' => 'bob',
        'rank' => 'underling',
    ),
    array (
        'name' => 'sally',
        'rank' => 'boss',
    ),
);
WP_CLI\Utils\format_items( 'table', $items, $fields );

Would output something similar to:

# +-------+-----------+
# | name  | rank      |
# +-------+-----------+
# | bob   | underling |
# +-------+-----------+
# | sally | boss      | 
# +-------+-----------+

Changing the format to ‘json’ or ‘yaml’ is as simple as swapping out the ‘table’ argument.

Input: handling arguments

Positional and associative arguments

wp-cli supports both positional arguments and associative arguments.

Positional arguments are interpreted based on the order they are specified:

wp command arg1 42

Associative arguments may be specified in any order, and they can accept values:

wp command --make-tacos=beef --supersize --fave-dog-name='Trixie the Mutt'

Both positional and associative arguments can be supported by the same command.

A function implemented with two parameters for $args and $assoc_args, such the one from the first big example in this guide exposed_function_with_args( $args, $assoc_args ), will be provided all positional arguments via the $args variable and all associative arguments via the $assoc_args variable.

Retrieving values passed to a command

Suppose we wanted to process all of the arguments for the command:

wp firx exposed_function_with_args arg1 42 --make-tacos=veggie --supersize --fave-dog-name='Trixie the Mutt'

The following example expands on the initial example’s implementation of the exposed_function_with_args() function to demonstrate how to access the values of each argument:

public function exposed_function_with_args( $args, $assoc_args ) {

    // process positional arguments - option 1
    $first_value = $args[0];  // value: "arg1"
    $second_value = $args[1]; // value: 42

    // OR - process positional arguments - option 2
    list( $first_value, $second_value ) = $args;

    // process associative arguments - option 1 
    $tacos = $assoc_args['make-tacos']; // value: "veggie"
    $supersize = $assoc_args['supersize'];  // value: true
    $dog_name = $assoc_args['fave-dog-name'];    // value: "Trixie the Mutt"

    // OR - process associative arguments - option 2 - preferred !! 
    $tacos = WP_CLI\Utils\get_flag_value($assoc_args, 'make-tacos', 'chicken' );
    $supersize = WP_CLI\Utils\get_flag_value($assoc_args, 'supersize', false );
    $dog_name = WP_CLI\Utils\get_flag_value($assoc_args, 'fave-dog-name' );

    // do cool stuff

    // provide output
    WP_CLI::success( 'successfully called exposed_function_with_args() !' );

}

Using get_flag_value() to help with associative arguments

wp-cli comes with a handy helper function for handling associative arguments that serves as the preferred way to access them:

WP_CLI\Utils\get_flag_value($assoc_args, $flag, $default = null)

This function is passed the $assoc_args variable, the $flag (argument name) you wish to access, and optionally a default value to fall-back on if you want it to be something other than null.

What makes get_flag_value() the preferred method for accessing values is that it takes care of implementing a few tricks, including supporting the no prefix on arguments to negate them. Consider that a professional implementation of the above --supersize option should also check for and handle the case if a --no-supersize version is passed. Using the get_flag_value() function to access the values of the argument (e.g. using “supersize” as the identifier for the middle $flag argument) would handle this case and you’d be assured that your function would automatically receive the correct ‘true’ or ‘false’ value to work with.

Working with arguments

This is an introductory guide and the examples are for illustrative purposes only. For a robust implementation, keep in mind that you will likely need to add additional logic to any function that accepts arguments to check if required items are specified or not, if expected/supported values were passed in or not, etc.

These cases can be handled by your php code and can leverage wp-cli’s output functions like warning() or error() to provide user feedback. Another option that can cover many validation cases is to leverage wp-cli’s PHPDoc features (more on that below).

Registering arguments with wp-cli

An easy way to register arguments/options, whether they’re mandatory or not, and specify any default values is to use PHPDoc comment blocks. These play an active role in wp-cli which intelligently interprets them. Refer to the section on PHPDoc near the end of this guide.

wp-cli custom functions will still work in the absence of PHPDoc comments but they won’t be as user-friendly and any arguments won’t be tightly validated.

Errors: error handling with wp-cli

Use the WP_CLI::error() function to throw an error within a class or function that implements wp-cli commands:

// throw an error that will exit the script (default behaviour)
WP_CLI::error( 'Something is afoot!' );

// throw an error that won't exit the script (consider using a warning instead)
WP_CLI::error( 'The special file is missing.', false );

Error output is written to STDERR (exit(1)), which is important to consider when writing scripts that use wp-cli and respond to error conditions.

Use WP_CLI::warning() to write a warning message to STDERR that won’t halt execution:

WP_CLI::warning( $message )

Use WP_CLI::halt() to halt execution with a specific integer return code (this one is mostly for those writing scripts):

WP_CLI::halt ( $return_code )

PHPDoc comments and controls

PHPDoc style comment blocks are more than comments when it comes to wp-cli: it interprets them to provide help text to cli users, document the command’s available options/arguments and their default values, and serve as the basis for a level of validation that that gets performed before a command’s underlying function gets called and any argument data is passed to it. Complete and descriptive comments will result in the enforcement of mandatory vs. optional parameters by wp-cli.

Implementing correct PHPDoc comments simplifies and expedites the implementation of custom commands because the developer defers certain validation checks to wp-cli (e.g. to ensure a required argument was specified) rather than implementing everything on their own.

PHPDoc comments are used as follows:

  • The first comment line corresponds to the “shortDesc” shown on the cli
  • The “meat” of a comment body corresponds to the “longDesc” may be shown when cli users mess up a command, and is shown when they specify the --help flag with a given command
  • Options (aka arguments aka parameters in this context) that are defined specify if each parameter is mandatory vs. optional, and if a single or multiple arguments are accepted

The “shortDesc” and “longDesc” may be displayed to cli users as they interact with the command. To show a full description, cli users can execute:

wp [command_name] --help

Basic PHPDoc comment structure

PHPDoc comments are placed immediately preceding both Class and method (function) declarations, and have a special syntax that starts with /**, has a 2-space-indented * prefixing each subsequent line, and ends with an indented */ on its own line:

/**
 * Implements example command that does xyz
 */
ClassOrMethodName {
    // ...
}

PHPDoc comments with parameters

The wp-cli’s Command Cookbook provides the following example of a PHPDoc comment:

    /**
     * Prints a greeting.
     *
     * ## OPTIONS
     *
     * <name>
     * : The name of the person to greet.
     *
     * [--type=<type>]
     * : Whether or not to greet the person with success or error.
     * ---
     * default: success
     * options:
     *   - success
     *   - error
     * ---
     *
     * ## EXAMPLES
     *
     *     wp example hello Newman
     *
     * @when after_wp_load
     */
    function say-hello( $args, $assoc_args ) {
        // ... 
    }

The #OPTIONS and #EXAMPLES headers within the “meat” of the comment (corresponding to the “longDesc”) are optional but are generally recommended by the wp-cli team.

The arguments/parameters are specified under the #OPTIONS headers:

  • <name> specifies a required positional argument
    • writing it as <name>... means the command accepts 1 or more positional arguments
  • [--type=<type>] specifies an optional associative argument that accepts a value
    • [some-option] without the = sign specifies an optional associative argument that serves as a boolean true/false flag
  • the example’s default: and options: provided for the [--type=<type>] argument can be specified under any argument to communicate that argument’s default value and the options available to cli users

In case word wrapping is a concern for you when help text is presented:

  • Hard-wrap (add a newline) option descriptions at 75 chars after the colon and a space
  • Hard-wrap everything else at 90 chars

More information about PHPDoc syntax in the wp-cli context is available in the docs:

https://make.wordpress.org/cli/handbook/documentation-standards/

Help with running wp-cli commands

There are a couple helpful reminders to keep in mind when using wp-cli that could prove useful to someone following this guide.

Specifying a correct wordpress path

wp-cli must always be run against a WordPress installation.

  • You can cd to the path where WordPress has been installed, so that the shell prompt’s working directory is at the base where all the WordPress files are (e.g. on many systems, this would be something similar to: cd /var/www/example-wordpress-site.com/public_html)
  • Alternatively you can provide the universal argument --path when specifying any wp-cli command (e.g. --path=/var/www/example-wordpress-site.com/public_html) to run a command from any folder

Running wp-cli as a non-root user

It is dangerous to run wp-cli as root and it will happily inform you of this fact should you try. That’s because php scripts related to your WordPress and wp-cli would be executed with full root permissions. They could be malicious (especially risky if your site allows uploads!) or poorly implemented with bugs, and that fact unreasonably puts your whole system at risk.

A popular option is to use sudo with its -u option to run a command as another user. Assuming that you have sudo installed and correctly configured, and assuming that the user ‘www-data’ is the “web server user” on your system and that it has read/write permissions to your WordPress installation folder (these are common defaults found on Ubuntu and Debian systems), you can prefix your commands with: sudo -u www-data --.

The following executes one of wp-cli’s built-in commands, wp post list, as the www-data user:

sudo -u www-data -- wp post list

Further reading

Check out the wp-cli docs and the commands cookbook at:

Avoiding duplicate entries in authorized_keys (ssh) in bash and ansible

Popular methods of adding an ssh public key to a remote host’s authorized_keys file include using the ssh-copy-id command, and using bash operators such as >> to append to the file.

An issue with ssh-copy-id is that this command does not check if a key already exists. This creates a hassle for scripts and automations because subsequent runs can add duplicate key entries. This command is also not bundled with MacOS, creating issues for some Mac users (though it can be installed with Homebrew).

This post covers a solution that adds a given key to authorized_keys only if that key isn’t already present in the file. Examples are provided in bash and for ansible using ansible’s shell module (old versions) and authorized_key module (newer versions).

For shell scripts, there seem to be a lot of solutions out there for this common problem, but I think a lot of them overcomplicate things with sed, awk, uniq, and similar commands; or go overboard by implementing standalone utilities for the task. One thing I don’t like about many of the working solutions that I’ve come across is when the authorized_keys file is reordered as a side-effect.

Note that ssh authentication works fine when there are multiple identical authorized_keys entries. However, accumulating junk in this file can create performance issues, and can make troubleshooting, auditing, and other admin tasks more difficult. When a remote host tries to authenticate, ssh works its way down the authorized_keys file until it comes across a match.

Adding a unique entry to authorized_keys

The following is a one-liner to be run by a user that can authenticate with the remote server.

Modify the snippet below to suit your needs:

ssh -T user@central.example.com "umask 0077 ; mkdir -p ~/.ssh ; grep -q -F \"$PUB_KEY\" ~/.ssh/authorized_keys 2>/dev/null || echo \"$PUB_KEY\" >> ~/.ssh/authorized_keys"

The command adds the public key stored in the shell variable $PUB_KEY to the authorized_keys file of the user on the server central.example.com. A umask ensures the correct file permissions.

To modify, replace user and central.example.com with values relevant to you, and either substitute your public key in place of the $PUB_KEY variable, or define the variable in a bash script or set it as an environment variable prior to executing the command.

Benefits of this approach:

  • unique entries: no duplicate authorized_keys
  • idempotent: subsequent runs given the same input will yield the same result
  • order preserved: entries in authorized_keys retain their order
  • correct permissions: in cases where the .ssh folder and/or authorized_keys file do not already exist, they will be created with the correct permissions for openssh thanks to the umask
  • quiet: the command is quiet
  • automation friendly: fast one-liner that’s easy to add to scripts, with minimized race conditions in situations that involve running automations (e.g. ansible playbooks) in parallel
  • KISS principle: its not as risky or difficult to configure as some other approaches that I have encountered online

Tip: If you want to suppress any motd/welcome banner content that might be outputted when connecting to the remote server via ssh, first touch a .hushlogin file in the target user’s home directory to suppress it.

Ansible implementation

Current: using the authorized_key module

The newer known_hosts module and authorized_key module (featuring numerous feature additions from its introduction through to 2.4+) were introduced to help manage ssh keys on a host.

The authorized_key module has a lot of useful options, including optional exclusivity, supporting sourcing keys from variables (and hence files via a lookup) as well as URL’s, and options to manage the authorized_keys/ folder (e.g. creating it with appropriate permissions if it doesn’t exist).

An example from the docs follows, with one addition: I added the exclusive option in keeping with the theme of this post.

- name: Set authorized key took from file
  authorized_key:
    user: charlie
    state: present
    key: "{{ lookup('file', '/home/charlie/.ssh/id_rsa.pub') }}"
    exclusive: yes

See the ansible documentation for more examples: http://docs.ansible.com/ansible/latest/authorized_key_module.html

Legacy: using bash in the shell module

One of the more annoying aspects of ansible can be getting escape characters right in templates and certain modules like shell, especially when variables are involved. The following example has valid syntax. You can modify the variables and the become and delegate_to args to suit your scenario:

# assume the ansible user on the control machine can access the remote target server via ssh 

- name: set_fact host_pub_key containing current host's pub key from local playbook_dir/keys
  set_fact:
    host_pub_key: "{{ lookup('file', playbook_dir + '/keys/{{ inventory_hostname }}-{{ authorized_user }}-id_rsa.pub') }}"

- name: add current host's pub key to repo server's authorized_keys if its not already present 
  shell: |
    ssh -T {{ example_user }}@{{ example_server }} "umask 0077 ; mkdir -p ~/.ssh ; grep -q -F \"{{ host_pub_key }}\" ~/.ssh/authorized_keys 2>/dev/null || echo \"{{ host_pub_key }}\" >> ~/.ssh/authorized_keys"
  args:
    executable: /bin/bash
  become: "{{ ansible_user_id }}"
  delegate_to: localhost

The first task populates the host_pub_key fact from a hypothetical id_rsa.pub key file.

The second task executes the bash snippet that adds the public key to the remote host’s authorized_keys file in a way that avoids duplicates.

Creating certificates and keys for OpenVPN server with EasyRSA on MacOS

This guide covers how to create certificates and keys for OpenVPN server and clients using the EasyRSA tool on MacOS.

The instructions are very similar for most flavours of linux such as Ubuntu once the correct packages are installed (e.g. on Ubuntu: apt-get install openvpn easy-rsa).

If privacy and security are of the utmost concern, generate all certificates and keys on a “clean” machine and verify the signatures of each download.

Step 1: Resolve MacOS Dependencies

This guide assumes that you’re running MacOS Sierra or later.

XCode and Command Line Tools

Ensure that you have installed the XCode Command Line Tools.

To check, the command xcode-select -p outputs a file path beginning with /Applications/Xcode.app/ if they are already installed.

If Command Line Tools is not installed, open the Terminal app and enter xcode-select --install to trigger the installation app.

Another way to trigger the installation app is to attempt to use a command line developer tool such as the GNU C compiler gcc (e.g. gcc --version). If the tools are not installed, you will be greeted by a graphical MacOS installation prompt instead of the expected Terminal output from gcc. You don’t necessarily need the full XCode so you can click the “install” button for just the command line tools.

Work your way through the installer and follow Apple’s steps until you can start working with the necessary commands. The CLI commands will become available to you after you agree to all of Apple’s terms and conditions.

If you experience troubles with the next step, assuming that it is the result of some future change by Apple, it may be beneficial to install the full XCode in addition to the CLI tools. It’s available for free on the App Store, but take note that it’s a hefty multi-gigabyte download.

OpenSSL

EasyRSA requires a late-version of the open-source openssl library.

Apple bundles its own crypto libraries in MacOS but these are generally out of date. At the time of writing, the openssl command bundled with MacOS is not likely compatible with EasyRSA and will produce errors if you try to use it (note: the binary is at /usr/bin/openssl).

A newer EasyRSA-compatible version of OpenSSL is easy to install with the brew package manager (https://brew.sh/). Installing via brew will not clobber or harm the Apple version that’s already on your system. If you need to install brew, go to the project’s website and follow the simple instructions on the landing page.

Assuming you have brew installed, open a Terminal and run the command:

brew install openssl

Brew will download and install openssl to its default package install directory of /usr/local/Cellar.

The package will be installed in “keg only” mode: brew will not create a symlink for its openssl in /usr/local/bin or anywhere else in your $PATH. You will not have a conflicting openssl command, and Apple’s binary will remain intact.

To get EasyRSA to use the openssl binary installed by the brew package, you will need to know its path. Run brew’s package info command and examine the output:

brew info openssl

In my example, I could see that openssl resolved to /usr/local/Cellar/openssl/1.0.2n. In your case, this may be a different path due to a more recent version being available in the future. Next, inspect this folder to locate the binary and determine the full path to it. In my example case, the full path to the binary was:

/usr/local/Cellar/openssl/1.0.2n/bin/openssl

Note down the correct path to the openssl binary for your case. When configuring EasyRSA in the next step, you will need to specify this path in an EASYRSA_OPENSSL variable.

Step 2: Download EasyRSA

Go to https://github.com/OpenVPN/easy-rsa/releases and download the latest .tgz version for your Mac.

Save the file to a folder that you wish to work from (your certificates and keys will be generated here) and unpack it using the Archive utility (double click on it in Finder).

Note that the easy-rsa tools were written with traditional linux/unix-type environments in mind and therefore assume that all paths to the scripts have no spaces in them.

Going forward I will assume the path of your unpacked EasyRSA folder is: ~/vpn/easyrsa. The ‘~’ character is a shortcut to your home folder that works in Terminal, i.e. on a Mac its a placeholder for /Users/your_username and on a typical linux environment /home/username.

Step 3: Configure EasyRSA

Assuming that the path of your unpacked EasyRSA folder is: ~/vpn/easyrsa, open Terminal and navigate to the unpacked folder:

cd ~/vpn/easyrsa

Copy the vars.example “starter” configuration file to vars:

cp vars.example vars

Now customize the initial “starter” configuration file’s settings in vars to reflect your own.

Open it in a text editor and look for the following lines. Uncomment them (i.e. delete the preceding # character) and fill them in with your appropriate values. Specify something for each field below:

#set_var EASYRSA_REQ_COUNTRY   "US"
#set_var EASYRSA_REQ_PROVINCE  "California"
#set_var EASYRSA_REQ_CITY  "San Francisco"
#set_var EASYRSA_REQ_ORG   "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL "me@example.net"
#set_var EASYRSA_REQ_OU        "My Organizational Unit"

Look for the following field and uncomment it:

#set_var EASYRSA_KEY_SIZE        2048

We’ll be using a 2048-bit key (the current default) for this example so the value will not be changed.

A larger key size is more secure but will result in longer connection + wait times over the VPN. At the time of writing in late 2017, its generally believed that a 2048-bit key is sufficient for most usage scenarios. A 4096-bit key is believed to provide additional privacy vs. more powerful state-sponsored actors.

EasyRSA by default uses the openssl binary found in the $PATH. Find the following line, uncomment it, and update the value with the path to the brew-installed openssl binary from Step 1. For example, in my case, the following line:

#set_var EASYRSA_OPENSSL   "openssl"

became:

set_var EASYRSA_OPENSSL "/usr/local/Cellar/openssl/1.0.2n/bin/openssl"

Step 4: Generate Certificate Authority (CA)

Navigate into your easyrsa/ folder. For example:

cd ~/vpn/easyrsa

Initialize the PKI (public key infrastructure) with the easyrsa script. This will create a pki/ subfolder:

./easyrsa init-pki

Create the CA (certificate authority):

./easyrsa build-ca nopass

You will be prompted to input a Common Name. Input the name server and hit ENTER.

The generated CA certificate can now be found at pki/ca.crt.

Step 5: Generate Server Certificate + Key + DH Parameters

Assuming you’re still inside your easyrsa/ folder from the previous step, generate your server certificate and key:

./easyrsa build-server-full server nopass

The generated server certificate can now be found at: pki/issued/server.crt

The generated server key can now be found at: pki/private/server.key

Now generate the Diffie-Hellman (DH) parameters for key exchange. This process can take several minutes depending on your system:

./easyrsa gen-dh

The generated DH parameters can be found at: pki/dh.pem.

You now have all of the files necessary to configure an OpenVPN server.

Step 6: Generate client credentials

You should generate a unique set of credentials for each and every client that will connect to your VPN. You can repeat this step for any client that you need to create credentials for.

All clients in your setup should have a unique name. Change exampleclient in the following to something descriptive that you will recognize and be able to associate with the user/client:

./easyrsa build-client-full exampleclient nopass

The generated client certificate: pki/issued/exampleclient.crt

The generated client key can be found at: pki/private/exampleclient.key

When distributing credentials to a client, they will need at least these 3 files:

  • A client certificate (e.g. pki/issued/exampleclient.crt)
  • The corresponding client key (e.g. pki/private/exampleclient.key)
  • A copy of the CA certificate (pki/ca.crt)

These client credentials can be loaded into a VPN app like Tunnelblick or Viscosity along with client configuration information that corresponds to your VPN server’s specific settings.

Understanding client config files

Client configuration information is usually provided in the form of an additional file: a plaintext config file with the .ovpn extension. Both Tunnelblick and Viscosity recognize the .ovpn extension and file format.

Later versions of openvpn support specifying all of the client configuration information, client certificate, client key, and CA certificate as demarcated blocks within the config file itself, so that clients only need to be provided with a single .ovpn file.

Security reminder

It is good to practice to try and keep all .ovpn, certificate, and key files as safe as possible with exposure to as few eyes/hands/hard-disks/clouds/etc as possible. Distribute them as securely as you can to your clients/users.

Next steps

Now you need a working openvpn server and a client that wishes to connect to your VPN!

I hope this guide was helpful to you. For all the server tutorials out there, as far as I know this is one of the few comprehensive guides out there for creating all required certificates and keys on MacOS.

Router as OpenVPN server

If your openvpn server is your router, you can now login to it’s admin control panel and input the server-related certificate + key + DH parameters that you created above.

Before you activate the VPN server, ensure that your router’s firmware is up-to-date and that you have set a long and reasonably secure password for the admin user.

Running your own server

If you are planning to setup your own openvpn server, there are numerous other resources available online to guide you through the server installation and configuration process for a variety of different operating systems.

You will find that you need all the keys and certificates that you created by following this guide.

These resources will generally include guidance for crafting .ovpn client configuration files to include specific settings that correspond to your server’s particular setup, so that clients can successfully connect.

Send emails from MacOS Terminal or scripts using Rackspace’s SMTP server

This guide will help configure MacOS so it can send emails from the command-line (Terminal) and shell scripts via an SMTP server. This is useful for enabling scripts, scheduled jobs, etc. to send email notifications.

The example in this post work for Rackspace’s SMTP server. The guide is adaptable to other SMTP servers and email providers, while acknowledging that you may need to adjust certain details. Email providers can vary in how they require users to authenticate and interface with their SMTP servers.

There are lot of examples online for using gmail’s SMTP servers, but since I couldn’t find any complete examples for Rackspace, I decided to write this post!

Configuring postfix on MacOS

MacOS/OSX comes bundled with the postfix mail server. We will configure that to use Rackspace’s SMTP server to send mail.

Prerequisites

To use Rackspace’s SMTP, its a given that you must be a Rackspace customer. Create an email address that you wish to be the “sender” of any notifications. Note this email account’s credentials as you’ll need to include them in your postfix configuration.

I recommend that you use a dedicated email address for sending automated notifications (e.g. no-reply@.., notify@.., etc), rather than an email address with an inbox that is important to your business. Within Rackspace’s control panel, you have the option to configure an auto-responder for your automated notification address so that anyone that replies to a notification or sends anything else to it will receive a canned reply in response.

Step1: Edit main postfix configuration file

Make a copy of your original postfix conf file:

sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.bak

Open postfix’s config file for editing:

sudo nano /etc/postfix/main.cf

In an untouched postfix configuration on MacOS, the following 3x separate lines have the following values:

  • mydomain_fallback = localhost
  • mail_owner = _postfix
  • setgid_group = _postdrop.

If you have previously edited your postfix settings, you may wish to search for each item and reset these back to their default values (first confirm that you aren’t about to break anything!).

Add the following to the very end of the open postfix config file:

relayhost = smtp.emailsrvr.com:587
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = AUTH LOGIN
smtp_sasl_security_options =
smtp_use_tls=yes
smtp_tls_security_level=encrypt

These setting are specific to Rackspace’s requirements. If you are using another SMTP server, you will need to specify different values.

Save and exit the editor.

Step 2: create the sasl_passwd file for postfix to use

Create the /etc/postfix/sasl_passwd file referenced in main.cf with the following line of text, substituting the email@example.com and password placeholders with your own:

smtp.emailsrvr.com:587 email@example.com:password

Next, use the postmap tool to create a lookup table from the sasl_passwd file:

sudo postmap /etc/postfix/sasl_passwd

This will create the file sasl_passwd.db

Step 3: Restart postfix

sudo postfix reload

You might see some warnings, but they should be benign. We don’t need a full mail system, just the ability to send smtp emails. Move on to the next step to test your configuration!

Step 4: Send a test email

The following command sends a test email to your-email@example.com. Give it a shot by substituting your email address:

echo "testing testing test email" | mail -s test your-email@example.com

Note this command does not produce any output when it runs successfully.

Next, check for the test email in your inbox (if its not there, be sure to also check the spam folder). You may want to add your notification email address to your recipient inbox’s address book to ensure better deliverability.

Troubleshooting tips

If you need to troubleshoot, it can be helpful to check the mail queue:

sudo mailq 

You can clear all queued emails with the following command:

sudo postsuper -d ALL

On OSX, postfix logged to the /var/log/mail.log file. On MacOS, this log file doesn’t exist.

For troubleshooting with MacOS, you can view the log in real-time with the command:

sudo log stream --predicate  '(process == "smtpd") || (process == "smtp")' --info

You could have this running in one Terminal tab, and then try to send a test email in another tab, to see what messages you receive.

Acknowledgements

Thanks to this article from <developerfiles.com> which has instructions for using gmail, as well as some commenters on various forums that provided tips to adapt SMTP settings for Rackspace. Rackspace’s documentation was also helpful in realizing a working configuration.

Stripping leading and trailing slashes from paths in Ansible

This post covers how to use ansible’s regex_replace filter to strip leading and/or trailing slashes from file paths and URL fragments.

The final example demonstrates how to generate a valid filename from a file or URL path by removing leading and trailing slashes, and replacing any remaining slashes with underscores. This could be useful for a variety of applications from backup scripts to web scraping.

Stripping slashes

Regexes and jinja2 expressions in ansible can be a pain in the ass, especially when it comes to escaping the right thing.

The key to the following examples is a double-escape of the forward slash character. They have been tested on ansible v.2.3.1.0.

Remove leading slashes

{{ variable_name | regex_replace('^\\/', '') }}

Remove trailing slashes

{{ variable_name | regex_replace('\\/$', '') }}

Stripping both leading and trailing slashes

This example makes use of the | (OR) to combine the previous two examples into one regex:

{{ variable_name | regex_replace('^\\/|\\/$', '') 

Here’s a quick debug task that demonstrates the above in action:

- debug:
    msg: "Look Ma! No leading or trailing slashes! -- {{ item | regex_replace('^\\/|\\/$', '') }}"
  with_items:
    - "/home/and/lots/of/folders/trailing_and_leading/"
    - "no_leading/but/there/is/trailing/"
    - "/leading/but/no_trailing"
    - "no_leading/and/no_trailing"

Creating a valid filename from a path

To create a valid filename from a path, we need to remove leading and trailing slashes, then replace any remaining slashes with underscores. This sort of thing can be useful for naming backup files, data obtained from URL scraping, etc.

This is accomplished by adding a second regex_replace to the previous example that replaces all slashes with underscores, e.g. regex_replace('\\/', '_').

The combined regex that creates a valid filename from given a path:

{{ variable_name | regex_replace('^\\/|\\/$', '') | regex_replace('\\/', '_') }}

If variable_name held a value like “/var/www” or “/var/www/” it would result in: “var_www”.

In case you want to replace the slashes in the path with a character other than an underscore, you can adjust the examples to use any valid filename character instead of a slash (e.g. the ‘^’ character).

Working around a regex_replace bug in the shell module

In my working version of ansible (2.3.x) the regex_replace filter is ignored (!) when it is applied to variables in a tasks using the shell module. The value of unfiltered variables are substituted into the shell command fine, and using any other filter works fine too.

Workaround: employ the set_fact module to build a new fact (variable) based on the original variable, applying the regex_replace filter here as required. Reference the new fact in the shell module to take advantage of the pre-filtered values.

In the following example, assume that the hypothetical {{ list_of_paths }} variable contains a list of strings containing file/dir/URL paths. The set_fact module builds the new {{ paths }} fact such that it contains a “pi” item corresponding to every item in the original list. Each of these items has a “stripped” property containing the filtered value and a “path” property containing the original unfiltered value.

- name: workaround example
  set_fact: 
    paths: "{{ paths|default([]) + [{'pi': {'path': item, 'stripped': item|regex_replace('^\\/|\\/$', '')|regex_replace('\\/', '_')}}] }}"
      with_items: "{{ list_of_paths }}"

When looping over {{ paths }} in a shell task (e.g. via with_items), the filtered slash-free values for items can be referenced via {{ item.stripped }}. The original unfiltered path can be referenced via {{ item.path }}.

Install and secure phpmyadmin: require access via an SSH tunnel

Phpmyadmin is a popular database administration and query tool for MySQL and MariaDB. This guide covers how to install it, plus add an additional layer of security that only permits access via an encrypted ssh tunnel (also known as ssh port forwarding). This guide covers Debian and Ubuntu systems running apache2.

This particular approach is easy to automate and improves on other examples that I’ve come across online. Its the only tutorial that I know of that does things The Debian Way, in this case meaning that phpmyadmin’s customized configuration is applied without editing any of its packaged-managed conf files. This ensures that future package updates do not risk overwriting your security-minded configuration.

Why extra security measures are recommended for phpmyadmin

Phpmyadmin is a popular choice for attackers because it is widely deployed, easy to find, easy to attack, and it provides convenient access to potentially highly sensitive and/or valuable data.

A default package install of phpmyadmin on a public-facing web server is wide open to the Internet. It is often easily found with the default directory pattern “/phpmyadmin”.

A quick look at almost any website’s logs will reveal an endless barrage of attempts to locate phpmyadmin, probing a wide range of likely URL patterns and alternate names. In cases where phpmyadmin is discovered, its practically certain the logs will subsequently reveal attackers attempting to brute force and/or exploit their way in.

Another significant concern with phpmyadmin is that it does not come with ssl configured out-of-the-box. Login credentials and any data that’s accessed can be picked up by anyone that cares to listen in anywhere between your computer and the database server. This is especially a concern for anyone who accesses phpmyadmin via an untrusted network, such as public wifi.

Part 1: install phpmyadmin

First, make sure that you have mysql or mariadb installed and configured, and have taken basic steps to secure the setup. The mysql_secure_installation command is bundled with the mysql + mariadb packages and helps cover the basics.

Install the phpmyadmin package:

sudo apt-get update
sudo apt-get install phpmyadmin

The configuration script will prompt with a number of questions. When asked if you would like to automatically configure phpmyadmin for your web server, choose ‘apache2’.

Once the process is complete, you should be able to access phpmyadmin in a web browser by appending “/phpmyadmin” to your server’s URL, e.g.:

http://your-server.com/phpmyadmin

Part 2: Secure phpmyadmin

The following assumes that both apache2 and phpmyadmin were installed via the package manager.

Step 1: create a new conf file

Open your favourite editor and create a file called phpmyadmin-secure.conf in /etc/apache2/conf-available, e.g.:

sudo nano /etc/apache2/conf-available/phpmyadmin-secure.conf

For apache2.4, add the following content:

# ensure that phpmyadmin can only be accessed via localhost 
# remote users must use an ssh tunnel for access
<Directory /usr/share/phpmyadmin>
    Require local
</Directory>

You can double check your apache version with the command: apache2 -v.

For the older apache 2.2.x you need to use an older syntax inside the <Directory> block. If you do not need IPv6 support, delete the last directive in the following block (the entire line containing ::1):

<Directory /usr/share/phpmyadmin>
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1
    Allow from ::1
</Directory>

These directives tell apache that anything in the package install location /usr/share/phpmyadmin should only be accessible by localhost (127.0.0.1). Every other IP address, i.e. anything else on the internet, should be forbidden.

Save and exit (control+x in nano).

Step 2: Enable the conf file

Use a2enconf to enable the conf file:

cd /etc/apache2/conf-available
sudo a2enconf phpmyadmin-secure.conf

This command will create a symlink in apache’s conf-enabled/ directory that points to the conf file in the conf-available/ directory.

Note that the a2disconf command also exists and is invoked in a similar manner to a2enconf. It performs the reverse operation and disables the conf file by removing it from conf-enabled/ after it performs some checks.

Step 3: Reload apache and test

Apache must be reloaded or restarted for the updated configuration to take effect:

sudo service apache2 reload

After reloading/restarting apache, try to access phpmyadmin at its URL, e.g.:

http://your-server.com/phpmyadmin

You should see an error 403 / forbidden.

Tip: when making changes to apache’s conf files, you can check that the syntax is valid prior to reloading/restarting the server with the command apachectl configtest. Its a good idea to test your configuration before applying it because Apache will refuse to start if it has an invalid config, taking your site(s) offline until you can start it with a valid one.

Tip: if you want to hide the fact that phpmyadmin is even installed on your server at all, you can use apache’s mod_rewrite and/or RedirectMatch directives to produce a 404 ‘not found’ instead of a 403 response code.

Part 3: Access phpmyadmin via an ssh tunnel (ssh port forwarding)

To access your phpmyadmin install, connect to your server with ssh, using its tunnel options.

On your local machine, pick an arbitrary unused port and open a terminal (or an ssh client such as puTTy if your machine is a Windows PC). I use 5050 in the example below, but you can replace this with any available port number. Connect to the remote server via ssh:

ssh -L 5050:localhost:80 user@host.example.com

The -L option is key to establishing the tunnel. It tells ssh to forward any connections made to your local machine’s port 5050 to the remote host’s port 80 for as long as the ssh session is active.

Any web server (apache in our case) listening on port 80 will perceive that connections via the tunnel are coming from localhost and not from over a network / the internet.

Once you have a successful connection the server, visit the following URL in your browser to confirm you can access phpmyadmin:

http://localhost:5050/phpmyadmin

All traffic between you and the remote host (including credentials and data) will be encrypted via ssh for as long as your session remains open.

All done!

You can now use phpmyadmin with the knowledge that this powerful and useful but also risky tool is not nearly as exposed to the open internet as it would be with a default install, and that you have taken a small but significant step in the endless battle to protect your credentials and data.

Troubleshooting tips

  • If you can’t access phpmyadmin, one possible gotcha to consider is if your website or app has mod_rewrite redirection rules in place (e.g. to force users to use https, to redirect certain url patterns, etc). These rules may be interfering when you try to access phpmyadmin via the tunnel. If you need to carve out an exception for localhost, but not other hosts, you could add a Rewrite Condition such as: RewriteCond %{HTTP_HOST} !^localhost to exempt it from your other rewrite conditions.

Using rbenv to support multiple versions of ruby on MacOS

This guide covers using the brew package manager to install rbenv on MacOS to enable you to install multiple versions of ruby and switch between them.

Apple bundles ruby with its operating system but the version is generally outdated and often unsuitable for developers working on their latest project. Later versions of Mac OSX through to MacOS Sierra include Ruby 2.0. Earlier OSX variants, specifically the “cats” (Mountain Lion, Lion, Snow Leopard), ship with Ruby 1.8.7. The current stable version at the time of this post is 2.4.x.

rvm is another tool that, similar to rbenv, can be used to switch between multiple versions of ruby. Overall, rbenv has a lighter touch on the MacOS system, and so is the preferred choice for this guide.

Prerequisites

  • brew

Steps

Install rbenv and ruby-build with brew:

brew install rbenv ruby-build

Add rbenv to your bash profile so that it loads every time you open the Terminal app:

echo 'if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi' >> ~/.bash_profile
source ~/.bash_profile

Thanks to: https://gorails.com/setup/osx/10.12-sierra for the snippet.

Next, close and re-open your Terminal app to ensure that rbenv is initialized from your updated bash profile.

Now you can install any ruby version that you like. The following example installs v2.4.0.

rbenv install 2.4.0

rbenv gives you options when choosing which version of ruby you’d like to use:

rbenv shell 2.4.0 temporarily changes the ruby version in your current shell and forces its use over other versions that might be specified by a .ruby-version file in your current working directory. This command works by setting the RBENV_VERSION environment variable in your session. To remove it, run unset RBENV_VERSION.

rbenv local 2.4.0 creates a .ruby-version file in your current directory such that any time ruby is invoked from that directory (or a subtree), rbenv will select the version of ruby that you specified. This is useful for project folders, and you can check the .ruby-version file into your project’s revision control.

rbenv global 2.4.0 changes your default ruby to the specified version in all cases where a version is not specified (e.g. due to a .ruby-version file or if the RBENV_VERSION environment variable is set). Your current Terminal session and any other sessions will be affected.

You can confirm the ruby version in play with the command:

ruby -v

If you don’t see your desired ruby version number in the command’s output (e.g. 2.4.0) and find an older one instead (e.g. 2.0.0 the MacOS default), be sure to close the Terminal and start fresh with a completely terminal window/session, and take note if your working folder has a .ruby-version file set and what the value of the RBENV_VERSION environment variable is (echo $RBENV_VERSION).

Once you are working in a newer ruby environment, you may need to check the gems and other libraries in a project that was started with an earlier version, to ensure they are compatible or need an update.

Using s3cmd to access S3 buckets from EC2 instances with IAM Role authentication

s3cmd is an open-source command line tool for uploading, retrieving and managing data in Amazon S3 and other providers that implement the S3 protocol (e.g. Google Cloud Storage, DreamHost DreamObjects, etc). It is popular tool with a variety of applications, including backup scripts.

This post covers using s3cmd within an EC2 instance, with authentication to S3 managed via IAM Roles (IAM = Identity and Access Management).

If your project doesn’t have dependencies on scripts and/or pre-existing code that require s3cmd, and you don’t mind locking your project deeper into AWS’ ecosystem, a good alternative is Amazon’s own AWS CLI.

Configuring an IAM Role

When an EC2 instance is launched with an associated IAM role, access keys and secret keys do not need to be stored in config files on the instance itself.

After such an instance is launched, AWS-aware services and scripts on the it can access AWS resources subject to the permissions defined in the IAM role’s associated IAM role policy. Authentication is seamlessly handled via the AWS SDK.

s3cmd versions 1.5.0-alpha2 and above support IAM roles and authentication via the AWS SDK.

IAM roles

Log in to your AWS account and access the AWS IAM Dashboard (also referred to as the IAM Management Console) to manage role-based permissions.

Create an IAM role for your instances if you haven’t already.

IAM role policy

In the IAM Management Console, ensure that your IAM role has an attached policy that provides access to any S3 resource(s) that you want your instance(s) to access via s3cmd. It’s advisable to ensure that role policies are restricted to the minimum amount of scope possible.

Within the IAM Management Console:

  • Choose Roles and select the role that your EC2 instances belong to (or will belong to)
  • Review your existing policies to confirm what your instances can currently access
  • If necessary, use the Create Role Policy button to add a policy with the Policy Generator

For example, suppose you had a an s3 bucket called example-log-bucket and you wanted to configure your EC2 instances to send access logs to it.

Following the order of the Policy Generator’s input fields, you’d create: an Allow policy for the Amazon S3 service regarding the actions s3:ListBucket, s3:PutObject, and s3:PutObjectAcl, and apply them to the ARN arn:aws:s3:::example-log-bucket.

To learn more about how to specify ARNs that identify your various AWS services, see the Amazon docs: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3

Note that you can use wildcards in ARNs when defining a Role Policy.

Installing and configuring s3cmd on the EC2 instance(s)

s3cmd versions 1.5.0-alpha2 and above support IAM roles.

In earlier versions, an access key and secret key had to be specified in a .s3cfg file in the home directory of the user running the s3cmd command, or in environment variables.

Option A: Install s3cmd with the Ubuntu/Debian apt package manager

If you’re running Ubuntu 16.04 or later (test box at the time of writing) the package version is sufficiently up-to-date (>1.5.0). Install s3cmd with apt-get:

sudo apt-get install s3cmd

Option B: Install s3cmd with the python pip package manager

Pip is an alternative to Debian/Ubuntu apt-get, especially if you are using a different distro or like to use python/pip.

sudo pip install s3cmd

Check the version number with s3cmd --version. If you discover that its <1.5.0, you can specify a particular version for pip to install as follows, substituting in your desired version number (e.g. 1.5.0-alpha3) in place of VERSION_NUMBER below:

pip install s3cmd==VERSION_NUMBER

Configure s3cmd on the EC2 instance(s)

Once an EC2 instance is launched with a suitable IAM Role and associated IAM Role Policy, its easy to get started with s3cmd. The following example is set in the context of the currently logged in user.

Open/create the file ~/.s3cfg using a text editor:

nano ~/.s3cfg

Populate the file with the following content. Rather than filling in hardcoded credentials, leave each of the fields blank (use the example as is; do not specify any values):

[default]
access_key =
secret_key = 
security_token =

Save the file and exit the editor.

Later versions of s3cmd that support IAM Roles will auto-magically detect the appropriate authentication values.

s3cmd will now be able to interact with any AWS resource(s) that it is permitted to access, subject to s3cmd’s capabilities and the IAM Role’s active Role Policies.

Use s3cmd

s3cmd has over 60 command line options. Check out the documentation:

http://s3tools.org/usage

How to install Ansible on MacOS using pip

Ansible is a powerful devops/automation tool that helps you control fleets of servers and their applications with ease.

This post covers installing ansible on MacOS using the installation method recommended by the project: pip, the python package manager. There are a few other ways to install ansible on MacOS, including with the brew package manager.

This post assumes that you are interested in installing ansible only for the current user only. If you require a system-wide/multi-user installation, skip to Step 8 to learn more about the MacOS System Integrity Protection (SIP) feature before proceeding.

Installing Ansible with pip

You may be able to skip steps 1, 2, and/or 3 if you already have XCode, XCode Command Line Tools, and/or pip installed on your system.

Step 1: install XCode

Most developers already have XCode installed. If you’re not sure, you can look for it in the Applications folder on your Mac.

XCode is available for free on the App Store. It can be helpful to run XCode at least once after installing it to help it complete its installation process, and get you to agree to any license terms.

The following steps will fail if you have not agreed to Xcode’s license terms.

Step 2: install XCode command line tools

The XCode command line tools include compilers and other tools necessary to proceed. In the MacOS Terminal app, run the command:

xcode-select --install

If this is a fresh install, you will be prompted to agree to Apple’s terms for the command line tools. After you consent, they will download and install.

If you already have the command-line tools installed. The script will let you know and exit.

Once the XCode command line tools are installed, you can use development tools like the gcc compiler. For example, you should be able to run the following command and successfully receive gcc’s version information as output:

gcc --version

If everything looks good and the above command does not throw an error, proceed to the next step.

Step 3: install pip

Pip is the python package manager. It is conceptually similar to other well-known package management tools such as npm (node/javascript) or apt (debian/ubuntu linux distributions).

MacOS comes pre-installed with python, but not pip. To install it, input the following into the Terminal app:

sudo easy_install pip

There is no harm to running this command in cases where pip is already installed.

Step 4: install ansible

To install ansible system-wide, run the following command:

sudo pip install ansible

If you prefer a local install within your user account, you can remove the ‘sudo’ and add the --user flag to the above.

After the installation process has completed, you can verify the install was a success by executing a basic command like: ansible --version. It should run without error.

Step 5: create /etc/ansible

Ansible looks for default configuration files in /etc/ansible. This directory is unlikely to exist on your machine. The following command will create this folder if it doesn’t already exist:

sudo mkdir -p /etc/ansible

Step 6: install optional/helpful packages

It can be helpful to install the python passlib library because MacOS doesn’t use the same types of password hashes as linux does, and you are likely going to want to manage linux machines with ansible. Passlib provides a common and consistent hash generator across operating systems.

To install passlib globally on your Mac:

sudo pip install passlib

If you installed ansible locally, you can use the command pip install --user passlib instead.

Step 7: test out your installation

Per above, if the install was successful, you can now use the ansible command from the Terminal:

ansible --version

A “hello world” of sorts for ansible is pinging yourself:

ansible localhost -m ping

You can also try running an arbitrary command on yourself (such as uname -a or whereis bash):

ansible localhost -a 'uname -a'

To get ansible to detect facts about your machine, try the following. Note the filter argument: it restricts what is returned so you don’t get BLASTED with facts.

By default ansible gathers almost everything there is to know about its target machines and populates them into variables that you can then use in your tasks, roles, and plays. As you learn ansible, you’ll discover this is incredibly useful. Ansible users can customize this behaviour and optionally turn it off to realize performance gains when it isn’t required. For now though, you’re just getting started:

ansible localhost -m setup -a 'filter=ansible_distribution'

Step 8: upgrade ansible

It can be a good idea to upgrade ansible right off the bat to ensure that everything is current, and to understand how the MacOS System Integrity Protection (SIP) feature impacts your installation:

sudo pip install ansible --upgrade 

If this command partially runs and then produces an error, it is because of the SIP feature, originally introduced in OSX El Capitan. You can work around that by adding the --user python parameter to the above command. With this argument, upgrades will be installed for the current user only vs. system-wide, and the upgrade will succeed. The revised command:

sudo pip install ansible --upgrade --user python

Understanding MacOS’ System Integrity Protection (SIP) Feature

SIP was introduced in OSX El Capitan and prevents certain system files, such as those associated with MacOS-bundled python, from being clobbered by anyone — including root.

To avoid SIP-related issues on a system-wide install, you may want to consider using a different python than the one bundled with MacOS. Using a python virtualenv is one option. Another option for multi-user systems is to install user-specific copies of python + pip + ansible within each appropriate user account (e.g. via brew package manager for MacOS).

A more thorough discussion of issues regarding the use of pip on MacOS systems with SIP can be found here:

Step 9: get started!

Congratulations! You have installed ansible on MacOS!

If you’re new to ansible, check out the documentation at http://docs.ansible.com/ansible/latest/index.html to get started!