How to run E2E Tests with docker-compose

This guide covers using docker-compose to spin up your application, run E2E tests, and then exit with the results.

The TL;DR is:

  • docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app

For the sake of an example, we’ll be testing a hypothetical API written in NodeJS that uses a Postgres database and a Redis instance. We’ll assume tests are run via jest. You can substitute any stack!

To begin, ensure that you have recent versions of docker and docker-compose installed on your machine.

Dockerfile

Ensure you have a Dockerfile in your project’s folder that specifies how to build an image for your app.

The following example is for a web API written in NodeJS.

FROM node:14.4-alpine As example

# install build dependencies
RUN apk update && apk upgrade
RUN apk add python3 g++ make

# install packages for sending mail (msmtp = sendmail for alpine)
RUN apk add msmtp
RUN ln -sf /usr/bin/msmtp /usr/sbin/sendmail

# make target directory for assigning permissions
RUN mkdir -p /usr/src/app/node_modules
RUN chown -R node:node /usr/src/app

# use target directory
WORKDIR /usr/src/app

# set user
USER node

# copy package*.json separately to prevent re-running npm install with every code change
COPY --chown=node:node package*.json ./
RUN npm install

# copy the project code (e.g. consider: --only=production)
COPY --chown=node:node . .

# expose port 3500
EXPOSE 3500

docker-compose

Create a docker-compose.e2e.yml file.

The following example creates a service called app that runs in a container named example.

Note the command property. This should specify the command that will run your tests inside the container. In our example, this is yarn test:e2e.

version: '3.8'

services:
  app:
    container_name: example
    build:
      context: .
      target: example # only build this part of the Dockerfile (see: '... As example' )
    volumes:
      - .:/usr/src/app
      - /usr/src/app/node_modules # 'hack' prevents node_modules/ in the container from being overridden
    working_dir: /usr/src/app
    command: yarn test:e2e
    environment:
      PORT: 3500
      NODE_ENV: test
      DB_HOSTNAME: postgres
      DB_PORT: 5432
      DB_NAME: example
      DB_USERNAME: postgres
      DB_PASSWORD: postgres
      REDIS_HOSTNAME: redis
      REDIS_PORT: 6379
    networks:
      - webnet
    depends_on:
      - redis
      - postgres

  redis:
    container_name: redis
    image: redis:5
    networks:
      - webnet

  postgres:
    container_name: postgres
    image: postgres:12
    networks:
      - webnet
    environment:
      POSTGRES_DB: example
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      PG_DATA: /var/lib/postgresql/data
    volumes:
      # - ./seed.db.sql:/docker-entrypoint-initdb.d/db.sql <- run only once when the pgdata volume is first created (when run via docker-compose)
      - pgdata:/var/lib/postgresql/data # or specify a local folder like ./docker-volumes/pgdata:/var/lib/postgresql/data

networks:
  webnet:

volumes:
  pgdata:
  logs:

Note how each service shares the same network so they can communicate with each other.

Tip: you can use .env files and reference variables from them in a docker-compose.yml file as follows: ${VARIABLE_NAME}.

If you wish to specify a particular .env file in your docker-compose.yml file:

env_file:
  - .env

Run E2E Tests

From your project folder, you can run the following command to run your tests:

docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app

The -f flag specifies a custom configuration file for docker-compose. If this is not specified, docker-compose will look for docker-compose.yml by default.

The up command tells docker-compose to bring the services and containers up.

The --abort-on-container-exit and --exit-code-from flags are an important combination.

The first flag shuts things down when our test run is complete, and the second flag will use the exit code from the specified service (in our case the one named app) as the exit code from the overall docker-compose command.

This is a good setup if you have scripts that run tests, or if you have a continuous integration pipeline that automatically runs tests and requires a pass/fail.

Test runners such as jest will generally exit with code 0 (success) if all tests pass, and exit with a non-zero code (failure) if any tests fail.

package.json

If your project uses npm, yarn or their ilk, you can specify commands to run tests in the scripts section.

Our docker-compose.yml file requires the app service to run the command yarn test:e2e. In our hypothetical example app, this is specified as follows:

"test:e2e": "NODE_ENV=test jest --config ./test/jest-e2e.json",

Of course, you will need to specify the command that initiates running E2E tests that’s particular to your project and environment.

To spin up your application via docker-compose and run tests from your workstation (or CI environment, etc), add the following script:

"test:e2e:docker": "docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app"

You could then run via npm run test:e2e:docker or yarn test:e2e:docker

If you are using a different package/dependency management solution, you can specify your test-related scripts there. You also have the option to define shell/bash scripts that can run your tests.

How to use aws-sdk for NodeJS with AWS Translate

This post covers using the aws-sdk for NodeJS with AWS Translate.

The code examples are written in ES and transpiled with Babel.

Install the AWS SDK

First, install the aws-sdk package in your project using your favourite package manager:

yarn add aws-sdk
# OR
npm i aws-sdk

Ensure There’s a Working AWS Profile

Ensure that you have an AWS profile and configuration properly setup for your user. An AWS Profile is typically stored inside the ~/.aws folder inside your home directory.

Suppose you have a profile named firxworx. An example entry of a useful entry in ~/.aws/config for that profile is:

[profile firxworx]
region = ca-central-1
output = json

A corresponding entry in the ~/.aws/credentials file that specifies credentials for the example firxworx profile looks like this:

[firxworx]
aws_access_key_id=ABCDXXXX
aws_secret_access_key=ABCDXXXX

Refer to the AWS Docs if you need to create a profile and obtain an Access Key ID and Secret Access Key.

Write Your Code

Start by importing the aws-sdk package:

import AWS from 'aws-sdk'

Next, configure AWS by specifying which profile’s credentials to use:

const credentials = new AWS.SharedIniFileCredentials({ profile: 'firxworx' })
AWS.config.credentials = credentials

Specify any other config options. The following line locks AWS to the most current API version (at the time of writing):

AWS.config.apiVersions = {
  translate: '2017-07-01',
}

Reference the AWS Translate homepage and take note of which regions AWS Translate is currently available in. If you need to specify a region that’s different than the default listed in your AWS profile, or you wish for your code to be explicit about which region it’s using, add the following line. Change the region to the valid region that you would like to use:

AWS.config.update({
  region: 'ca-central-1'
})

If you are using any Custom Terminologies, be sure to define them in the same region that you are about to use for AWS Translate. Custom Terminologies are lists of translation overrides that can be uploaded into the AWS Console. They are useful for ensuring that brand names, terms of art, trademarks, etc are translated correctly. Custom Terminology definitions are only available within the region that they were created and saved in.

Next, create an instance of AWS Translate:

const awsTranslate = new AWS.Translate()

At this point everything is setup to write a function that can translate text.

The following implements an async function called awsTranslate(). The function’s params include specifying a hypothetical custom terminology named example-custom-terminology-v1. Do not specify any value in the TerminologyNames array if you are not using any custom terminologies.

A key insight here is the .promise() method in the line containing awsTranslate.translateText(params).promise() which causes the API to return a promise.

async function asyncTranslate(langFrom, langTo, text) {
  const params = {
    SourceLanguageCode: langFrom,
    TargetLanguageCode: langTo,
    Text: text,
    TerminologyNames: [
      'example-custom-terminology-v1'
    ]
  }

  try {
    const translation = await awsTranslate.translateText(params).promise()
    return translation.TranslatedText
  } catch (err) {
    console.log(err, err.stack)
  }
}

The langFrom and langTo must be language codes as understood by AWS Translate. Refer to the docs for a current list of supported language codes: https://docs.aws.amazon.com/translate/latest/dg/what-is.html.

If you had a hypothetical index.js entry point for your NodeJS application and wanted to use the above function, an example invocation could be:

(async () => {

  const translation = await asyncTranslate('en', 'fr', 'Hello World')
  console.log(translation)

})()

Editing WordPress’ wp-config.php with wp-cli and adding variables with sed

The WordPress CLI (command line interface) is a huge step for enabling developers and devops/sysadmin folks manage their WordPress installations. It’s awesome for writing scripts to automate key tasks.

This post covers editing the WordPress configuration file wp-config.php with the WP-CLI’s wp config command, as well as using the sed command to address a key missing feature of WP-CLI: the ability to add new config variables.

There are plenty of reasons you might want to edit wp-config.php via a script or directly via the command line. For example, developers might appreciate a bash script that sets WP_DEBUG to true, and a devops person might want to create an automated deploy process that ensures key SMTP settings are in place.

The rest of this post will assume your WP-CLI command can be invoked with wp. Depending on how you installed it, the command might be available to you as wp-cli or wp-cli.phar. If you are running wp-cli’s phar file directly, substitute php wp-cli.phar in place of wp in the examples.

Editing config variables with wp-cli’s config command

WP-CLI supports modifying config variables in wp-config.php via wp config.

This is a great feature, albeit with the noted catch that wp config only works for a given variable if that variable is already defined in wp-config.php. I’ll show you how to work around that and add variables with sed in the next section.

The following example uses sudo to run wp config as the _www web server user, the default web server user on MacOS. On Ubuntu and many other linux distros, this user is likely www-data:

sudo -u _www wp config set FS_METHOD 'direct'
sudo -u _www wp config set DISABLE_WP_CRON true
sudo -u _www wp config set WP_DEBUG true
sudo -u _www wp config set WP_DEBUG_LOG true

These are some of the most popular config options that developers and admins want to modify.

Adding config variables to wp-config.php using sed

There are a number of command-line utilities on Linux and Unix-like systems that can edit text files. One of the most popular is sed, the quintessential stream editor. Unix admins have been working with streams forever, long before NodeJS made it cool :).

sed is pre-installed on most systems and can be used directly in the Terminal or inside a bash (or other shell) script.

The following example uses sed to add config variables to wp-config.php right before the well-known “That’s all, stop editing!” comment line found in the file.

This snippet works on MacOS and elsewhere. MacOS and OSX, as well as their related family in the BSD/unix world, generally bundle a classic POSIX-compliant version of the sed command which is more limited vs. the more common and more popular GNU sed that ships with major linux distributions like Ubuntu. If you’re on linux, delete the double quotes '' immediately following the -i flag to be compatible with GNU sed.

Editing wp-config.php with sed:

sed -i '' '/\/\* That.s all, stop editing! Happy blogging. \*\// i\
// FX_SCRIPT FS_METHOD \
define( "FS_METHOD", "direct" ); \
\
// FX_SCRIPT WP_DEBUG \
define( "WP_DEBUG", true ); \
define( "WP_DEBUG_LOG", true ); \
\
// FX_SCRIPT DISABLE_WP_CRON \
define( "DISABLE_WP_CRON", true ); \
\
' wp-config.php

The -i option tells sed to edit the file in-place i.e. modify the file directly. Otherwise, sed lives up to its name and streams output to stdout.

The MacOS version of sed requires a backup file to be specified as the first argument whenever the -i option is used. You can pass empty quotes '' to specify no backup file as demonstrated in the example.

The linux version of sed does not require a backup filename to be specified. You can simply delete the '' arguments as noted above.

The way that single and double quotes are used is very important for getting this command to work. Getting them right is one of the trickiest parts about using sed.

Also note how backslashes are used at the end of each line. This is required to make the command portable and universal: classic sed (BSD/Unix/MacOS) does not recognize \n as a placeholder for the newline character while GNU sed (linux) does. The backslashes enable a trick to use actual newlines instead of placeholders.

Finally, my example adds comment lines that start with // FX_SCRIPT before each change (get it? FX = firxworx, the name of this blog!). I do this to make it easy to search with grep and/or visually look for changes in wp-config.php files that were made by my scripts. You may wish to follow a similar practice. This makes it easier to write other scripts that might find and comment out or delete these entries at a later time.

Installing gulp4 with babel to support an ES6 gulpfile

This guide covers installing gulp4 with babel to support ES6 syntax in your gulpfile.

Gulp is a task automation tool that has emerged as one of the standard build tools to automate the web development workflow. Babel is a compiler/transpiler that enables developers to use next-generation ECMAScript syntax (ES6 and beyond) instead of older JavaScript (ES5) syntax.

Gulp4 and ES6+ work together swimmingly to help you write cleaner, easier-to-read, and more maintainable gulpfile’s.

Installing gulp4

At the time of writing, the default gulp package installs gulp 3.x. The following will install and configure gulp4.

Gulp has two key parts: gulp and the gulp-cli command line tools. The idea is that gulp-cli should be installed globally on a developer’s machine while gulp should be installed locally on a per-project basis. This helps ensure compatibility with different versions of gulp that will inevitably arise when maintaining projects of different vintages.

To use gulp4, cli version 2.0 or greater is required. Check the version on your system with:

gulp -v

If the command returns a command not found error, then you probably don’t have gulp installed at all (or at least don’t have it available in your PATH).

If the command outputs a version lower than 2.0, you may need to uninstall any globally-installed gulp (and/or gulp-cli) and then install the current version gulp-cli before proceeding.

To install gulp-cli globally, run ONE of the following commands, depending on your preference of package manager. npm is the classic node package management tool and yarn is a newer tool developed by Facebook that addresses certain shortcomings with npm.

yarn global add gulp-cli
# OR
npm install gulp-cli -g

Test the install by running gulp -v and ensuring the version output is greater than 2.0. Next, install the gulp@next package. The @next part specifies the next-generation gulp4.

Assuming you have already run npm init or yarn init and have a package.json file, execute the following command in your project’s root directory:

yarn add gulp@next --dev
npm install gulp@next --save-dev

Installing babel

yarn add @babel/core --dev
yarn add @babel/preset-env --dev
yarn add @babel/register --dev
# OR 
npm install @babel/core --save-dev
npm install @babel/preset-env --save-dev
npm install @babel/register --save-dev

Next, create a .babelrc file in your project’s root folder and specify the current version of node as the target:

{
  "presets": [
    ["@babel/preset-env", {
      "targets": {
        "node": "current"
      }
    }]
  ]
}

Create your gulpfile

Create your gulpfile with the filename gulpfile.babel.js. The babel.js suffix ensures that babel will be used to process the file.

The following example demonstrates a few ES6+ features: optional semicolons, import statements, and the “fat arrow” syntax for defining functions:

'use strict'

import gulp from 'gulp'

gulp.task('task-name', () => {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

Gulp4 features a new task execution system that introduces the functions gulp.series() and gulp.parallel() that can execute gulp tasks in either series (one-after-another) or parallel (at the same time). This makes a lot of workflows much easier to define vs. previous versions!

Another nice feature is that gulp4 supports returning a child process to signal task completion. This makes it cleaner to execute commands within a gulp task, which can help with build and deployment related tasks.

The following example defines a default build task that runs two functions/tasks in series using gulp.series(). The build task is defined using the ES6 const keyword and exported as the default function/task for the gulpfile. The example doSomething() and doAnotherThing() functions/tasks are also exported.

'use strict'

import gulp from 'gulp'

export function doSomething {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

export function doAnotherThing {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

const build = gulp.series(doSomething, doAnotherThing)

export default build

Set up MacOS’ built-in Apache + PHP as a LAMP/WordPress Dev Environment

This guide covers configuring MacOS High Sierra, Mojave, and beyond as a local development server for LAMP-stack projects. This is a high-performance option for running projects locally on a Mac and is useful for PHP and WordPress development.

At the end of this guide, you will be able to create new folders within a special virtual hosts directory such as ‘example.com/’ with the sub-folder ‘public_html/’ and the ‘http://example.com.test’ will automatically be available with apache serving files out of the ‘public_html/’ folder.

Apple already bundles many of the necessary pieces with MacOS including apache and PHP. This guide covers configuring them to support WordPress, as well as installing other dependencies such as mysql/mariadb, and useful tools like sequel-pro and wp-cli.

Alternatives

Self-contained solutions for running a LAMP environment on a Mac can be found in products like MAMP and the free XAMPP. They double up what’s already on your Mac but do have some features that you might find appealing.

Virtual Machines are another option. Tools such as Vagrant are great and extremely useful for more complex projects where customized server configuration(s) are required. However, spinning up an entire virtual server with its own CPU, RAM and disk allocation simply to run WordPress is a huge tax on system resources. It’s often a better idea to reserve VM tools for better-suited applications.

A lighter-weight alternative to full VM’s is container virtualization such as with Docker. This brings its own configuration and deployment challenges that may introduce unnecessary effort if Docker doesn’t play a greater role in a given project.

Pre-checks

Confirm your system’s apache version with the command apachectl -v, and its PHP version with the command php -v.

MacOS Sierra and beyond should be running apache 2.4+ and PHP 7+.

Apache configuration

Apple’s default configuration requires several changes to create a suitable development environment.

Backup httpd.conf

First, make a backup copy of apache’s httpd.conf file:

sudo cp /etc/apache2/httpd.conf /etc/apache2/httpd.conf.pre-dev-env

Apache httpd.conf configuration

Open httpd.conf using a text editor such as nano:

sudo nano /etc/apache2/httpd.conf

Each of the following lines are found at different locations within the file. Find each one of them and ensure they are uncommented — delete any preceding # character so they start with a letter instead:

#LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so
#LoadModule rewrite_module libexec/apache2/mod_rewrite.so
#LoadModule php7_module libexec/apache2/libphp7.so
#Include /private/etc/apache2/extra/httpd-vhosts.conf

If you’re using the nano editor, its “where” Control+W feature can help you locate each line. It works similar to the “find” command (Control+F) found in most web browsers and word processors.

When you’re done, use Control+O to save (output) your changes and Control+X to exit nano.

The above changes have:

  • enabled the module mod_vhost_alias to enable dynamic virtual hosts;
  • enabled the module mod_rewrite which WordPress uses to rewrite URL paths;
  • enabled PHP7 support so WordPress’ php scripts can execute; and
  • included an extra conf file where we will define a dynamic apache virtualhost for your projects.

Apache dynamic virtual host configuration

In the previous step we referenced a httpd-vhosts.conf file in httpd.conf.

Create a backup of the original file:

sudo cp /private/etc/apache2/extra/httpd-vhosts.conf /private/etc/apache2/extra/httpd-vhosts.conf.pre-dev-env

Now edit that file:

sudo nano /private/etc/apache2/extra/httpd-vhosts.conf

This file is populated by default with a couple of example virtual host definitions: blocks beginning with <VirtualHost *:80> and ending with </VirtualHost>.

Comment out (i.e. prefix each line with #) or delete the two example virtual host definitions.

Next, add your own dynamic virtual host definition to the file.

I’ve decided to use /usr/local/var/www as the base folder to serve projects from. You can modify this to another path if you’d like.

Add the following block:

# Custom configuration for local dev environment

<Directory "/usr/local/var/www">
  Options Indexes MultiViews FollowSymLinks
  AllowOverride All
  Require all granted
</Directory>

<Virtualhost _default_:80>

  # Dynamic virtual hosts for local development
  UseCanonicalName Off
  VirtualDocumentRoot "/usr/local/var/www/%-2+/public_html"

  # Show 404's in the apache error log
  LogLevel info

</Virtualhost>

Save (Control+O) and exit (Control+X) the nano editor.

Explaining the Dynamic Virtual Host definition

In httpd-vhosts.conf as modified above, the <Directory> block is required for Apache 2.4+ and tells Apache that it can serve files from the /usr/local/var/www folder. This explicitly overrides the rigorous security-focused default specified in httpd.conf that tells Apache it can’t serve up any file on the hard disk (/).

The <VirtualHost> block defines a _default_ VirtualHost that listens on port 80.

The VirtualDocumentRoot directive tells Apache where to look for the dynamic VirtualHost’s files. The %-2+ placeholder stands in for “the penultimate (second to last) and all preceding parts” of the server name in the request. Therefore, per the directive, when handling a request for http://example.com.test, Apache will look for that site’s files in /usr/local/var/www/example.com/public_html.

I like to have my project folder names in dev environments mirror my likely production values, such as example.com/ or hypothetical.app/. You may wish to name your project folders differently. You may prefer to use the %0 placeholder for the entire server name (example.com.test), or %1 for the first-part only (example).

To learn more about these directives, refer to the docs:

Create the required paths

Finally, ensure the paths that you specified for your project folders exist. In my case:

mkdir -pv /usr/local/var/www

If you have any project build folders, you can add them (or better yet: create symlinks to them) here!

To give you an idea of how this works:

In /usr/local/var/www if you were to create the folder helloworld.com/ and sub-folder helloworld.com/public_html, and put a basic index.php containing hello world! inside it, at the end of this tutorial Apache will serve it up when you visit http://helloworld.com.test on your machine.

Smoke-check (config check)

As a smoke-check to make sure you haven’t made any typos or introduced any bugs, run the following command to check the syntax of your apache config files:

sudo apachectl configtest

If you followed all of the above steps carefully, it should output “Syntax OK”.

If everything checks out, restart apache so your latest conf changes can take effect:

sudo apachectl restart

Configuring DNS for local domain names

Now we will solve the problem of directing local requests for http://example.com.test to Apache.

One option is to manually add an entry to /private/etc/hosts for every one of your projects. For example, you might add the line 127.0.0.1 myproject.test to enable the local URL ‘http://myproject.test’.

We are going to use a more dynamic approach using a DNS server so that all requests for any URL at the *.test domain will resolve to localhost.

Any DNS server could be employed for this job. We will use dnsmasq, a perfect choice for this type of lightweight local routing.

Follow the steps in my guide Using dnsmasq on MacOS to setup a local domain for development to complete this step.

Customizing PHP settings (php.ini)

MacOS’ PHP uses a default php.ini file based on /private/etc/php.ini.default.

To customize your PHP environment, if a php.ini file doesn’t already exist at /private/etc/php.ini, copy the default template to create a main php.ini file:

sudo cp /private/etc/php.ini.default /private/etc/php.ini

Make any changes you wish to php.ini and restart apache to reload all configuration files:

sudo apachectl restart

If you were to run phpinfo() in a php file from the web server, you should now see that the Loaded Configuration File property now has the value /etc/php.ini.

A very common tweak to the default PHP configuration is to allow larger file upload sizes. The post_max_size and upload_max_filesize properties are only a few megs by default. These limits can be raised as you see fit.

Many developers also tweak the max_execution_time, max_input_time, and memory_limit settings depending on their project.

Always remember to restart apache after making changes to your PHP configuration.

Installing and configuring mysql server

WordPress and many other PHP apps require a MySQL or compatible database such as MariaDB to store its data.

I prefer to use mariadb, which is available via brew. Run the following command to install it for your user:

brew install mariadb

This installation method is recommended in the official docs:

Following installation, manually start the server by running:

mysql.server start

Running mysql as a service

By default you will need to manually start and stop your mysql/mariadb server.

If you want the server to start when you login to your Mac, use Homebrew’s services feature to have it automatically launch in the background:

brew services start mariadb

Homebrew integrates with MacOS’ launchctl to make this happen the correct way as supported by Apple.

Test logging into mysql via the cli client

Once your server has started, you can test logging in as the root user with the Terminal command:

mysql -u root

Issue the quit command to exit the MySQL prompt and return to your Terminal session.

Set a root password and improve security

Even though a default mysql/mariadb configuration only accepts connections from localhost, it is still wise to run the quick interactive mysql_secure_installation command-line-interface script to set a root password and a few other security-minded options.

Troubleshooting: resolve possible socket incompatibility

There may be a slight configuration mismatch between the defaults of brew’s mysql/mariadb packages and the defaults of MacOS’ built-in PHP mysqli extension. It may be fixed in newer packages.

If try to get a LAMP app such as WordPress to connect to mysql on localhost and encounter a “Connection Refused” error even when the database settings are confirmed correct (as defined in wp-config.php for WordPress), then you may need to make a small fix. You can confirm the issue by trying to connect to 127.0.0.1 instead: if it works but localhost doesn’t then it is a socket-related issue. 

For localhost (socket) connections to the database, brew’s package version tends to use the socket file /tmp/mysql.sock. MacOS’ built-in Apache+PHP mysqli extension assumes the socket file is at /var/mysql/mysql.sock (i.e. the default setting of its mysqli.default_socket property). You can verify the value of this property by running phpinfo(); in a php script and finding it in the list.

To resolve, one could edit php.ini or another conf file. A simple solution is to create a symlink at /private/var/mysql/mysql.sock that points to /tmp/mysql.sock so everything works out:

sudo mkdir -p /var/mysql
sudo ln -s /tmp/mysql.sock /private/var/mysql/mysql.sock

On MacOS /var, a classic folder for unix/linux systems, is symlinked to /private/var so to avoid a symlink-on-symlink type situation, the above uses the real path. 

Note that only socket connections (i.e. database connections made to ‘localhost’) would be impacted by this incompatibility. Connections to ‘127.0.0.1’ force connecting over TCP due to the IP address, bypassing any socket concerns.

Install a database management GUI

If you’d like to use a GUI to help manage your mysql databases, one popular choice is sequel-pro. This serves the same purpose as the popular phpmyadmin webapp except it runs as a desktop app on your Mac. To install it:

brew cask install sequel-pro

You can now launch it from your Applications folder.

Install wp-cli

WordPress’ Command-Line-Tools are a huge timesaver for managing both local and remote WordPress installs, and they are essential for writing scripts that automate WordPress deployment, updates, etc.

Install wp-cli for your user with the command:

brew install wp-cli

Confirm your install by running wp --version in Terminal.

Assuming your project folder includes a public_html/ folder that contains your WordPress install, and that you have created a database called “DB_NAME”, you can download and setup WordPress with a handful of wp-cli commands:

cd /Users/username/web-projects/my-project.com/public_html
wp core download
wp config create --dbname=DB_NAME --dbuser=DB_USERNAME --dbpass=DB_PASSWORD --dbhost=localhost
wp core install --url=my-project.com --title="My Project" --admin_name="example_admin" --admin_password="example_password" --admin_email=you@example.com

Check out the command reference and docs/handbook at https://wp-cli.org/

Serving up your project

To make a project available at http://example.com.test you need to:

  • create a folder example.com/ in your chosen base directory (e.g. /usr/local/var/www)
  • create the public_html subfolder: example.com/public_html/
  • copy project files to public_html/ (manually or as an automated task e.g. with gulp)

OR

  • create a symlink from your project location to the chosen base directory

For the symlink option:

ln -s ~/some/project/folder/example.com /usr/local/var/www/example.com

Be sure to replicate the requirement for a public_html/ sub-folder.

When creating symlinks from elsewhere on your system to /usr/local/var/www be careful of permissions issues!

Managing permissions

MacOS’ Apache is running as the user _www by default. That user must have at least read permissions for any folder that you symlink to in order to serve up your project files.

It’s important to note that ~/Documents and any folders created within are not accessible to any group or other users. Apple thankfully assumes that you don’t want other users on a system to be able to access your private documents! As a result, Apache’s _www user will not be able to follow symlinks into the Documents folder belonging to your user.

If you have any web projects in ~/Documents you could save them somewhere else where Apache can read them and symlink to them from there, or you could simply copy your project files over to /usr/local/var/www.

Depending on your desired setup, you may find it convenient to change the group ownership of /usr/local/var/www to _www and set the groupID bit (setgid):

sudo chgrp _www /usr/local/var/www
sudo chmod g+s /usr/local/var/www

The setgid bit ensures that any new files or folders created underneath www/ will inherit the group ID of the directory vs. the default behaviour of having it set to the primary group ID of the user that created the file. This measure can help ensure that Apache’s _www user can access the contents of the www/ folder.

Finally, remember that our particular Apache VirtualHost configuration requires a public_html/ sub-directory under the my-project.com folder for web-servable files. To illustrate: an index.php file in ~/web-projects/my-project.com/public_html will be served up to a local web browser that requests the URL http://my-project.com.test. This works because there is a symlink in /usr/local/var/www named my-project.com that points to the folder ~/web-projects/my-project.com.

Working on a project

In the future, if you haven’t setup apache and mysql server to run automatically as services, you will need to start them:

sudo apachectl start
mysql.server start

To stop or restart apache, use the following commands:

sudo apachectl stop
sudo apachectl restart

You can stop mysql with:

mysql.server stop

Remember: if you make any changes to your apache config, such as adding a new VirtualHost, you will need to restart or reload apache for your changes to take effect.

Finally remember that the settings covered in this guide will serve up your projects to any clients on your network. If you want to hide them from everyone on your local coffee shop’s shared wifi, turn off the servers, harden your firewall (via System Preferences), and/or tweak your Apache conf to only serve to localhost.

Upgrading MacOS

When you upgrade MacOS, the installer will create versions of your conf files with the suffix ~previous before replacing with its own.

You can also create your own backup copies to be safe.

Following an OS update, if the Apache files are changed, you can compare the new versions with your ~previous versions using the diff command to make sure there’s no major updates (very unlikely) then restore your customized versions. After restoring your files, remember to restart apache for the settings to take effect: sudo apachectl restart.

The files customized by following this guide are:

  • /etc/apache2/httpd.conf
  • /private/etc/apache2/extra/httpd-vhosts.conf

Script to Deploy WordPress

Check out this gist for a bash script to quickly deploy a fresh WordPress install:

https://gist.github.com/firxworx/bb9b7f8d71915f9d4e7f8d3a8a531b26

Using dnsmasq on MacOS to setup a local domain for development

This guide covers using dnsmasq as a local DNS server on MacOS to resolve all URL’s with the .test domain name to localhost (127.0.0.1).

This can be very useful to developers wishing to setup an efficient local development and test environment on their machine. For example, if http://myproject.com.test resolved to localhost, a local apache, nginx, or other server could respond to the request.

While any DNS server could be employed for this job, dnsmasq is a perfect choice for this type of lightweight local routing.

Regarding the .test domain, note that this is a recommended choice per the IETF RFC’s (e.g. RFC-2606, RFC-6761). While it’s possible to configure dnsmasq for any arbitrary domain name, only .test, .example, .invalid, and .localhost are reserved for non-Internet use. A popular choice used to be .dev until Google purchased the top-level-domain (TLD) and changed how Chrome handles .dev URL’s.

Install and configure dnsmasq

To get started, use brew to install dnsmasq for your user:

brew install dnsmasq

At the time of writing, the brew package installer creates a sample conf file with all entries commented out. It can be found at ./etc/dnsmasq.conf under brew’s default install folder.

Execute following commands in sequence to configure dnsmasq to resolve *.test requests to 127.0.0.1:

echo "" >> $(brew --prefix)/etc/dnsmasq.conf
echo "# Local development server (custom setting)" >> $(brew --prefix)/etc/dnsmasq.conf
echo 'address=/.test/127.0.0.1' >> $(brew --prefix)/etc/dnsmasq.conf
echo 'port=53' >> $(brew --prefix)/etc/dnsmasq.conf

The above commands simply append the required lines to the dnsmasq.conf configuration file.

Next, use brew’s services feature to start dnsmasq as a service. Use sudo to ensure that it is started when your Mac boots, otherwise it will only start after you login:

sudo brew services start dnsmasq

Add a resolver

Add a resolver for .test TLD’s:

sudo mkdir -pv /etc/resolver
sudo bash -c 'echo "nameserver 127.0.0.1" > /etc/resolver/test'

This tells your system to use localhost (and therefore dnsmasq) as the DNS resolver for the .test domain.

Test your configuration

Flush your DNS cache:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

Check your configuration by running:

scutil --dns

Inspect the output. One of the entries should specify that for domain test the system will use the nameserver 127.0.0.1. This means that dnsmasq will now be consulted as the nameserver for any .test URLs.

Finally, test the full stack by trying to ping random *.test URL’s such as example.test or myproject.test and confirming that they resolve to 127.0.0.1. For example:

ping -c2 example.test

Provides the output:

PING example.test (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.022 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.055 ms

OR (in cases where your Mac’s firewall settings are configured to not respond to ping):

PING example.test (127.0.0.1): 56 data bytes
Request timeout for icmp_seq 0
...

The important part to confirm is that example.test resolved to (127.0.0.1).

Include the client IP in apache logs for servers behind a load balancer

When servers are behind a load balancer, Apache’s default configuration produces logs that show the load balancer’s IP address instead of the IP of the remote client that initiated the request. Furthermore, if multiple server’s logs are consolidated into one, it can be difficult to determine which server created a given log entry.

This post covers how to improve on Apache’s default logging situation such that:

  • every web server behind the load balancer includes a unique identifier for itself in its log entries, and that;
  • access log entries include the client’s remote IP address as found in the X-Forwarded-For header set by the load balancer.

This setup is more helpful for troubleshooting, configuring monitoring and alerts, etc. and it helps to maximize the value of 3rd-party log aggregation and analysis services like Papertrail or Loggly).

The example commands in this post are applicable to Ubuntu/Debian however they are easily adapted to other environments.

Enable required modules

Start by ensuring that the required Apache modules: env and remoteip, are enabled:

a2enmod env
a2enmod remoteip 
service apache2 restart

Identify each web server

Add a SetEnv directive in the site/app’s apache conf to instruct the env module to set a new environment variable with a value that uniquely identifies each server behind the load balancer.

The example below is a snippet from an Apache VirtualHost’s conf file. It defines a variable called APP_LB_WORKER with the value ‘unique_identifier’. If you were using a devops automation tool such as Ansible, you could use the template module and use a handy variable such as {{ ansible_host }} in place of the example’s hard-coded ‘unique_identifier’ value.

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
</VirtualHost>

Configure the Apache RemoteIP module

Create a file named /etc/apache2/conf-available/remoteip.conf and use it to set: the RemoteIPHeader to ‘X-Forwarded-For’, any appropriate RemoteIPInternalProxy or other RemoteIP directives, and to define a new LogFormat that includes both the environment variable that contains the server’s identifier and the client’s remote IP as sourced from the X-Forwarded-For header.

The RemoteIPInternalProxy directive tells the RemoteIP module which IP address(es) or IP address blocks it can trust to provide a valid RemoteIPHeader that contains the client’s IP.

The following example’s RemoteIPInternalProxy value is representative of an environment where the load balancer’s internal network IP address belongs to a public subnet with the CIDR block 10.0.1.0/24. Choose an appropriate value (or values) for your environment.

A full list of configuration directives for RemoteIP can be found in the Apache docs: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html.

The following example names the new LogFormat as “loadbalance_combined”. You can choose any name you like that isn’t already in use.

/etc/apache2/conf-available/remoteip.conf:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.1.0/24

LogFormat "%a %v %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{APP_LB_WORKER}e\"" loadbalance_combined

Finally, enable the conf:

a2enconf /etc/apache2/conf-available/remoteip.conf

The Apache LogFormat is extensively customizable. Learn more about the placeholders and options from here: https://httpd.apache.org/docs/2.4/logs.html. Remote log aggregation service Loggly also has an excellent overview: https://www.loggly.com/ultimate-guide/apache-logging-basics/.

Tell Apache to use the new LogFormat

Next, tell Apache to use the custom LogFormat that we named loadbalance_combined by editing your site/app’s conf file. The following example builds upon the previous example of a VirtualHost conf:

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
    CustomLog /var/log/app_name/access.log loadbalance_combined
...
</VirtualHost>

The following example is a more elaborate case that uses the tee command to send the log entry to both an access.log file and to the /usr/bin/logger command to include it in the syslog, only if an environment variable named “dontlog” is not set. An example use-case for an env variable like “dontlog” is to set it (e.g. via the SetEnvIf directive) for any requests that correspond to a “health check” from the load balancer to the web server. This helps keep logs clean and clutter-free.

CustomLog "|$/usr/bin/tee -a /var/log/app_name/access.log | /usr/bin/logger -t apache2 -p local6.notice" loadbalance_combined env=!dontlog

Restart Apache

Confirm the validity of your confs using apachectl configtest and restart apache for the new configuration to take effect:

service apache2 restart

Encrypting a USB Drive in MacOS, including formatting to JHFS+ or APFS

MacOS Sierra doesn’t feature an option to encrypt a USB drive in Disk Utility or in Finder (at least at the time of writing). This post covers how to format a USB drive to either the JHFS+ or the new APFS filesystem and encrypt it using the Terminal and Disk Utility.

Instructions

First, plug your USB drive into your computer and open the Terminal app.

Use the following command to list your disks:

diskutil list

Look for an entry (or entries) like /dev/disk2 (external, physical) and make absolutely sure that you understand the difference between your system’s hard disk and the external USB drive you want to encrypt.

The “IDENTIFIER” for my USB drive, found at the top of the list, was disk2. My system showed a subsequent entry disk2s1 but note how this still refers to disk2. Only the disk2 part is required. Use whatever diskn number corresponds to your target drive, where n is an integer.

Only proceed if you are certain that you have correctly identified your USB drive!

The following command formats the drive to Apple’s HFS+ with Journaling format, JHFS+. GPT is a crucial argument here to specify the GUID Partition Map option vs. the Master Boot Record option. You can replace the text inside the quoted string (“Flash Drive”) with your desired drive name:

diskutil eraseDisk JHFS+ "Flash Drive" GPT disk2

It is now possible to encrypt the drive with Finder (right-click and choose “Encrypt ‘Flash Drive'”) if you wish to simply keep the JHFS+ file system. If you wish to use the newer APFS file system, do not Encrypt the drive just yet, and read on.

Using the new APFS file system

APFS is Apple’s latest filesystem and it features good support for encryption. Before formatting your drive to APFS, be aware that older Macs (i.e. those without MacOS Sierra and up) will not support it.

To proceed with APFS, open the Disk Utility app.

With the drive formatted to JHFS+, Disk Utility will no longer grey out the “Convert to APFS” option when you right/control+click it.

Find your drive and choose “Convert to APFS”.

Once the file system has been converted to APFS you can go back to Finder, right/control+click on your drive, and choose “Encrypt ‘Flash Drive'” from the menu.

Don’t forget your passphrase 😉

Troubleshooting the fast.ai AWS setup scripts (setup_p2.sh)

Fast.ai offers a well-regarded free online course on Deep Learning that I thought I’d check out.

It seems that a lot of people struggle getting the fast.ai setup scripts running. Complaints and requests for help are on reddit, in forums, etc. This doesn’t surprise me because the scripts are not very robust. On top of that, AWS has a learning curve so troubleshooting following a script failure can be a challenge.

Hopefully this post helps other people that have hit snags. It is based on my experience on MacOS, however should be very compatible for those running Linux or Windows with Cygwin.

Understanding the setup script’s behaviour

It leaves a mess when it fails

If running the setup script fails, which is possible for a number of reasons, it will potentially have created a number of AWS resources in your account and a local copy of an SSH key at ~/.ssh/aws-key-fast-ai.pem. It does not clean up after itself in failure cases.

The setup script doesn’t check for existing fast-ai tagged infrastructure, so subsequent runs can create additional VPC’s and related resources on AWS, especially as you attempt to resolve the reason(s) it failed. The setup script might generate fast-ai-remove.sh and fast-ai-commands.txt but it overwrites these each time its run with only its current values, potentially leaving “orphan” infrastructure.

Thankfully all AWS resources are created with the same “fast-ai” tags so they are easy to spot within the AWS Console.

It makes unrealistic assumptions

The setup script assumes your aws config’s defaults specify a default region in one of its three supported regions: us-west-2, eu-west-1, and us-east-1.

I’m not sure why the authors assumed that a global tech crowd interested machine learning would be unlikely to have worked with AWS in the past and thus no existing aws configuration that might conflict.

The commands in the script do not use the --region argument to specify an explicit region so they will use whatever your default is. If your default happens to be one of the three supported ones, but you don’t have a sufficient InstanceLimit or there’s another problem, more issues could follow.

Troubleshooting

If you encountered an error after running the script, prior to re-running the script, take note of the following checks when attempting to resolve:

Check 1: Ensure you have an InstanceLimit > 0

Most AWS users will have a default InstanceLimit of 0 on P2 instances. You may need to apply for an increase and get it approved (this is covered in the fast.ai setup video).

If a first run of the script gave you something like the following, there was an issue with your InstanceLimit:

Error: *An error occurred (InstanceLimitExceeded) when calling the RunInstances operation: You have requested more instances (1) than your current instance limit of 0 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.* 

InstantLimits are specific to a given resource in a given region. Take note of which region your InstanceLimit increase request was for and verify that it was granted in the same region.

Check 2: Ensure the right region

Verify your current default aws region by running: aws configure get region. The script assumes this is one of three supported regions: us-west-2, eu-west-1, or us-east-1.

The script also assumes that you have an InstanceLimit > 0 for P2 instances in whichever region you would like to use (or T2 instances if you are using setup_t2.sh).

To get things running quickly, I personally found it easiest to make the script happy and temporarily set my aws default to a supported region in ~/.aws/config, i.e.:

[default]
region=us-west-2

Another option is to modify the scripts and add an explicit --region argument to every aws command that will override the default region. If you have multiple aws profiles defined as named profiles, and the profile that you wish to use for fast.ai specifies a default region, you can use the --profile PROFILENAME argument instead.

For example, the following hypothetical aws config file (~/.aws/config) specifies a profile called “fastai”. A --profile fastai argument could then be added to every aws command in the setup script:

[default]
region=ca-central-1

[profile fastai]
region=us-west-2

Check 3: Delete cruft from previous failed runs

This check is what inspired me to write this post!

Delete AWS resources

Review any resources were created in your AWS Console, and delete any VPC’s (and any dependencies) that were spun up. They can be identified because they were created with the “fast-ai” tag which is shown in any tables of resources in the AWS Console.

Cruft resources will have been created in any region that the setup script was working with (i.e. whatever your default region was at the time you ran it).

If you’ve found cruft, start by trying to delete the VPC itself, as this generally will delete most if not all dependencies. If this fails because of a dependency issue, you will need to find and delete those dependencies first.

IMPORTANT: AWS creates a default VPC and related dependencies (subnets, etc.) in every region available to your account. Do NOT delete any region’s default VPC. Only delete resources tagged with “fast-ai”.

Delete SSH keys

Check to see if ~/.ssh/aws-key-fast-ai.pem was created, and if so, delete it before running the script again.

The setup script has logic that checks for this pem file. We do not want the script to find the file on a fresh run.

After a successful run

After the setup script ran successfully, I got output similar to:

{
    "Return": true
}
Waiting for instance start...

All done. Find all you need to connect in the fast-ai-commands.txt file and to remove the stack call fast-ai-remove.sh
Connect to your instance: ssh -i /Users/username/.ssh/aws-key-fast-ai.pem ubuntu@ec2-XX-YY-ZZ-XXX.us-west-2.compute.amazonaws.com

Reference fast-ai-commands.txt for information about your VPC and EC2 instance. An ssh command to connect is in the file, and you can find your “InstanceUrl”.

I suggest picking up the video from here and following along from the point where you connect to your new instance. It guides you through checking the video card with the nvidia-smi command and running jupyter: http://course.fast.ai/lessons/aws.html

Starting and stopping your instance

The fast-ai-commands.txt file outlines the commands to start and stop your instance after the setup has completed successfully, e.g.:

aws ec2 start-instances --instance-ids i-0XXXX
aws ec2 stop-instances --instance-ids i-0XXXX

Its important to stop instances when you are finished using them so that you don’t get charged hourly fees for their continued operation. P2 instances run about $0.90/hr at the time of writing.

Avoiding duplicate entries in authorized_keys (ssh) in bash and ansible

Popular methods of adding an ssh public key to a remote host’s authorized_keys file include using the ssh-copy-id command, and using bash operators such as >> to append to the file.

An issue with ssh-copy-id is that this command does not check if a key already exists. This creates a hassle for scripts and automations because subsequent runs can add duplicate key entries. This command is also not bundled with MacOS, creating issues for some Mac users (though it can be installed with Homebrew).

This post covers a solution that adds a given key to authorized_keys only if that key isn’t already present in the file. Examples are provided in bash and for ansible using ansible’s shell module (old versions) and authorized_key module (newer versions).

For shell scripts, there seem to be a lot of solutions out there for this common problem, but I think a lot of them overcomplicate things with sed, awk, uniq, and similar commands; or go overboard by implementing standalone utilities for the task. One thing I don’t like about many of the working solutions that I’ve come across is when the authorized_keys file is reordered as a side-effect.

Note that ssh authentication works fine when there are multiple identical authorized_keys entries. However, accumulating junk in this file can create performance issues, and can make troubleshooting, auditing, and other admin tasks more difficult. When a remote host tries to authenticate, ssh works its way down the authorized_keys file until it comes across a match.

Adding a unique entry to authorized_keys

The following is a one-liner to be run by a user that can authenticate with the remote server.

Modify the snippet below to suit your needs:

ssh -T user@central.example.com "umask 0077 ; mkdir -p ~/.ssh ; grep -q -F \"$PUB_KEY\" ~/.ssh/authorized_keys 2>/dev/null || echo \"$PUB_KEY\" >> ~/.ssh/authorized_keys"

The command adds the public key stored in the shell variable $PUB_KEY to the authorized_keys file of the user on the server central.example.com. A umask ensures the correct file permissions.

To modify, replace user and central.example.com with values relevant to you, and either substitute your public key in place of the $PUB_KEY variable, or define the variable in a bash script or set it as an environment variable prior to executing the command.

Benefits of this approach:

  • unique entries: no duplicate authorized_keys
  • idempotent: subsequent runs given the same input will yield the same result
  • order preserved: entries in authorized_keys retain their order
  • correct permissions: in cases where the .ssh folder and/or authorized_keys file do not already exist, they will be created with the correct permissions for openssh thanks to the umask
  • quiet: the command is quiet
  • automation friendly: fast one-liner that’s easy to add to scripts, with minimized race conditions in situations that involve running automations (e.g. ansible playbooks) in parallel
  • KISS principle: its not as risky or difficult to configure as some other approaches that I have encountered online

Tip: If you want to suppress any motd/welcome banner content that might be outputted when connecting to the remote server via ssh, first touch a .hushlogin file in the target user’s home directory to suppress it.

Ansible implementation

Current: using the authorized_key module

The newer known_hosts module and authorized_key module (featuring numerous feature additions from its introduction through to 2.4+) were introduced to help manage ssh keys on a host.

The authorized_key module has a lot of useful options, including optional exclusivity, supporting sourcing keys from variables (and hence files via a lookup) as well as URL’s, and options to manage the authorized_keys/ folder (e.g. creating it with appropriate permissions if it doesn’t exist).

An example from the docs follows, with one addition: I added the exclusive option in keeping with the theme of this post.

- name: Set authorized key took from file
  authorized_key:
    user: charlie
    state: present
    key: "{{ lookup('file', '/home/charlie/.ssh/id_rsa.pub') }}"
    exclusive: yes

See the ansible documentation for more examples: http://docs.ansible.com/ansible/latest/authorized_key_module.html

Legacy: using bash in the shell module

One of the more annoying aspects of ansible can be getting escape characters right in templates and certain modules like shell, especially when variables are involved. The following example has valid syntax. You can modify the variables and the become and delegate_to args to suit your scenario:

# assume the ansible user on the control machine can access the remote target server via ssh 

- name: set_fact host_pub_key containing current host's pub key from local playbook_dir/keys
  set_fact:
    host_pub_key: "{{ lookup('file', playbook_dir + '/keys/{{ inventory_hostname }}-{{ authorized_user }}-id_rsa.pub') }}"

- name: add current host's pub key to repo server's authorized_keys if its not already present 
  shell: |
    ssh -T {{ example_user }}@{{ example_server }} "umask 0077 ; mkdir -p ~/.ssh ; grep -q -F \"{{ host_pub_key }}\" ~/.ssh/authorized_keys 2>/dev/null || echo \"{{ host_pub_key }}\" >> ~/.ssh/authorized_keys"
  args:
    executable: /bin/bash
  become: "{{ ansible_user_id }}"
  delegate_to: localhost

The first task populates the host_pub_key fact from a hypothetical id_rsa.pub key file.

The second task executes the bash snippet that adds the public key to the remote host’s authorized_keys file in a way that avoids duplicates.