Category: Automation

How to run E2E Tests with docker-compose

This guide covers using docker-compose to spin up your application, run E2E tests, and then exit with the results.

The TL;DR is:

  • docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app

For the sake of an example, we’ll be testing a hypothetical API written in NodeJS that uses a Postgres database and a Redis instance. We’ll assume tests are run via jest. You can substitute any stack!

To begin, ensure that you have recent versions of docker and docker-compose installed on your machine.

Dockerfile

Ensure you have a Dockerfile in your project’s folder that specifies how to build an image for your app.

The following example is for a web API written in NodeJS.

FROM node:14.4-alpine As example

# install build dependencies
RUN apk update && apk upgrade
RUN apk add python3 g++ make

# install packages for sending mail (msmtp = sendmail for alpine)
RUN apk add msmtp
RUN ln -sf /usr/bin/msmtp /usr/sbin/sendmail

# make target directory for assigning permissions
RUN mkdir -p /usr/src/app/node_modules
RUN chown -R node:node /usr/src/app

# use target directory
WORKDIR /usr/src/app

# set user
USER node

# copy package*.json separately to prevent re-running npm install with every code change
COPY --chown=node:node package*.json ./
RUN npm install

# copy the project code (e.g. consider: --only=production)
COPY --chown=node:node . .

# expose port 3500
EXPOSE 3500

docker-compose

Create a docker-compose.e2e.yml file.

The following example creates a service called app that runs in a container named example.

Note the command property. This should specify the command that will run your tests inside the container. In our example, this is yarn test:e2e.

version: '3.8'

services:
  app:
    container_name: example
    build:
      context: .
      target: example # only build this part of the Dockerfile (see: '... As example' )
    volumes:
      - .:/usr/src/app
      - /usr/src/app/node_modules # 'hack' prevents node_modules/ in the container from being overridden
    working_dir: /usr/src/app
    command: yarn test:e2e
    environment:
      PORT: 3500
      NODE_ENV: test
      DB_HOSTNAME: postgres
      DB_PORT: 5432
      DB_NAME: example
      DB_USERNAME: postgres
      DB_PASSWORD: postgres
      REDIS_HOSTNAME: redis
      REDIS_PORT: 6379
    networks:
      - webnet
    depends_on:
      - redis
      - postgres

  redis:
    container_name: redis
    image: redis:5
    networks:
      - webnet

  postgres:
    container_name: postgres
    image: postgres:12
    networks:
      - webnet
    environment:
      POSTGRES_DB: example
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      PG_DATA: /var/lib/postgresql/data
    volumes:
      # - ./seed.db.sql:/docker-entrypoint-initdb.d/db.sql <- run only once when the pgdata volume is first created (when run via docker-compose)
      - pgdata:/var/lib/postgresql/data # or specify a local folder like ./docker-volumes/pgdata:/var/lib/postgresql/data

networks:
  webnet:

volumes:
  pgdata:
  logs:

Note how each service shares the same network so they can communicate with each other.

Tip: you can use .env files and reference variables from them in a docker-compose.yml file as follows: ${VARIABLE_NAME}.

If you wish to specify a particular .env file in your docker-compose.yml file:

env_file:
  - .env

Run E2E Tests

From your project folder, you can run the following command to run your tests:

docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app

The -f flag specifies a custom configuration file for docker-compose. If this is not specified, docker-compose will look for docker-compose.yml by default.

The up command tells docker-compose to bring the services and containers up.

The --abort-on-container-exit and --exit-code-from flags are an important combination.

The first flag shuts things down when our test run is complete, and the second flag will use the exit code from the specified service (in our case the one named app) as the exit code from the overall docker-compose command.

This is a good setup if you have scripts that run tests, or if you have a continuous integration pipeline that automatically runs tests and requires a pass/fail.

Test runners such as jest will generally exit with code 0 (success) if all tests pass, and exit with a non-zero code (failure) if any tests fail.

package.json

If your project uses npm, yarn or their ilk, you can specify commands to run tests in the scripts section.

Our docker-compose.yml file requires the app service to run the command yarn test:e2e. In our hypothetical example app, this is specified as follows:

"test:e2e": "NODE_ENV=test jest --config ./test/jest-e2e.json",

Of course, you will need to specify the command that initiates running E2E tests that’s particular to your project and environment.

To spin up your application via docker-compose and run tests from your workstation (or CI environment, etc), add the following script:

"test:e2e:docker": "docker-compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from app"

You could then run via npm run test:e2e:docker or yarn test:e2e:docker

If you are using a different package/dependency management solution, you can specify your test-related scripts there. You also have the option to define shell/bash scripts that can run your tests.

Editing WordPress’ wp-config.php with wp-cli and adding variables with sed

The WordPress CLI (command line interface) is a huge step for enabling developers and devops/sysadmin folks manage their WordPress installations. It’s awesome for writing scripts to automate key tasks.

This post covers editing the WordPress configuration file wp-config.php with the WP-CLI’s wp config command, as well as using the sed command to address a key missing feature of WP-CLI: the ability to add new config variables.

There are plenty of reasons you might want to edit wp-config.php via a script or directly via the command line. For example, developers might appreciate a bash script that sets WP_DEBUG to true, and a devops person might want to create an automated deploy process that ensures key SMTP settings are in place.

The rest of this post will assume your WP-CLI command can be invoked with wp. Depending on how you installed it, the command might be available to you as wp-cli or wp-cli.phar. If you are running wp-cli’s phar file directly, substitute php wp-cli.phar in place of wp in the examples.

Editing config variables with wp-cli’s config command

WP-CLI supports modifying config variables in wp-config.php via wp config.

This is a great feature, albeit with the noted catch that wp config only works for a given variable if that variable is already defined in wp-config.php. I’ll show you how to work around that and add variables with sed in the next section.

The following example uses sudo to run wp config as the _www web server user, the default web server user on MacOS. On Ubuntu and many other linux distros, this user is likely www-data:

sudo -u _www wp config set FS_METHOD 'direct'
sudo -u _www wp config set DISABLE_WP_CRON true
sudo -u _www wp config set WP_DEBUG true
sudo -u _www wp config set WP_DEBUG_LOG true

These are some of the most popular config options that developers and admins want to modify.

Adding config variables to wp-config.php using sed

There are a number of command-line utilities on Linux and Unix-like systems that can edit text files. One of the most popular is sed, the quintessential stream editor. Unix admins have been working with streams forever, long before NodeJS made it cool :).

sed is pre-installed on most systems and can be used directly in the Terminal or inside a bash (or other shell) script.

The following example uses sed to add config variables to wp-config.php right before the well-known “That’s all, stop editing!” comment line found in the file.

This snippet works on MacOS and elsewhere. MacOS and OSX, as well as their related family in the BSD/unix world, generally bundle a classic POSIX-compliant version of the sed command which is more limited vs. the more common and more popular GNU sed that ships with major linux distributions like Ubuntu. If you’re on linux, delete the double quotes '' immediately following the -i flag to be compatible with GNU sed.

Editing wp-config.php with sed:

sed -i '' '/\/\* That.s all, stop editing! Happy blogging. \*\// i\
// FX_SCRIPT FS_METHOD \
define( "FS_METHOD", "direct" ); \
\
// FX_SCRIPT WP_DEBUG \
define( "WP_DEBUG", true ); \
define( "WP_DEBUG_LOG", true ); \
\
// FX_SCRIPT DISABLE_WP_CRON \
define( "DISABLE_WP_CRON", true ); \
\
' wp-config.php

The -i option tells sed to edit the file in-place i.e. modify the file directly. Otherwise, sed lives up to its name and streams output to stdout.

The MacOS version of sed requires a backup file to be specified as the first argument whenever the -i option is used. You can pass empty quotes '' to specify no backup file as demonstrated in the example.

The linux version of sed does not require a backup filename to be specified. You can simply delete the '' arguments as noted above.

The way that single and double quotes are used is very important for getting this command to work. Getting them right is one of the trickiest parts about using sed.

Also note how backslashes are used at the end of each line. This is required to make the command portable and universal: classic sed (BSD/Unix/MacOS) does not recognize \n as a placeholder for the newline character while GNU sed (linux) does. The backslashes enable a trick to use actual newlines instead of placeholders.

Finally, my example adds comment lines that start with // FX_SCRIPT before each change (get it? FX = firxworx, the name of this blog!). I do this to make it easy to search with grep and/or visually look for changes in wp-config.php files that were made by my scripts. You may wish to follow a similar practice. This makes it easier to write other scripts that might find and comment out or delete these entries at a later time.

Installing gulp4 with babel to support an ES6 gulpfile

This guide covers installing gulp4 with babel to support ES6 syntax in your gulpfile.

Gulp is a task automation tool that has emerged as one of the standard build tools to automate the web development workflow. Babel is a compiler/transpiler that enables developers to use next-generation ECMAScript syntax (ES6 and beyond) instead of older JavaScript (ES5) syntax.

Gulp4 and ES6+ work together swimmingly to help you write cleaner, easier-to-read, and more maintainable gulpfile’s.

Installing gulp4

At the time of writing, the default gulp package installs gulp 3.x. The following will install and configure gulp4.

Gulp has two key parts: gulp and the gulp-cli command line tools. The idea is that gulp-cli should be installed globally on a developer’s machine while gulp should be installed locally on a per-project basis. This helps ensure compatibility with different versions of gulp that will inevitably arise when maintaining projects of different vintages.

To use gulp4, cli version 2.0 or greater is required. Check the version on your system with:

gulp -v

If the command returns a command not found error, then you probably don’t have gulp installed at all (or at least don’t have it available in your PATH).

If the command outputs a version lower than 2.0, you may need to uninstall any globally-installed gulp (and/or gulp-cli) and then install the current version gulp-cli before proceeding.

To install gulp-cli globally, run ONE of the following commands, depending on your preference of package manager. npm is the classic node package management tool and yarn is a newer tool developed by Facebook that addresses certain shortcomings with npm.

yarn global add gulp-cli
# OR
npm install gulp-cli -g

Test the install by running gulp -v and ensuring the version output is greater than 2.0. Next, install the gulp@next package. The @next part specifies the next-generation gulp4.

Assuming you have already run npm init or yarn init and have a package.json file, execute the following command in your project’s root directory:

yarn add gulp@next --dev
npm install gulp@next --save-dev

Installing babel

yarn add @babel/core --dev
yarn add @babel/preset-env --dev
yarn add @babel/register --dev
# OR 
npm install @babel/core --save-dev
npm install @babel/preset-env --save-dev
npm install @babel/register --save-dev

Next, create a .babelrc file in your project’s root folder and specify the current version of node as the target:

{
  "presets": [
    ["@babel/preset-env", {
      "targets": {
        "node": "current"
      }
    }]
  ]
}

Create your gulpfile

Create your gulpfile with the filename gulpfile.babel.js. The babel.js suffix ensures that babel will be used to process the file.

The following example demonstrates a few ES6+ features: optional semicolons, import statements, and the “fat arrow” syntax for defining functions:

'use strict'

import gulp from 'gulp'

gulp.task('task-name', () => {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

Gulp4 features a new task execution system that introduces the functions gulp.series() and gulp.parallel() that can execute gulp tasks in either series (one-after-another) or parallel (at the same time). This makes a lot of workflows much easier to define vs. previous versions!

Another nice feature is that gulp4 supports returning a child process to signal task completion. This makes it cleaner to execute commands within a gulp task, which can help with build and deployment related tasks.

The following example defines a default build task that runs two functions/tasks in series using gulp.series(). The build task is defined using the ES6 const keyword and exported as the default function/task for the gulpfile. The example doSomething() and doAnotherThing() functions/tasks are also exported.

'use strict'

import gulp from 'gulp'

export function doSomething {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

export function doAnotherThing {
  // example 
  return gulp.src('/path/to/src')
    .pipe(gulp.dest('/path/to/dest'))
})

const build = gulp.series(doSomething, doAnotherThing)

export default build

How to install Ansible on MacOS using pip

Ansible is a powerful devops/automation tool that helps you control fleets of servers and their applications with ease.

This post covers installing ansible on MacOS using the installation method recommended by the project: pip, the python package manager. There are a few other ways to install ansible on MacOS, including with the brew package manager.

This post assumes that you are interested in installing ansible only for the current user only. If you require a system-wide/multi-user installation, skip to Step 8 to learn more about the MacOS System Integrity Protection (SIP) feature before proceeding.

Installing Ansible with pip

You may be able to skip steps 1, 2, and/or 3 if you already have XCode, XCode Command Line Tools, and/or pip installed on your system.

Step 1: install XCode

Most developers already have XCode installed. If you’re not sure, you can look for it in the Applications folder on your Mac.

XCode is available for free on the App Store. It can be helpful to run XCode at least once after installing it to help it complete its installation process, and get you to agree to any license terms.

The following steps will fail if you have not agreed to Xcode’s license terms.

Step 2: install XCode command line tools

The XCode command line tools include compilers and other tools necessary to proceed. In the MacOS Terminal app, run the command:

xcode-select --install

If this is a fresh install, you will be prompted to agree to Apple’s terms for the command line tools. After you consent, they will download and install.

If you already have the command-line tools installed. The script will let you know and exit.

Once the XCode command line tools are installed, you can use development tools like the gcc compiler. For example, you should be able to run the following command and successfully receive gcc’s version information as output:

gcc --version

If everything looks good and the above command does not throw an error, proceed to the next step.

Step 3: install pip

Pip is the python package manager. It is conceptually similar to other well-known package management tools such as npm (node/javascript) or apt (debian/ubuntu linux distributions).

MacOS comes pre-installed with python, but not pip. To install it, input the following into the Terminal app:

sudo easy_install pip

There is no harm to running this command in cases where pip is already installed.

Step 4: install ansible

To install ansible system-wide, run the following command:

sudo pip install ansible

If you prefer a local install within your user account, you can remove the ‘sudo’ and add the --user flag to the above.

After the installation process has completed, you can verify the install was a success by executing a basic command like: ansible --version. It should run without error.

Step 5: create /etc/ansible

Ansible looks for default configuration files in /etc/ansible. This directory is unlikely to exist on your machine. The following command will create this folder if it doesn’t already exist:

sudo mkdir -p /etc/ansible

Step 6: install optional/helpful packages

It can be helpful to install the python passlib library because MacOS doesn’t use the same types of password hashes as linux does, and you are likely going to want to manage linux machines with ansible. Passlib provides a common and consistent hash generator across operating systems.

To install passlib globally on your Mac:

sudo pip install passlib

If you installed ansible locally, you can use the command pip install --user passlib instead.

Step 7: test out your installation

Per above, if the install was successful, you can now use the ansible command from the Terminal:

ansible --version

A “hello world” of sorts for ansible is pinging yourself:

ansible localhost -m ping

You can also try running an arbitrary command on yourself (such as uname -a or whereis bash):

ansible localhost -a 'uname -a'

To get ansible to detect facts about your machine, try the following. Note the filter argument: it restricts what is returned so you don’t get BLASTED with facts.

By default ansible gathers almost everything there is to know about its target machines and populates them into variables that you can then use in your tasks, roles, and plays. As you learn ansible, you’ll discover this is incredibly useful. Ansible users can customize this behaviour and optionally turn it off to realize performance gains when it isn’t required. For now though, you’re just getting started:

ansible localhost -m setup -a 'filter=ansible_distribution'

Step 8: upgrade ansible

It can be a good idea to upgrade ansible right off the bat to ensure that everything is current, and to understand how the MacOS System Integrity Protection (SIP) feature impacts your installation:

sudo pip install ansible --upgrade

If this command partially runs and then produces an error, it is because of the SIP feature, originally introduced in OSX El Capitan. You can work around that by adding the --user python parameter to the above command. With this argument, upgrades will be installed for the current user only vs. system-wide, and the upgrade will succeed. The revised command:

sudo pip install ansible --upgrade --user python

Understanding MacOS’ System Integrity Protection (SIP) Feature

SIP was introduced in OSX El Capitan and prevents certain system files, such as those associated with MacOS-bundled python, from being clobbered by anyone — including root.

To avoid SIP-related issues on a system-wide install, you may want to consider using a different python than the one bundled with MacOS. Using a python virtualenv is one option. Another option for multi-user systems is to install user-specific copies of python + pip + ansible within each appropriate user account (e.g. via brew package manager for MacOS).

A more thorough discussion of issues regarding the use of pip on MacOS systems with SIP can be found here:

Step 9: get started!

Congratulations! You have installed ansible on MacOS!

If you’re new to ansible, check out the documentation at http://docs.ansible.com/ansible/latest/index.html to get started!