Blue/Green Deployment with PM2 and nginx

I'm part of an interesting commitment group (https://listskit.com/commit365/), I'm not that active as I should, but recently an interesting question came up about deployments with zero downtime. I've got a small set of scripts to automate blue/green deployments with nginx and Node.js, and I thought I share them here.

Let's talk about the basic setup first. I've got one server running a Node.js (Next.js) app for production and test. All processes are managed by pm2. Nginx is also running on the same machine as a reverse proxy (there is a Cloudflare layer in front too, but that's not relevant for the deployment setup).

For the process of deploying, I've written two small Node.js scripts, although I think everything could be done via bash only too. But I'm getting ahead of myself, we'll start with the base setup.

So let's start with the same git repository in two different directories.

/opt/app/production/blue
/opt/app/production/green

Let's assume our app runs on port 3000. We need to configure both repositories with a different port (ex. 3000 and 3001) so both can run at the same time. We also assume that our app has a health Endpoint that returns the current deployment (blue or green, I'll share some code later).

Here are now two sample pm2 configurations to start both apps.

Blue:

module.exports = {
  apps: [
    {
      name: 'app-production-blue',
      port: 3000,
      exec_mode: 'cluster',
      instances: 4,
      script: './node_modules/.bin/next',
      cwd: '/opt/app/production/blue',
      args: 'start',
      env: {
        NODE_ENV: 'production',
      },
      log_date_format: 'YYYY-MM-DDTHH:mm:ss.SSS',
    },
  ],
}

Green:

module.exports = {
  apps: [
    {
      name: 'app-production-green',
      port: 3001,
      exec_mode: 'cluster',
      instances: 4,
      script: './node_modules/.bin/next',
      cwd: '/opt/app/production/green',
      args: 'start',
      env: {
        NODE_ENV: 'production',
      },
      log_date_format: 'YYYY-MM-DDTHH:mm:ss.SSS',
    },
  ],
}

We can start and stop each app with:

pm2 start app-production-blue
pm2 stop app-production-blue

pm2 start app-production-green
pm2 stop app-production-green

We also need a small nginx setup:

http {
    include /opt/nginx/current.conf; ## holds the correct upstream (blue or green)
    
    server {
        listen 80;
        listen [::]:80;

        server_name domain.tld;

        location / {
            return 301 https://$host$request_uri;
        }
    }

    server {

            listen 443 ssl http2;
            listen [::]:443 ssl http2;

            server_name domain.tld;

            # Reverse Proxy Settings
            location / {
                    proxy_pass http://app; ## proxy to the current upstream
            }
    }
}

As you can see, we include a config file at "/opt/nginx/current.conf". In that file, we store the "upstream" directive of nginx and every time we switch deployments, we update that file, to load one of the two following files.

/opt/nginx/blue.conf

upstream app {
    server 127.0.0.1:3000;
}

or /opt/nginx/green.conf

upstream app {
    server 127.0.0.1:3001;
}

Now, here is an example file on how you could switch from one deployment to another.

;(async () => {
  console.log('Switching Environments')

  const currentDeployment = await getCurrentDeployment() // either reads from the nginx config or calls an endpoint that returns the current deployment (blue or green)
  const nextDeployment = currentDeployment === 'blue' ? 'green' : 'blue'

  console.log('Current Deployment:', currentDeployment)
  console.log('Next Deployment:', nextDeployment)

  const nginxConfPath = join('/opt/nginx/current.conf')

  writeFileSync(nginxConfPath, `include /opt/nginx/${nextDeployment}.conf;`)

  console.log('Checking nginx Config...')

  const { stderr } = await promiseExec('sudo nginx -t')

  if (
    !stderr.includes('test is successful') &&
    !stderr.includes('syntax is ok')
  ) {
    console.error(stderr)
    throw new Error('Nginx Config is not valid: ' + stderr)
  }

  exec(`pm2 restart app-production-${nextDeployment}`) // start new environment

  console.log('Reloading nginx...')

  exec('sudo systemctl reload nginx')

  console.log('Nginx Reload done.')

  exec(`pm2 stop app-production-${currentDeployment}`) // stop existing environment

  console.log(`${hostname()}: switched to *${nextDeployment}*!`)
})()

Now that we can switch deployments around anytime we want, we can integrate that into our deployment script.

;(async () => {
  console.log('Deploying')

  const { version } = await getHealthResponse()

  console.log('Current Version:', version)

  const currentDeployment = await getCurrentDeployment()
  const deploymentToUpdate = currentDeployment === 'blue' ? 'green' : 'blue'

  console.log('Current Deployment:', currentDeployment)
  console.log('Deployment to update:', deploymentToUpdate)

  const workingDirectory = join('/opt/app/production/', deploymentToUpdate)

  console.log('Working Directory:', workingDirectory)

  exec(`pm2 stop app-production-${deploymentToUpdate}`) // stop in case it's running

  exec('git pull', { cwd: workingDirectory })

  exec('npm ci', { cwd: workingDirectory })

  exec(`npm run build`, { cwd: workingDirectory })

  exec(`pm2 restart app-production-${deploymentToUpdate}`)

  console.log('Updated, switching deployments...')

  exec(`./switch-env.sh')

  const { version: vn } = await getHealthResponse()

  console.log(
    `${hostname()}: updated *${deploymentToUpdate}* (old version: ${version}, new version: ${vn})!`,
  )

  console.log('Deployment done.')
})()

Obviously, everything is in pseudocode (Node.js based), but I think it should suffice to craft your own scripts from it.