Introduction to @makerx/cloudwatch-error-alarms

At MakerX, we use a pattern where exceptions in our AWS services will result in Slack alert messages being posted in dedicated channels so that they can be easily tracked and acted on.

TL;DR: we built this cool NPM package to send AWS alerts to Slack https://www.npmjs.com/package/@makerx/cloudwatch-error-alarms

How does it work?

To achieve this, there are 2 components

1. The Cloudwatch Error Alarms lambda

It subscribes to an AWS CloudWatch log group. Log events will be parsed and filtered then sent to Slack.

A sample Slack alert would look like

Lambda function {the function name} is failing
Time: Mon Aug 08 2022 11:55:19 GMT+0000 (Coordinated Universal Time)
Log Link: https://{aws-region}.console.aws.amazon.com/cloudwatch/home?region={aws-region}#logsV2:log-groups/log-group/{log-group-id}
Message:
---
The error message
---

This gives the dev team enough information to act on the alarm, including:

• A quick look at the error message
• A link to the log so that they can do further investigation

2. AWS CDK infrastructure

We love AWS CDK. It allows us to define our infrastructure as code. Below is the code to deploy the above lambda

export class CloudWatchErrorAlarmLambda extends Function {
  constructor(scope: Construct, id: string, props: CloudWatchErrorAlarmLambdaProps) {
    const environment: Record<string, string> = {
      SLACK_WEBHOOK_URL: props.slackWebhookUrl,
      FUNCTION_NAME: props.erroringFunctionName,
    }
    props.errorFilterRegexes?.forEach((filter, index) => {
      environment[`ERROR_FILTER_REGEX_${index + 1}`] = filter
    })

    const pathToLambda = path.join(__dirname, 'lambda')
    super(scope, id, {
      code: Code.fromAsset(pathToLambda),
      functionName: props.functionName,
      handler: 'index.handler',
      memorySize: 128,
      runtime: Runtime.NODEJS_16_X,
      timeout: Duration.seconds(60),
      retryAttempts: 2,
      environment: environment,
    })
  }
}

export type CloudWatchErrorAlarmLambdaProps = {
  // Slack webhook 
  slackWebhookUrl: string
  functionName: string
  // The function name that caused the error
  erroringFunctionName: string
  // Regular expressions to ignore error messages
  errorFilterRegexes?: string[]
}

Pulling them together

To make it convenient for everyone, we put everything together into this NPM package https://www.npmjs.com/package/@makerx/cloudwatch-error-alarms

For example, if we need to track a lambda, we only need to

import { CloudWatchErrorAlarmLambda } from '@makerxstudio/cloud-watch-error-alarm'

const lambda = new lambda.Function(...)

const errorsLambda = new CloudWatchErrorAlarmLambda(this, `${id}-cloud-watch-error-alarms`, {
  // The function name that caused the error, this will be included in the Slack message
  erroringFunctionName: `${erroringFunctionName}`,
  // The cloud watch error alarm lambda function name
  functionName: `${id}-cloud-watch-error-alarms`,
  // Slack webhook https://slack.com/intl/en-au/help/articles/115005265063-Incoming-webhooks-for-Slack
  slackWebhookUrl: `${slackWebhookUrl}`, 
  errorFilterRegexes: [
    // Regex to ignore error messages
  ],
})

// Allow cloud watch to trigger the alarm lambda on error
lambda.logGroup.addSubscriptionFilter(`${id}-cloud-watch-error-alarms-subscription`, {
  destination: new destinations.LambdaDestination(errorsLambda),
  filterPattern: FilterPattern.stringValue('$.level', '=', 'error'),
})

We would love to hear what you think about it and pull requests are always welcome.

Happy coding!