Secure AWS Secret Handling

Context

When dealing with runtime secrets in AWS there are a few different considerations:

• Which AWS service to use: AWS Secrets Manager vs AWS Systems Manager Parameter Store vs AWS Key Management Service (KMS)
• How to add the secret values: manually vs via CloudFormation vs via SDK vs via automated rotation
• How to retrieve the secret at runtime: environment injection vs via SDK using role
• How to handle local deployment vs CI/CD deployment

This document shows code examples of deployment of a TypeScript application with AWS infrastructure that is deployed via the TypeScript CDK.

Principles

• Security - We want to ensure that all secrets are kept secure through their lifecycle
• Developer experience - We want to make sure that local development experience is really seamless
• Automated deployments - We want to make sure deployments don't involve manual intervention and are completely automated
• Code-driven CI/CD - We want to minimise manual intervention where possible to set up CI/CD

Implementation

AWS service

AWS Secrets Manager

Allows you to store, rotate, version and retrieve secrets. It uses KMS under the covers and also applies encryption at rest of its own on top of that. It has in-built integration to RDS, Redshift and DocumentDB as well as Lambdas to facilitate automated secret rotation. It has secret auditing (rotation and usage) and fine-grained permissions via IAM. It can also replicate secrets to different regions.

Pros:

• Specifically designed to store and audit secrets i.e. this is what it does
• Provides encryption at rest for the encrypted KMS value i.e. double encryption
• Provides secret rotation capabilities out of the box
• You can access Secrets Manager from a different AWS account
• Supports cross-region replication

Cons:

• It costs $0.40 per secret per month to store secrets (+ really low transaction costs)

AWS Systems Manager Parameter Store

Allows you to store, version and retrieve config data, including secrets. Secrets (SecureStrings) are encrypted using KMS. It has parameter usage auditing and IAM permissions.

Pros:

• Free to store parameters (outside of really low transaction costs)

Cons:

• There is a risk you could misconfigure a secret and it gets stored unencrypted (String rather than SecureString)
• Doesn't support out-of-the-box secret rotation, calling from a different AWS account or cross-region replication

AWS Key Management Service

Allows you to encrypt values using centrally managed keys, but doesn't store the encrypted values themselves. It has usage auditing and IAM permissions.

Pros:

• Core, robust AWS service
• Easy to use Bring Your Own Key

Cons:

• You have to store the encrypted value somewhere to later decrypt it, adding complexity of state management to store the encrypted values and coordinate decrypting them at runtime

Decision

Based on this, as long as you aren't storing a lot of secrets (and cost becomes an issue) or aren't storing a complex set of secret AND non-secret parameters together then it will usually make sense to use AWS Secrets Manager, however all of them are decent options and can be used to implement this pattern.

The rest of this pattern documentation will assume use of AWS Secrets Manager.

How to add the secret values

There's four key ways to set the secret values:

• Manually - create the Secret instances using Infrastructure as Code, but rely on a systems administrator or other suitably authorised person to manually add the secret values
• Via CloudFormaton - set the secret value using CloudFormation; this is not recommended since the secrets end up in plain text and are accessible in the CloudFormation history for suitable authorised people in the AWS console / APIs
• Via SDK - this is a secure, automated way of setting values into the AWS service, you could either programmatically generate the values (if they are random passwords), or get a suitably authorised person to add the values to the CI/CD tool as an encrypted secret as part of initial deployment pipeline setup
• Via automated rotation - If the secret is stored in a user account in an AWS service (e.g. RDS etc.) then you could use the automated secret rotation functionality within AWS Secrets Manager

Doing it via SDK can be done via the following script in conjunction with CDK:

// set-secrets.ts
// You can't set secrets securely without the value leaking in CloudFormation
// This script allows us to output SecretsManager::Secret ARNs by convention
//  and the values populated.

import { AWSError, SecretsManager } from 'aws-sdk'
import { PutSecretValueResponse } from 'aws-sdk/clients/secretsmanager'
import { readFileSync } from 'fs'

var client = new SecretsManager({
  region: process.env.AWS_DEFAULT_REGION,
})

// Once CloudFormation runs all outputs are written to cdk-outputs.json
const cfnOutputBuffer = readFileSync('./cdk-outputs.json')
const cfnOutputJson = cfnOutputBuffer.toString()
const cfnOutputs = JSON.parse(cfnOutputJson)

// Pull out any CloudFormation outputs that end in "SECRETARN"
const outputs: any[] = []
Object.keys(cfnOutputs).forEach((stackKey) => {
  Object.keys(cfnOutputs[stackKey])
    .filter((outputKey) => {
      return /secretarn$/i.test(outputKey)
    })
    .forEach((outputKey) => {
      outputs.push(JSON.parse(cfnOutputs[stackKey][outputKey]))
    })
})

// The format of the CloudFormation output should be a string with JSON of format: {arn: 'arn of secret', environmentKey: 'environment variable key of the secret value'}
const setSecretsResults = outputs.map((output) => {
  return new Promise<PutSecretValueResponse | null>((resolve, reject) => {
    if (!process.env[output.environmentKey]) {
      resolve(null)
      return
    }

    client.putSecretValue({ SecretId: output.arn, SecretString: process.env[output.environmentKey] }, (err, data) => {
      if (err) {
        reject(err)
        return
      }
      resolve(data)
    })
  })
})

;(async () => {
  await Promise.all(setSecretsResults).then(
    (successes) => {
      successes
        .filter((s) => s !== null)
        .map((s) => s as PutSecretValueResponse)
        .forEach((success: PutSecretValueResponse) => {
          console.log(`Secret set for ${success.Name} with version ${success.VersionId}`)
        })
    },
    (failures: any) => {
      if (failures instanceof Array) {
        failures.forEach((failure: AWSError) => {
          console.error(`Secret set FAILED: [${failure.code}] ${failure.name}: ${failure.message}`)
        })
        throw 'Failed'
      } else {
        throw failures
      }
    }
  )
})()

The way this works is by extracting Secrets Manager secret ARN identifiers that are output from CDK (using the cdk-outputs.json file) with a convention that these variables end in SecretARN (case insensitive) and then sets the value in process.env.{SECRET_NAME} to the secret using the Secrets Manager SDK. It also relies on the AWS account that is present in the environment variables (i.e. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) to have permission to write the secret (which it should if it's running the same user as the one that executed the CDK).

To achieve this, you can use code like this in your CDK stack:

const passwordSecret = new secrets.Secret(this, 'PASSWORD', {
  secretName: `/${id}/PASSWORD`,
})

// ...

new cdk.CfnOutput(this, 'passwordSecretArn', {
  exportName: `${id}-passwordSecretArn`,
  value: JSON.stringify({ arn: passwordSecret.secretArn, environmentKey: 'PASSWORD' }),
})

Notes:

• The name of the secret in this example is called password
• This code uses a convention that resource names are prefixed with the stack id (id)
• There is a convention that the name of the secret is used (in UPPERCASE) to identify it (and in our code that we name the runtime environment variable with the same name) and that same value is prefixed (in camelCase) to SecretArn as the exported name (that ends up in cdk-outputs.json)
• The value output from CDK is an object with the following structure (which the set-secrets.ts file picks up):

  {
    "arn": "{the-secret-arn}",
    "environmentKey": "{they-UPPERCASE-environment-variable-name}"
  }
  

• In order to ensure the CDK outputs are in a cdk-outputs.json file you should:
  • Pass --outputs-file ./cdk-outputs.json to the npm run cdk -- deploy call
  • Ensure cdk-outputs.json is in your .gitignore file so it's never accidentally committed
• Ensure that environment variables are set with the secret values when calling the script, e.g. for the above example process.env.PASSWORD should have the secret value
• This set-secrets.ts file should be called from the CI/CD deployment pipeline after running CDK deploy (ensuring all of the secrets are set into environment variables)

How to retrieve the secret at runtime

In order to retrieve the secrets at runtime you need to make an SDK call to Secrets Manager to retrieve the secrets.

Using AWS Lambda you can do this by convention by setting the ARN of the secret in environment variables and then retrieving the secret value using the ephemeral AWS Lambda role (assuming you grant read access to the secret for the Lambda).

To set that up in CDK you need something like this (continuing from the above example):

const myLambda = new lambda.Function(this, '{lambdaName}', {
  // ...
  environment: {
    PASSWORD_ARN: passwordSecret.secretArn,
    // ...
  },
})

passwordSecret.grantRead(myLambda)

Then we can use the following function in the Lambda itself to get hold of the secret value (assuming you have done an npm install aws-sdk @types/aws-sdk):

import { SecretsManager } from 'aws-sdk'

export async function getSecret(secretArn: string): Promise<string> {
  var client = new SecretsManager({
    // This is automatically specified with in the Lambda runtime environment
    //  https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime
    region: process.env.AWS_REGION,
  })

  return new Promise((resolve, reject) => {
    client.getSecretValue({ SecretId: secretArn }, (err, data) => {
      if (err) {
        console.log(JSON.stringify(err))
        reject(err)
        return
      }

      if ('SecretString' in data) {
        resolve(data.SecretString as string)
      } else {
        resolve(Buffer.from(data.SecretBinary as any, 'base64').toString('ascii'))
      }
    })
  })
}

To use that function you can use code like this in your Lambda entrypoint:

process.env.PASSWORD = await getSecret(process.env.PASSWORD_ARN as string)

This approach allows you to always use process.env.PASSWORD to access the secret so the complexity of switching how you retrieve the secret is confined to the code entrypoint rather than the main body of code (e.g. lambda-handler.ts in AWS using Secrets Manager and index.ts for local development with value in a .gitignore'd .env file, etc.).

How to handle local deployment vs CI/CD deployment

Local CDK deployment

When deploying the CDK from a developer machine you want to make use of a .gitignore'd.env to store the secrets and then call set-secrets.ts after doing the cdk deploy. If you are using npm scripts in package.json it might look something like this (assuming you have done an npm install --save-dev dotenv-cli ts-node):

{
  // ...
  "scripts": {
    "deploy:local": "dotenv -e .env -- npm run deploy && npm run set-secrets:local",
    "deploy": "npm run cdk -- deploy \"*\" --ci --require-approval never --outputs-file ./cdk-outputs.json",
    "set-secrets:local": "dotenv -e .env -- ts-node --transpile-only set-secrets.ts"
    // ...
  }
  // ...
}

Then to allow breakpoint debugging you can add a launch configuration in .vscode/launch.json with something like (assuming the CDK project is in a folder called infrastructure):

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Deploy AWS",
      "type": "node",
      "request": "launch",
      "cwd": "${workspaceFolder}/infrastructure",
      "console": "integratedTerminal",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "deploy:local"],
      "preLaunchTask": "npm: cdk bootstrap - /infrastructure"
    },
    {
      "name": "Debug AWS set-secrets.ts script",
      "type": "node",
      "request": "launch",
      "cwd": "${workspaceFolder}/infrastructure",
      "console": "integratedTerminal",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "set-secrets:local"]
    }
    // ...
  ]
}

Technically, you can leave out the separate launch configuration for set-secrets, but if you want to debug that code quickly without waiting for CDK to run it's handy.

The preLaunchTask for cdk bootstrap is optional depending on whether your CDK configuration requires CDK bootstrap to have been run. To make this work, you need something like the following in your .vscode/tasks.json:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "npm: cdk bootstrap - /infrastructure",
      "type": "npm",
      "script": "bootstrap:local",
      "path": "${workspaceFolder}/infrastructure",
      "options": {
        "cwd": "${workspaceFolder}/infrastructure"
      },
      "dependsOn": ["npm: install - /infrastructure"],
      "problemMatcher": []
    },
    {
      "label": "npm: install - /infrastructure",
      "type": "npm",
      "script": "install",
      "path": "${workspaceFolder}/infrastructure",
      "options": {
        "cwd": "${workspaceFolder}/infrastructure"
      },
      "problemMatcher": []
    }
  ]
}

The npm:install bit is optional, but it's a nice way of making the whole thing able to be clone -> F5'd without executing other commands like npm install (acknowledging that you also need to set up the .env file first).

For this to work you also need a bootstrap:local npm script, which can look something like this (which requires a npm install cross-env):

{
  // ...
  "scripts": {
    "bootstrap:local": "dotenv -e .env -e ../nft-minter/.env -- npm run bootstrap",
    "bootstrap": "cross-env-shell npm run cdk -- bootstrap aws://$CDK_DEFAULT_ACCOUNT/$AWS_DEFAULT_REGION"
    // ...
  }
  // ...
}

As well as any secrets, the above configurations need the following variables defined in the .env file, as a suggestion, include these values in .env.sample and add .env to the .gitignore file:

CDK_DEFAULT_ACCOUNT={ACCOUNT_NUMBER}
AWS_DEFAULT_REGION={AWS_ENVIRONMENT e.g. us-east-1}
AWS_ACCESS_KEY_ID={ACCESS_KEY_ID} # https://console.aws.amazon.com/iam/home#/security_credentials > Create access key
AWS_SECRET_ACCESS_KEY={ACCESS_KEY_SECRET} # https://console.aws.amazon.com/iam/home#/security_credentials > Create access key

CI/CD CDK deployment

Assuming that you have a Continuous Integration step that compiles TypeScript to JavaScript then the way you execute set-secrets.ts on an automated deployment will be slightly different than locally.

The suggested way to tackle this is to include a non-local set-secrets npm script and a step in the deployment pipeline to call it, e.g.:

{
  // ...
  "scripts": {
    "set-secrets": "node set-secrets.js"
    // ...
  }
  // ...
}

And then something like the following template via your deployment pipeline (this example using Azure DevOps YAML):

- task: AWSShellScript@1
  displayName: 'set-secrets'
  inputs:
  awsCredentials: $(AWS_SERVICE_CONNECTION)
  regionName: $(AWS_DEFAULT_REGION)
  scriptType: 'inline'
  inlineScript: 'npm run set-secrets'
  workingDirectory: 'infrastructure'
  disableAutoCwd: true
  env:
    'PASSWORD': '$(PASSWORD)'

Notes:

• The need to explicitly pass in PASSWORD via env is needed for Azure DevOps because secrets aren't added as an environment variable by default so you have to manually set them in.
• This assumes that AWS_SERVICE_CONNECTION, AWS_DEFAULT_REGION and PASSWORD are set and that PASSWORD is a secret.