Secure AWS accounts using AWS IAM Access Analyzer

tl;dr, a hands-on post on how to secure AWS accounts using AWS IAM Access Analyzer by detecting the least privileges for service accounts.

Quality (and security) in our DNA

In MakerX, we build quality into every aspect of product development and security is no exception. Since inception, we follow strong security practices. Psssst: don't forget to have a look at Rob's blog post "Secure AWS secret handling with TypeScript CDK"

We structured our AWS tenant based on AWS Well-Architected framework. We implemented many recommendations such as, just to name a few:

• Rely on a centralized identity provider
• Configure service and application logging
• Manage accounts centrally using an AWS organisation
• Separate workloads using accounts
• Grant least privileges access

"Grant least privileges access" Principle

In this blog post, we will deep dive into the last point "Grant least privileges access". In a nutshell, it means that accounts should be permitted to only perform actions to perform their duties. The accounts can be used by humans or systems. If you are not familiar with system/service accounts, they are accounts used by systems to communicate with other systems.

This principle sounds more important when we talk about non-human accounts. Why? because of the nature and duties of these accounts, they have higher privileges to create, configure, destroy services. They will become more attractive to attackers and thus need more attention (source, source).

Use Case Background

Our cloud-native environment in AWS includes a mix of services and we use AWS CDK as Infrastructure as Code (IaC) plus a CI/CD to deploy to with every merge to the main branch. This continuous deployment uses our deployment service account which will refer to it from now on as our DevOpsAccount.

How to find the least privileges

The way to find the least privileges is painful by trial and error or by exploring what services and permissions are required for your environment. I guarantee that this might take hours or days depending on the size of your system. AWS recently provided a solution by monitoring the IAM account using the AWS IAM Access Analyzer. Access Analyzer is a relatively new AWS service that allows you to record a user action using AWS CloudAudit and then it generates an IAM policy.

We decided to use AWS Access Analyzer to help us understand what actions are performed by our DevOpsAccount.

Note that you may be charged for using the AWS services below. Please make sure that you understand each service pricing. Some services may have free-tier. find out more here: https://aws.amazon.com/pricing/

Let's walk through how we created our DevOpsAccount Policy template from environment preparation, recording, performing day-to-day duties, generating the policy, and finally testing and applying the newly-generated policy.

A. Environment preparation

• The first step is to find where to run our experiment. We decided to create an AWS sandbox environment. It was easy for us to do so as we are using AWS Org and creating a new account is an easy and manageable task. Note: you don't have to do that. You can run it in any of your AWS accounts.
• Make sure that you have a clean canvas. If you are using an existing account, make sure that you remove all application-related services in your AWS account. If you are using CDK, first destroy all CDK stacks
• If you are using CDK, manually delete any CDK related services as well i.e. CloudFormation bootstrap stack (normally called "CDKToolkit") and CDK S3 bucket normally called the name started with "cdk-..." There is as there is no CLI command to destroy CDK yet (source).
• Create the DevOpsAccount and attach it to AWS managed AdministratorAccess policy. Given the risk of that access level, we are doing these steps in a sandbox isolated from our other accounts.
• In case you don't have a specific application deployment to run this experiment against, you can create a sample CDK and application this tutorial: AWS sample https://cdkworkshop.com/20-typescript.html.
• Connect AWS account to the deployment agent. You have two options here either to use your day-to-day DevOps Service to point to this designated AWS environment or to use an isolated setup. I preferred the latter. I connected my machine to the designated AWS account/region using AWS user DevOpsAccount using AWS access key/secret combination to simulate the actual flow.

cdk destroy [STACKS..]

B. Start recording using AWS CloudTrail

• After you finish preparing the environment, we need to start recording the user events using CloudTrial. To create it, go to AWS Console in your account, go to the CloudTrial service and select your desired region. Go to Create Trail and give the Trial a name, e.g. LeastPrivilegesTrail. Use the default setup and make sure to include Management events and API (both read and write).

If you need to know more about AWS CloudTrial go to: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html.

C. Perform/simulate day-to-day duties

In this step, we will simulate what the DevOpsAccount account does. This includes all activities such as deploying brand new services, redeploying a new version or tearing down the environment.

• As we use CDK, I performed the following terminal commands:
• To simulate updating the environment, I made cosmetic changes to the application code to trigger a new CDK deployment i.e. CloudFormation changeset. And then I run
• To simulate the tearing down the application stacks, I run

# AWS CDK Bootstrap 
cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$AWS_DEFAULT_REGION"

# cd into your infrastructure disctory (where all CDK stack)
# Deploy pre-defined CDK stacks
cdk deploy

# approve your deployment and wait till it finishes

// to make sure there is a new CDK deployment
cdk diff

// trigger new deployment
cdk deploy

// to destroy the stacks 
cdk destroy

# approve your destroy and wait till it finishes

D. Generate the recommended policy

It is time to use IAM Access Analyzer to scan the CloudTrail to find out what are the actions performed by the DevOpsAccount account. You can do this by going to AWS console -> IAM service -> Users -> DevOpsAccount user-> Permissions tab. then Click "Generate policy". and configure the required fields as follows

• Time: it should cover the time when you run the day-to-day activities.
• CloudTrail: select the region where you created the "LeastPrivilegesTrail" trail and the trail. For the region, I selected all regions to make sure that my generated policy will cover all global AWS services such as S3.
• And for the IAM Role to query CloudTrail, I selected "Create and use a new service role"

Click generate and this will take you to the user IAM screen. Wait till your policy generation ends successfully and click "View Generated Policy". The screen will show what are the required permissions (AWS services used and the action performed by that specific user). And you will have the option to add permission more if you like.

E. Generate and test the policy

Now AWS will give you the permissions based on the action of your DevOpsAccount. Please copy and paste the newly generated policy template into your IDE to save it in your codebase.

Please find a sample of the newly generated policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:DescribeRepositories",
        "s3:ListAllMyBuckets"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:AttachRolePolicy",
        "iam:CreateRole",
        "iam:GetRole",
        "iam:GetRolePolicy",
        "iam:PutRolePolicy",
        "sts:AssumeRole"
      ],
      "Resource": "arn:aws:iam::${Account}:role/${RoleNameWithPath}"
    },
    {
      "Effect": "Allow",
      "Action": "kms:GenerateDataKey",
      "Resource": "arn:aws:kms:${Region}:${Account}:key/${KeyId}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucketPolicy",
      ],
      "Resource": "arn:aws:s3:::${BucketName}"
    }
   ]
}

Now, replace the placeholders with your values for example:  ${Region} with your region code, ${Account} with your account number etc. And now save it in your AWS account as  DevOpsLeastPrivilegePolicy.

Now, it is time to test the new policy. Go to your DevOps user and remove the policies and any permissions previously assigned to this user. Assign only the newly-generated policy to this user. The next step is to perform the same action we did earlier to simulate day-to-day and by using your DevOps pipeline to redeploy your application to see if the new policy is enough for the user to perform the job.

Why were the Cloudformation permission statements missing? We found from the CloudTrail logs that the CDK using Cloudformation executes via a different username starting with aws-cdk- instead of the DevOpsAccount user. I'm not sure why this is and I am happy for someone to shed some light here for me!

The section that we had to add to our policy to run cdk/CloudFormation is as follows:

{
	"Sid": "CloudformationPolicy",
	"Effect": "Allow",
	"Action": [
        "cloudformation:CreateChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStacks",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DeleteStack",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:GetTemplate"
	],
	"Resource": ["arn:aws:cloudformation:${Region}:${account}:stack/*/*"]
}

F. Applying the new policy

After testing the newly-generated policy, it is time to apply it to your AWS policy to the DevOpsAccount and save the template in your Git repository.

A sample of a policy template is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "IAMPolicy",
      "Effect": "Allow",
      "Action": [
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy"
      ],
      "Resource": ["arn:aws:iam::123456789:role/*"]
    },
    {
      "Sid": "S3Policy",
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucketPolicy",
        "s3:GetBucketPolicy",
        "s3:PutBucketPublicAccessBlock",
        "s3:PutBucketPolicy",
        "s3:PutBucketVersioning",
        "s3:PutEncryptionConfiguration",
        "s3:ListAllMyBuckets"
      ],
      "Resource": ["arn:aws:s3:::*"]
    },
    //..... more services
    {
      "Sid": "CloudformationPolicy",
      "Effect": "Allow",
      "Action": [
        "cloudformation:CreateChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStacks",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DeleteStack",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:GetTemplate"
      ],
      "Resource": ["arn:aws:cloudformation:${AWS_Region}:${AWS_account}:stack/*/*"]
    }
  ]
}

Cleaning up

Don't forget to delete the services you created as part of this guide. You might need to delete the

• The CDK stack (if you used a sandbox environment)
• The newly-generated AWS IAM Policy
• CloudTrial trail and associated services such as S3 bucket and the IAM role

The verdict

AWS Access Analyzer is a powerful tool to find out the least privileges to secure your AWS accounts. It's easy to set up and the outcome is exactly what you are looking for with the caveat that CloudFormation permissions are not recorded.