AWS Organizations operator framework

AWS Organizations operator framework

Operate a AWS organization often lead to the need to create scripts to inventory resources (EC2 instances, RDS instances, S3 buckets…), remediate tags, delete unused resources…. There is no

When it comes to run these operation scripts accross an AWS organization, parallelization is the key te speed up the process. Sequential processing of the accounts and regions in a unique script can take hours to finish. Thanks to Step Functions and Lamba, we can process all accounts and operated regions in minutes from an operator account. Cherry on the cake, it’s serverless and it almost does not incur additional costs.

Cross-account roles

In order to execute actions in other accounts, we deploy cross-account roles in all accounts via Cloudformation Stackset. These cross-accounts roles are assumed by operation lambas.

Clouformation template for cross-account roles deployed organization-wide via StackSet

AWSTemplateFormatVersion: 2010-09-09
Description: Cross account roles for operations

Parameters:

  OperationAccountId:
    Type: String
    Description: Account ID of the account where the role will be created

Resources:
  
  OperationAdmin:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: operation-admin
      AssumeRolePolicyDocument:
        Statement:
          - Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${OperationAccountId}:role/operation-admin-lambda'
        Version: 2012-10-17
      Description: Ops admin role assumed by Lambda functions from Operations Accounts
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AdministratorAccess'

  OperationReadonly:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: operation-readonly
      Path: '/'
      Description: Ops readonly role assumed by Lambda functions from Operations Accounts
      AssumeRolePolicyDocument:
        Statement:
          - Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${OperationAccountId}:role/operation-readonly-lambda'
      ManagedPolicyArns:
       - 'arn:aws:iam::aws:policy/ReadOnlyAccess'

Clouformation template for lambda roles deployed only in the operation account

AWSTemplateFormatVersion: 2010-09-09
Description: Cross account lambda roles for operations
  
Resources:

  OperationAdminLambda:
    Type: AWS::IAM::Role
    Properties:
      RoleName: operation-admin-lambda
      Path: "/"
      Description: "Allows lambda to assume role operation-admin in target accounts"
      AssumeRolePolicyDocument: 
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
      Policies:
        - PolicyName: "assume-role"
          PolicyDocument:
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Resource:
                  - arn:aws:iam::*:role/operation-admin
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  OperationReadonlyLambda:
    Type: AWS::IAM::Role
    Properties:
      RoleName: operation-readonly-lambda
      Path: "/"
      Description: "Allows lambda to assume role operation-readonly in target accounts"
      AssumeRolePolicyDocument: 
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
      Policies:
        - PolicyName: "assume-role"
          PolicyDocument:
            Statement:
              - Action:
                  - sts:AssumeRole
                Effect: Allow
                Resource:
                  - arn:aws:iam::*:role/operation-readonly
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Step machines

The step machines is split into 3 parts.

1. The Lambda fanout

It aims is to fetch the account list from the organization master account, and build an array of object for each combinaison account/region where we want to execute our operation script.

Output example:

{
  "AccountRegions": [
    {
      "AccountId": "2345678765434545343432",
      "AccountName": "project-1-prod",
      "Region": "eu-west-1"
    },
    {
      "AccountId": "2345678765434545343432",
      "AccountName": "project-1-prod",
      "Region": "us-east-1"
    },
    {
      "AccountId": "987654345567575364564",
      "AccountName": "project-2-dev",
      "Region": "eu-west-1"
    },
    {
      "AccountId": "987654345567575364564",
      "AccountName": "project-2-dev",
      "Region": "us-east-1"
    },
    ...
  ]
}

The object could contains also other properties required by the worker lambda.

2. The workers map

The map that iterate through the array to execute a Lambda worker with concurrency. As I’m writing this article, the maximum execution is 50. This step can contains at least one Lambda, but you can add tasks if you need to split your process even more.

3. The result handler

As a last step, we can export the results of the map as a csv file into S3, send an email, a slack notification….

How to handle the payload limit

As your organization or the amount of resources grows, you will reach the 256 KB size limit of the task outputs. It often occurs after the map step because it joins the output of all the Lambda workers. FYI, I suppose that this limit is related to the Lambda asynchronous invocation payload limit.

Fanout output

Let’s say you have common parameters for the Lamba workers. For example a list of resource ids such as Security Hub standard arns. Repeating these arns in the AccountRegions items would generate a too big payload. To only have them once, we will leverage the parameters property of the map to build the input payload for each concurrent workers.

Fanout output :

{
  "StandardArns": [
    "arn:aws:securityhub:REGION::standards/aws-foundational-security-best-practices/v/1.0.0",
    "arn:aws:securityhub:REGION::standards/cis-aws-foundations-benchmark/v/1.4.0"
  ],
  "AccountRegions": [
    {
      "AccountId": "2345678765434545343432",
      "AccountName": "project-1-prod",
      "Region": "eu-west-1"
    },
    {
      "AccountId": "2345678765434545343432",
      "AccountName": "project-1-prod",
      "Region": "us-east-1"
    },
    {
      "AccountId": "987654345567575364564",
      "AccountName": "project-2-dev",
      "Region": "eu-west-1"
    },
    {
      "AccountId": "987654345567575364564",
      "AccountName": "project-2-dev",
      "Region": "us-east-1"
    },
    ...
  ]
}

Map definition :

Type: Map
ItemsPath: $.AccountRegions
Parameters:
  AccountId.$: $$.Map.Item.Value.AccountId
  AccountName.$: $$.Map.Item.Value.AccountName
  Region.$: $$.Map.Item.Value.Region
  StandardArns.$: $.StandardArns
Iterator:
  StartAt: inventory
  States:
    inventory:
      End: 'true'
      Resource: arn:aws:lambda:eu-west-1:23456765432332435:function:securityhubInventory-47IgsXHr66xW
      Type: Task
MaxConcurrency: 0
Next: export

Input worker example:

{
  "AccountId": "2345678765434545343432",
  "AccountName": "project-1-prod",
  "Region": "eu-west-1",
  "StandardArns": [
    "arn:aws:securityhub:REGION::standards/aws-foundational-security-best-practices/v/1.0.0",
    "arn:aws:securityhub:REGION::standards/cis-aws-foundations-benchmark/v/1.4.0"
  ]
}

Workers map output

One solution is to push the data in dynamoDB or S3, and read it from the last step.

You can either push your data to dynamoDB from your lambda or via the “arn:aws:states:::dynamodb:putItem” resource from Step Functions.

Task definition :

Type: Task
Resource: arn:aws:states:::dynamodb:putItem
Parameters:
  Item:
    accountId:
      S.$: $.accountId
    region:
      S.$: $.region
    standards:
      S.$: $.standards
    status: 
      S.$: $.status
  TableName: MyInventoryTable
OutputPath: $.SdkHttpMetadata.HttpStatusCode
End: true

Notice the OutputPath to minimize the output payload size.

What’s next?

It depends on your needs. Step functions is a very powerfull orchestrator. Here more example of scripts :

  • Inventories : unused EBS volumes, unused NAT gateways, lambda functions, VPC, accounts, Cloudfront distribution, tags, public IPs, public endpoints
  • Compliance : tags remediation, security groups open ingress remediation
  • Security : security hub standards and controls management, Inspector activation

Code example available on github.

Start free trial