AWS Terraform Account Closure Automation

Automate closing and suspending AWS accounts in an AWS Control Tower environment using Terraform and a Lambda function triggered by AFT audit trail DynamoDB streams.

Highlights

• Event-driven: reacts to AFT audit trail stream events
• Cross-account safe: assumes a minimal CT role for Organizations and Service Catalog actions
• Idempotent: tracks Service Catalog termination with record polling and guarded retries
• Observable: structured CloudWatch logging with configurable retention

---

Repository structure

aws-tf-close-account/
├── .gitlab-ci.yml                     # GitLab pipeline (plan/apply/destroy)
├── close-and-suspend/
│   ├── configuration/                 # Root Terraform stack that consumes the module
│   │   ├── main.tf
│   │   ├── provider.tf
│   │   └── variables.tf
│   └── module/                        # Terraform module (Lambda, IAM, Events)
│       ├── data.tf
│       ├── iam-ct.tf
│       ├── iam..tf                    # Lambda role + inline policy
│       ├── lambda.tf
│       ├── variables.tf
│       ├── versions.tf
│       └── lambda/
│           └── aft-close-account.py   # Lambda implementation
└── images/

What it does (workflow)

1. DynamoDB stream event from AFT audit trail triggers the Lambda.
2. Lambda looks up the target account ID in the AFT metadata table (by email index).
3. Lambda assumes the CT management account role aft-account-closure-role.
4. It terminates the Service Catalog provisioned product for the account (with guarded retry),
5. Moves the account to the configured SUSPENDED OU,
6. Then calls organizations:CloseAccount to close it.

Requirements

• AWS Control Tower and AFT are deployed and operational
• A DynamoDB audit table with streams enabled for AFT (stream ARN provided to Terraform)
• A KMS key ARN that encrypts the audit stream (provided to Terraform)
• SSM parameter /aft/resources/ddb/aft-request-metadata-table-name resolvable in the AFT account
• IAM trust and permissions for cross-account role assumption
• Tools:
  • Terraform >= 0.15.0
  • AWS CLI
  • Python 3.11 (Lambda runtime)

Terraform at a glance

The root stack is under close-and-suspend/configuration, and it consumes the module in close-and-suspend/module.

Backend and providers (from configuration/provider.tf)

terraform {
  required_version = ">= 0.15.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.15"
    }
  }
  backend "s3" {
    bucket               = "aaws-tf-close-account-statefile"
    key                  = "offboarding-module.tfstate"
    region               = "eu-west-1"
    use_lockfile         = true # note: consider DynamoDB for reliable locking
    encrypt              = true
    workspace_key_prefix = "offboarding-module"
  }
}

provider "aws" {
  region = var.region
}

# AFT account
provider "aws" {
  alias  = "aft"
  region = var.region
  assume_role {
    role_arn    = "arn:aws:iam::123456789012:role/AWSAFTExecution"
    external_id = "ASSUME_ROLE_ON_TARGET_ACC"
  }
}

# CT account
provider "aws" {
  alias  = "ct"
  region = var.region
  assume_role {
    role_arn    = "arn:aws:iam::210987654321:role/AWSAFTExecution"
    external_id = "ASSUME_ROLE_ON_TARGET_ACC"
  }
}

Module usage (from configuration/main.tf)

module "offboarding_lambda" {
  source                                   = "../module"
  cloudwatch_log_group_retention           = "90"
  region                                   = "eu-west-1"
  aft_account_id                           = "123456789012"
  ct_account_id                            = "210987654321"
  ct_destination_ou                        = "ou-euup-d1e061ao"
  ct_root_ou_id                            = "r-cuup"
  aft-request-audit-table-encrption-key-id = "arn:aws:kms:eu-west-1:123456789012:key/5c9e23e2-3e83-4fe6-98ec-b87f11c772fb"
  aft-request-audit-table-stream-arn       = "arn:aws:dynamodb:eu-west-1:123456789012:table/aft-request-audit/stream/2021-03-13T06:52:29.259"

  default_tags = {
    Environment = "AFT"
    Project     = "Offboarding Automation"
  }

  # IMPORTANT: pass provider aliases to the module
  providers = {
    aws    = aws.aft
    aws.ct = aws.ct
  }
}

Module inputs (from module/variables.tf)

• cloudwatch_log_group_retention (number, default 90) – CloudWatch log retention in days
• default_tags (map(string)) – default resource tags
• region (string, default eu-west-1)
• ct_account_id (string) – CT management account ID
• aft_account_id (string) – AFT management account ID
• ct_destination_ou (string) – Destination OU for suspended accounts
• ct_root_ou_id (string) – Root OU ID
• aft-request-audit-table-stream-arn (string) – DynamoDB stream ARN for AFT audit table
• aft-request-audit-table-encrption-key-id (string) – KMS key ARN used for the stream encryption

Note: variable names include hyphens by design and are referenced as var.aft-request-audit-table-stream-arn in Terraform.

What the module creates

In the AFT account:

• Lambda function aft-close-account-lambda (runtime Python 3.11)
• Lambda execution role aft-close-account-lambda-role with inline policy
• Event source mapping from the AFT audit DynamoDB stream to the Lambda
• CloudWatch log group /aws/lambda/aft-close-account-lambda

In the CT account:

• IAM role aft-account-closure-role that the Lambda assumes
• Inline policy allowing limited Service Catalog and Organizations operations

Lambda environment variables

• REGION – AWS region
• CT_ACCOUNT – CT management account ID
• DESTINATION_OU – OU to move accounts into prior to closure
• ROOT_OU_ID – Root OU ID
• LOG_LEVEL – optional, defaults to INFO

CI/CD (GitLab)

Pipeline stages (from .gitlab-ci.yml):

stages:
  - terraform-plan
  - terraform-apply
  - terraform-destroy

Jobs:

• terraform-plan-close-and-suspend

  • Runner tag: aws-org-runner
  • Runs terraform init and terraform plan in close-and-suspend/configuration
  • Artifacts: close-and-suspend/configuration/tfplan, close-and-suspend/module/lambda/aft-close-account.zip
  • Rules: trigger on main branch when paths under close-and-suspend/** change

• terraform-apply-close-and-suspend

  • Depends on plan job artifacts
  • Runs terraform apply -auto-approve tfplan

• terraform-destroy-close-and-suspend (manual)

  • Destroys the stack after terraform init

How to deploy

Option A: via GitLab CI/CD (recommended)

1. Ensure your runner has the tag aws-org-runner and Terraform/AWS CLI installed.
2. Provide AWS credentials to the runner (environment variables or IAM role).
3. Commit changes to main under close-and-suspend/ to trigger plan/apply.

Option B: manual (local)

cd close-and-suspend/configuration
terraform init
terraform plan -out=tfplan
terraform apply tfplan

Verification

• Lambda function exists: aft-close-account-lambda in the AFT account
• Log group exists: /aws/lambda/aft-close-account-lambda
• Event source mapping between the AFT audit DynamoDB stream and the Lambda
• CT account has role aft-account-closure-role with the inline policy

Security model

• The Lambda runs in the AFT account and assumes the CT role aft-account-closure-role.
• Minimal permissions for:
  • Service Catalog: terminate/describe provisioned products and records
  • Organizations: list/move/close accounts, describe/list accounts
  • DynamoDB Streams: read stream shards and records
  • SSM: read the AFT metadata table name parameter
  • KMS: decrypt for the audit stream key

Troubleshooting

• Service Catalog termination keeps failing:
  • The Lambda attempts a second termination with IgnoreErrors=True and then proceeds; check record status in logs.
• No account found by email:
  • Ensure the AFT metadata table and emailIndex are populated; verify the email matches the source event.
• AssumeRole failures:
  • Check trust policy in CT role and AFT Lambda role ARN in iam-ct.tf.
• Event not triggering the Lambda:
  • Ensure the correct stream ARN is configured and the stream is enabled on the table.

Known caveats

• State locking: the S3 backend block uses use_lockfile, but reliable state locking in Terraform on S3 requires a DynamoDB table (set dynamodb_table = "<table>").
• Example values: account IDs, OU IDs, ARNs in examples are placeholders—replace with your real values.
• Variable names: some inputs use hyphens—this is valid in HCL2 and used by this module.

License

MIT License. See LICENSE for details.

References

• AWS Control Tower: https://docs.aws.amazon.com/controltower/
• AWS Organizations: https://docs.aws.amazon.com/organizations/
• AFT: https://docs.aws.amazon.com/controltower/latest/userguide/aft-overview.html
• AWS Lambda: https://docs.aws.amazon.com/lambda/
• Terraform AWS Provider: https://registry.terraform.io/providers/hashicorp/aws/latest/docs

AWS Terraform Account Closure Automation

An automated solution for closing and suspending AWS accounts in AWS Control Tower environments using Terraform and Lambda functions. This project provides a secure, event-driven approach to account lifecycle management with proper governance controls.

Table of Contents

• Overview
• Architecture
• Features
• Prerequisites
• Quick Start
• Configuration
• Deployment
• How It Works
• Security
• Monitoring
• Troubleshooting
• Best Practices
• Contributing

Overview

This solution automates the process of closing AWS accounts within an AWS Control Tower environment. When an account removal request is detected in the AFT (Account Factory for Terraform) audit trail, the system automatically:

1. Terminates the Service Catalog provisioned product
2. Moves the account to a SUSPENDED organizational unit
3. Closes the AWS account permanently

The automation ensures proper governance, audit trails, and secure cross-account operations.

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   AFT Account   │    │  Control Tower   │    │   Target OU     │
│                 │    │    Account       │    │  (SUSPENDED)    │
├─────────────────┤    ├──────────────────┤    ├─────────────────┤
│ DynamoDB Stream │───▶│ Lambda Function  │───▶│ Closed Account  │
│ (Audit Trail)   │    │ (Account Closer) │    │                 │
│                 │    │                  │    │                 │
│ IAM Role        │    │ Service Catalog  │    │                 │
│ (Lambda Exec)   │    │ Organizations    │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Components

• AFT Account: Contains the Lambda function and DynamoDB stream trigger
• Control Tower Account: Provides cross-account role for account operations
• Lambda Function: Processes account closure requests
• DynamoDB Stream: Triggers automation on audit trail changes
• IAM Roles: Secure cross-account access with least privilege

Features

• Event-Driven: Automatically triggered by AFT audit trail changes
• Secure: Cross-account role assumption with minimal permissions
• Auditable: Comprehensive logging and CloudWatch integration
• Resilient: Error handling and retry mechanisms
• Configurable: Customizable organizational units and timeouts
• Code Signing: Lambda functions are signed for security
• Monitoring: CloudWatch logs with configurable retention

Prerequisites

Before deploying this solution, ensure you have:

AWS Environment

• AWS Control Tower deployed and configured
• Account Factory for Terraform (AFT) set up
• Appropriate AWS Organizations structure with SUSPENDED OU
• Cross-account trust relationships configured

Permissions

• Administrative access to AFT management account
• Administrative access to Control Tower management account
• Permissions to create IAM roles and policies
• Permissions to deploy Lambda functions

Tools

• Terraform >= 1.0
• AWS CLI configured
• Python 3.11 (for Lambda runtime)

Quick Start

Prerequisites Setup

1. GitLab Runner Configuration:

   # Ensure runner has required tags
   tags: ["test-runner"]
   
   # Install required tools on runner
   terraform --version  # >= 0.15.0
   aws --version        # Latest AWS CLI

2. AWS Credentials:

   # Configure runner with appropriate AWS credentials
   # Must have access to assume roles in both AFT and CT accounts
   export AWS_ACCESS_KEY_ID="your-access-key"
   export AWS_SECRET_ACCESS_KEY="your-secret-key"
   export AWS_DEFAULT_REGION="eu-west-1"

Deployment via GitLab CI/CD

1. Clone and configure:

   git clone <repository-url>
   cd aws-tf-close-account

2. Update configuration:

   # Edit close-and-suspend/configuration/main.tf
   # Update account IDs, OUs, and other environment-specific values

3. Deploy via pipeline:

   git add .
   git commit -m "Configure account closure automation"
   git push origin main

4. Monitor deployment:

   • Navigate to GitLab → CI/CD → Pipelines
   • Review terraform-plan-close-and-suspend job output
   • Verify terraform-apply-close-and-suspend completes successfully

5. Verify deployment:

   • Check AWS Lambda console for aft-close-account-lambda
   • Verify IAM roles in both AFT and CT accounts
   • Review CloudWatch logs for any initialization issues

Manual Deployment (Alternative)

If you prefer manual deployment or need to troubleshoot:

1. Local setup:

   cd close-and-suspend/configuration
   terraform init

2. Plan and apply:

   terraform plan -out=tfplan
   terraform apply tfplan

3. Cleanup (if needed):

   terraform destroy

Configuration

Required Variables

Edit close-and-suspend/configuration/main.tf with your environment-specific values:

module "offboarding_lambda" {
  source = "../module"
  
  # CloudWatch Configuration
  cloudwatch_log_group_retention = "90"  # Days to retain logs
  
  # AWS Configuration
  region         = "eu-west-1"           # Primary region
  aft_account_id = "123456789012"        # AFT management account ID
  ct_account_id  = "210987654321"        # Control Tower management account ID
  
  # Organizational Units
  ct_destination_ou = "ou-juup-d1e061ao" # SUSPENDED OU ID
  ct_root_ou_id     = "r-juup"           # Root OU ID
  
  # DynamoDB Configuration
  aft-request-audit-table-encrption-key-id = "arn:aws:kms:eu-west-1:123456789012:key/..."
  aft-request-audit-table-stream-arn       = "arn:aws:dynamodb:eu-west-1:123456789012:table/aft-request-audit/stream/..."
  
  # Tagging
  default_tags = {
    Environment = "AFT"
    Project     = "Offboarding Automation"
  }
}

Environment Variables

The Lambda function uses these environment variables:

• REGION: AWS region for operations
• CT_ACCOUNT: Control Tower management account ID
• DESTINATION_OU: Target OU for suspended accounts
• ROOT_OU_ID: Root organizational unit ID

GitLab CI/CD Pipeline

Pipeline Overview

The GitLab CI/CD pipeline provides automated, secure deployment of the account closure infrastructure with proper state management and cross-account role assumptions.

Pipeline Stages

1. Terraform Plan (terraform-plan-close-and-suspend)

Purpose: Creates and validates Terraform execution plan

Triggers:

• Commits to main branch
• Changes in close-and-suspend/configuration/**/*
• Changes in close-and-suspend/module/**/*

Process:

cd close-and-suspend/configuration
terraform init
terraform plan -out=tfplan

Artifacts:

• tfplan: Terraform execution plan
• aft-close-account.zip: Lambda deployment package
• Expiration: 3 hours

2. Terraform Apply (terraform-apply-close-and-suspend)

Purpose: Applies the validated Terraform plan

Dependencies: Requires successful plan stage

Process:

cd close-and-suspend/configuration
terraform init
terraform apply -auto-approve tfplan

Safety Features:

• Uses pre-validated plan from artifacts
• No interactive approval required
• Automatic rollback on failure

3. Terraform Destroy (terraform-destroy-close-and-suspend)

Purpose: Removes all infrastructure (manual trigger only)

Trigger: Manual execution only

Process:

cd close-and-suspend/configuration
terraform init
terraform destroy -auto-approve

Safety: Manual trigger prevents accidental destruction

Runner Configuration

Required Tags

tags:
  - test-runner

Required Software

• Terraform >= 0.15.0
• AWS CLI (latest version)
• Git
• Bash/Shell access

AWS Permissions

The runner must have permissions to:

• Assume AWSAFTExecution role in AFT account
• Assume AWSAFTExecution role in CT account
• Access S3 backend bucket
• DynamoDB state locking

State Management

S3 Backend Configuration

backend "s3" {
  bucket               = "aft-management-gitlab-runner-tfstate"
  key                  = "offboarding-module.tfstate"
  region               = "eu-west-1"
  use_lockfile         = true    # S3 native locking
  encrypt              = true    # State encryption
  workspace_key_prefix = "offboarding-module"
}

State Security Features

• Encryption: All state files encrypted at rest
• Locking: Prevents concurrent modifications
• Versioning: S3 versioning for state history
• Access Control: IAM-based access restrictions

Pipeline Variables

GitLab CI Variables (Optional)

Set these in GitLab → Settings → CI/CD → Variables:

# AWS Credentials (if not using IAM roles)
AWS_ACCESS_KEY_ID: "your-access-key"
AWS_SECRET_ACCESS_KEY: "your-secret-key"
AWS_DEFAULT_REGION: "eu-west-1"

# Terraform Variables (if overriding defaults)
TF_VAR_region: "eu-west-1"
TF_VAR_aft_account_id: "123456789012"
TF_VAR_ct_account_id: "210987654321"

Pipeline Monitoring

Success Indicators

• ✅ Plan stage completes without errors
• ✅ Apply stage creates all resources
• ✅ Lambda function is deployed and active
• ✅ IAM roles created in both accounts

Failure Scenarios

• ❌ Terraform validation errors
• ❌ AWS permission issues
• ❌ Cross-account role assumption failures
• ❌ Resource creation conflicts

Troubleshooting Pipeline Issues

Plan Stage Failures:

# Check Terraform syntax
terraform validate

# Verify AWS credentials
aws sts get-caller-identity

# Test role assumptions
aws sts assume-role --role-arn arn:aws:iam::123456789012:role/AWSAFTExecution --role-session-name test

Apply Stage Failures:

# Check resource conflicts
terraform state list

# Verify permissions
aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/AWSAFTExecution --action-names lambda:CreateFunction

# Review CloudWatch logs
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/aft-close-account

Pipeline Best Practices

Development Workflow

1. Create feature branch for changes
2. Test changes in development environment
3. Create merge request to main
4. Review pipeline output before merging
5. Monitor production deployment

Security Practices

• Use IAM roles instead of access keys when possible
• Regularly rotate access credentials
• Monitor pipeline execution logs
• Implement proper approval workflows for sensitive changes
• Use branch protection rules

Operational Practices

• Monitor pipeline execution times
• Set up notifications for pipeline failures
• Regularly review and update runner configurations
• Maintain backup of Terraform state
• Document any manual interventions

Deployment

GitLab CI/CD Pipeline

This project uses GitLab CI/CD for automated deployment with a three-stage pipeline:

stages:
  - terraform-plan
  - terraform-apply
  - terraform-destroy

Pipeline Configuration

The pipeline is configured in .gitlab-ci.yml with the following jobs:

Planning Stage:

• terraform-plan-close-and-suspend: Creates Terraform execution plan
• Triggers on changes to close-and-suspend/ directory
• Stores plan artifacts for apply stage

Apply Stage:

• terraform-apply-close-and-suspend: Applies the Terraform plan
• Requires successful planning stage
• Automatically applies on main branch

Destroy Stage:

• terraform-destroy-close-and-suspend: Destroys infrastructure
• Manual trigger only for safety

Pipeline Execution Flow

1. Trigger Conditions:

   rules:
     - if: $CI_COMMIT_BRANCH == "main"
       changes:
         - close-and-suspend/configuration/**/*
         - close-and-suspend/module/**/*

2. Artifact Management:

   • Terraform plans stored as artifacts
   • Lambda deployment packages included
   • 3-hour expiration for security

3. Runner Requirements:

   • Tagged with test-runner
   • Must have AWS credentials configured
   • Terraform and AWS CLI installed

Backend Configuration

The solution uses S3 backend for state management:

backend "s3" {
  bucket               = "aft-management-gitlab-runner-tfstate"
  key                  = "offboarding-module.tfstate"
  region               = "eu-west-1"
  use_lockfile         = true
  encrypt              = true
  workspace_key_prefix = "offboarding-module"
}

Multi-Provider Setup

The configuration uses multiple AWS providers for cross-account deployment:

# Default provider
provider "aws" {
  region = var.region
}

# AFT Management Account
provider "aws" {
  alias  = "aft"
  region = var.region
  assume_role {
    role_arn    = "arn:aws:iam::123456789012:role/AWSAFTExecution"
    external_id = "ASSUME_ROLE_ON_TARGET_ACC"
  }
}

# Control Tower Management Account
provider "aws" {
  alias  = "ct"
  region = var.region
  assume_role {
    role_arn    = "arn:aws:iam::210987654321:role/AWSAFTExecution"
    external_id = "ASSUME_ROLE_ON_TARGET_ACC"
  }
}

Deployment Methods

Option 1: GitLab CI/CD (Recommended)

1. Push changes to repository:

   git add .
   git commit -m "Update account closure configuration"
   git push origin main

2. Monitor pipeline:

   • Navigate to GitLab CI/CD → Pipelines
   • Review plan output in planning stage
   • Verify successful apply stage

3. Manual destroy (if needed):

   • Navigate to GitLab CI/CD → Pipelines
   • Click "Run pipeline" → Select "terraform-destroy-close-and-suspend"

Option 2: Manual Deployment

1. Configure AWS credentials:

   # Set up profiles for both accounts
   aws configure --profile aft-account
   aws configure --profile ct-account

2. Initialize and deploy:

   cd close-and-suspend/configuration
   terraform init
   terraform plan -out=tfplan
   terraform apply tfplan

3. Verify deployment:

   # Check Lambda function
   aws lambda get-function --function-name aft-close-account-lambda --profile aft-account
   
   # Check CT role
   aws iam get-role --role-name aft-account-closure-role --profile ct-account

Multi-Account Resource Distribution

AFT Management Account (123456789012):

• Lambda function (aft-close-account-lambda)
• Lambda execution role (aft-close-account-lambda-role)
• CloudWatch log group (/aws/lambda/aft-close-account-lambda)
• DynamoDB stream event source mapping
• Code signing configuration and profile

Control Tower Management Account (210987654321):

• Cross-account IAM role (aft-account-closure-role)
• Service Catalog and Organizations permissions
• Account closure execution permissions

Pipeline Security

• State Encryption: Terraform state encrypted in S3
• State Locking: DynamoDB locking prevents concurrent runs
• Role Assumption: Cross-account access via IAM roles
• Artifact Security: Limited artifact expiration (3 hours)
• Manual Destroy: Destroy operations require manual approval

How It Works

Workflow Overview

1. Trigger: AFT audit trail DynamoDB stream detects account removal request
2. Processing: Lambda function processes the stream event
3. Validation: Verifies account details and permissions
4. Service Catalog: Terminates the provisioned product
5. Organizations: Moves account to SUSPENDED OU
6. Closure: Initiates AWS account closure
7. Logging: Records all operations in CloudWatch

Event Processing

The Lambda function processes DynamoDB stream events:

# Event structure
{
    "Records": [
        {
            "eventName": "INSERT",
            "dynamodb": {
                "NewImage": {
                    "control_tower_parameters": {
                        "M": {
                            "AccountName": {"S": "test-account"},
                            "AccountEmail": {"S": "test@example.com"},
                            "ManagedOrganizationalUnit": {"S": "ou-source"}
                        }
                    },
                    "ddb_event_name": {"S": "REMOVE"}
                }
            }
        }
    ]
}

Account Closure Process

1. Account Lookup: Query AFT metadata table by email
2. Role Assumption: Assume cross-account role in CT account
3. Product Termination: Terminate Service Catalog product
4. Account Movement: Move from current OU to SUSPENDED OU
5. Account Closure: Call Organizations CloseAccount API
6. Verification: Confirm successful closure

Security

IAM Permissions

Lambda Execution Role (AFT Account):

• CloudWatch Logs access
• DynamoDB stream and table access
• SSM parameter access
• KMS decrypt permissions
• Cross-account role assumption

Account Closure Role (CT Account):

• Service Catalog terminate permissions
• Organizations account management
• Limited to specific operations only

Security Features

• Code Signing: Lambda functions are signed for integrity
• Least Privilege: Minimal required permissions only
• Cross-Account: Secure role assumption between accounts
• Encryption: KMS encryption for DynamoDB streams
• Audit Trail: Comprehensive logging of all operations

Network Security

• Lambda functions run in AWS managed VPC
• No internet access required for core functionality
• All AWS API calls use service endpoints

Monitoring

CloudWatch Logs

The Lambda function creates detailed logs:

/aws/lambda/aft-close-account-lambda

Log retention is configurable (default: 90 days).

Key Metrics to Monitor

• Lambda function invocations
• Lambda function errors
• Lambda function duration
• DynamoDB stream processing lag
• Account closure success/failure rates

Alerting

Consider setting up CloudWatch alarms for:

# Lambda errors
aws cloudwatch put-metric-alarm \
  --alarm-name "AFT-Account-Closure-Errors" \
  --alarm-description "Lambda function errors" \
  --metric-name Errors \
  --namespace AWS/Lambda \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold

Troubleshooting

Common Issues

Lambda Function Timeout

Symptom: Function times out during execution Solution: Increase timeout value (current: 900 seconds)

Permission Denied

Symptom: Cross-account role assumption fails Solution: Verify trust relationship and IAM policies

Account Not Found

Symptom: Cannot find account in AFT metadata Solution: Verify account email and AFT table structure

Service Catalog Product Not Found

Symptom: Cannot terminate provisioned product Solution: Verify product name matches account name exactly

Debugging Steps

Pipeline Debugging

1. Check pipeline logs:

   # In GitLab UI: CI/CD → Pipelines → Select pipeline → View job logs
   # Look for specific error messages in plan/apply stages

2. Verify runner configuration:

   # On GitLab runner
   gitlab-runner verify
   terraform --version
   aws --version
   aws sts get-caller-identity

3. Test Terraform locally:

   cd close-and-suspend/configuration
   terraform init
   terraform validate
   terraform plan

Application Debugging

4. Check Lambda logs:

   aws logs filter-log-events \
     --log-group-name /aws/lambda/aft-close-account-lambda \
     --start-time $(date -d '1 hour ago' +%s)000 \
     --profile aft-account

5. Verify DynamoDB stream:

   aws dynamodb describe-table \
     --table-name aft-request-audit \
     --query 'Table.StreamSpecification' \
     --profile aft-account

6. Test cross-account role assumption:

   aws sts assume-role \
     --role-arn arn:aws:iam::210987654321:role/aft-account-closure-role \
     --role-session-name test-session \
     --profile aft-account

State Management Debugging

7. Check Terraform state:

   # List state resources
   terraform state list
   
   # Check specific resource
   terraform state show aws_lambda_function.aft-close-account-lambda
   
   # Verify backend connectivity
   terraform init -backend-config="bucket=aft-management-gitlab-runner-tfstate"

Error Codes

Pipeline Errors

• terraform init failed: Backend configuration or credentials issue
• terraform plan failed: Configuration validation or permission errors
• terraform apply failed: Resource creation or dependency issues
• Job failed: exit code 1: General Terraform execution failure
• Runner system failure: GitLab runner connectivity or resource issues

Application Errors

• ResourceNotFoundException: Account or Service Catalog product not found
• AccessDeniedException: Insufficient permissions for AWS operations
• ValidationException: Invalid parameters passed to AWS APIs
• ThrottlingException: API rate limits exceeded
• AssumeRoleFailure: Cross-account role assumption failed
• LambdaTimeoutException: Function execution exceeded 900 seconds

Best Practices

Operational

• Test in non-production environment first
• Use GitLab CI/CD for consistent deployments
• Monitor both pipeline and CloudWatch logs regularly
• Set up appropriate alerting for pipeline failures
• Document account closure procedures and pipeline workflows
• Maintain audit trails for all deployments
• Regularly review and update GitLab runner configurations
• Implement proper backup strategies for Terraform state

Security

• Regularly review IAM permissions
• Use least privilege principles
• Enable CloudTrail logging
• Implement proper backup procedures
• Regular security assessments

Development

• Use infrastructure as code
• Version control all configurations
• Implement proper testing
• Document all changes
• Follow AWS Well-Architected principles

Account Management

• Maintain accurate OU structure
• Document account lifecycle processes
• Implement proper approval workflows
• Regular compliance reviews
• Backup critical account data before closure

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Update documentation
6. Submit a pull request

Development Guidelines

• Follow Terraform best practices
• Use consistent naming conventions
• Add appropriate comments
• Update README for any changes
• Test in development environment

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For questions or issues:

• Create an issue in this repository
• Review AWS Control Tower documentation
• Consult AWS Organizations documentation
• Check AWS Lambda best practices

References

• AWS Control Tower Documentation
• AWS Organizations Documentation
• Account Factory for Terraform
• AWS Lambda Documentation
• Terraform AWS Provider

Changelog

Version 1.0.0

• Initial release
• Basic account closure automation
• Cross-account role support
• CloudWatch logging integration
• Code signing implementation