Companion to ../MICROSERVICES_TEACHING_GUIDE.md §15.

Goal: take the eight microservices in this repository from docker compose up to a production-grade deployment on AWS using ECS Fargate, RDS, MSK, ElastiCache, ALB, and Terraform.

Table of Contents

  1. Prerequisites
  2. Reference architecture on AWS
  3. Environments and naming
  4. Phase 0 — Account bootstrap
  5. Phase 1 — Network (VPC, subnets, NAT, endpoints)
  6. Phase 2 — Platform (RDS, ElastiCache, MSK, ECR, Secrets)
  7. Phase 3 — Compute (ECS cluster, services, ALB)
  8. Phase 4 — Observability
  9. Phase 5 — CI/CD
  10. Phase 6 — Hardening
  11. Day-2 operations
  12. Cost optimisation
  13. Disaster recovery
  14. Migration path: ECS → EKS

1. Prerequisites

Tools on your laptop:

# macOS via Homebrew
brew install awscli terraform jq docker
brew install --cask docker

AWS account access:

  • A dedicated AWS account per environment is strongly recommended (paymentgw-dev, paymentgw-prod). Use AWS Organizations + SSO.
  • For first-time setup, an admin IAM user with MFA. After Phase 0 you will switch to OIDC federation for CI.
  • Region choice: pick one near your customers. Examples: eu-west-2 (London), us-east-1, ap-southeast-1. We use eu-west-2 in samples.

Pre-flight check:

aws sts get-caller-identity
# You should see your account, user/role, and ARN.

2. Reference architecture on AWS

                        ┌───────────────────────────────────────┐
                        │ Route 53 (paygw.example.com)          │
                        └──────────────┬────────────────────────┘
                                       │
                              ┌────────▼────────┐
                              │  ACM (TLS cert) │
                              └────────┬────────┘
                                       │
   public ┌────────────────────────────▼────────────────────────┐
   subnet │ Application Load Balancer (api-gateway target only) │
          └─────┬────────────────────────────────────────┬──────┘
                │                                        │  WAF (in front)
                │ http(s)                                │
                ▼                                        ▼
        ┌───────────────────────────────────────────────────┐
        │ private subnet — VPC                              │
        │                                                   │
        │  ECS Fargate cluster: paymentgw-prod              │
        │  ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │
        │  │ api-gateway │ │ payment-svc  │ │ query-svc   │ │
        │  └─────────────┘ └──────────────┘ └─────────────┘ │
        │  ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │
        │  │ bank-adptr  │ │ fraud-check  │ │ notif-svc   │ │
        │  └─────────────┘ └──────────────┘ └─────────────┘ │
        │   (no Eureka/Config — replaced by:                │
        │    Cloud Map + AppConfig/Secrets Manager)         │
        │                                                   │
        │  ┌──────────┐  ┌──────────────┐  ┌──────────────┐ │
        │  │ RDS PG   │  │ ElastiCache  │  │  MSK Kafka   │ │
        │  │ Multi-AZ │  │  Redis       │  │  3 brokers   │ │
        │  └──────────┘  └──────────────┘  └──────────────┘ │
        │                                                   │
        │  VPC endpoints: S3, ECR (api+dkr), Logs, SecretsMgr│
        └───────────────────────────────────────────────────┘
                                       │
                            NAT Gateways (per AZ)
                                       │
                                  Internet

Notes:

  • Only the ALB lives in public subnets. Every service runs in private subnets.
  • Three AZs for production; two for dev.
  • VPC endpoints keep ECR/S3/Logs/Secrets traffic on the AWS backbone — both cheaper and more secure.
  • api-gateway is the only ECS service registered to the ALB. Internal services talk via ECS Service Connect (Cloud Map).

3. Environments and naming

Environment Purpose Account Region
dev Engineers' integration env paymentgw-dev eu-west-2
staging Pre-prod, prod-like data paymentgw-staging eu-west-2
prod Live customer traffic paymentgw-prod eu-west-2

Name pattern: {component}-{env} everywhere. Tags: Project=paymentgw, Environment=prod, Owner=payments-team, CostCenter=12345.

4. Phase 0 — Account bootstrap

One-off, per AWS account.

4.1 Create the Terraform state backend

aws s3api create-bucket \
  --bucket tfstate-paymentgw-prod-eu-west-2 \
  --region eu-west-2 \
  --create-bucket-configuration LocationConstraint=eu-west-2

aws s3api put-bucket-versioning \
  --bucket tfstate-paymentgw-prod-eu-west-2 \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption \
  --bucket tfstate-paymentgw-prod-eu-west-2 \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256" }
    }]
  }'

aws dynamodb create-table \
  --table-name tfstate-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region eu-west-2

4.2 GitHub OIDC federation (no long-lived AWS keys)

Provision an IAM identity provider and a deployer role assumable by GitHub:

# infra/terraform/modules/github-oidc/main.tf (excerpt)
resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "gh_deployer" {
  name = "gh-deployer"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = { Federated = aws_iam_openid_connect_provider.github.arn },
      Action = "sts:AssumeRoleWithWebIdentity",
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        },
        StringLike = {
          "token.actions.githubusercontent.com:sub" = "repo:my-org/payment-gateway-microservices:*"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "deployer_admin" {
  role       = aws_iam_role.gh_deployer.name
  policy_arn = "arn:aws:iam::aws:policy/PowerUserAccess"   # tighten in prod
}

In your GitHub Actions workflow:

permissions:
  id-token: write
  contents: read

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/gh-deployer
    aws-region: eu-west-2

4.3 Org-wide guardrails

  • Enable CloudTrail (org trail to a central S3 bucket).
  • Enable GuardDuty.
  • Enable AWS Config with conformance packs (CIS, PCI-DSS).
  • Configure Cost Anomaly Detection alerts to a Slack/email channel.

5. Phase 1 — Network

5.1 VPC layout

Subnet AZ count CIDR (example) Notes
Public 3 10.0.0.0/24, … ALB only; route to IGW
Private (apps) 3 10.0.10.0/24, … ECS tasks; route to NAT
Private (data) 3 10.0.20.0/24, … RDS, ElastiCache, MSK; no NAT route, no internet

Why two private tiers? It limits the blast radius if an app subnet is breached: data-tier security groups only accept traffic from app SGs, not from the internet or from public subnets.

5.2 Terraform skeleton

infra/terraform/modules/network/main.tf:

resource "aws_vpc" "this" {
  cidr_block           = var.cidr
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags                 = merge(var.tags, { Name = "${var.name}-vpc" })
}

resource "aws_subnet" "public" {
  for_each                = toset(var.azs)
  vpc_id                  = aws_vpc.this.id
  cidr_block              = cidrsubnet(var.cidr, 8, index(var.azs, each.value))
  availability_zone       = each.value
  map_public_ip_on_launch = true
  tags = merge(var.tags, { Name = "${var.name}-public-${each.value}", Tier = "public" })
}

resource "aws_subnet" "private_app" {
  for_each          = toset(var.azs)
  vpc_id            = aws_vpc.this.id
  cidr_block        = cidrsubnet(var.cidr, 8, index(var.azs, each.value) + 10)
  availability_zone = each.value
  tags = merge(var.tags, { Name = "${var.name}-app-${each.value}", Tier = "private-app" })
}

resource "aws_subnet" "private_data" {
  for_each          = toset(var.azs)
  vpc_id            = aws_vpc.this.id
  cidr_block        = cidrsubnet(var.cidr, 8, index(var.azs, each.value) + 20)
  availability_zone = each.value
  tags = merge(var.tags, { Name = "${var.name}-data-${each.value}", Tier = "private-data" })
}

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
  tags   = merge(var.tags, { Name = "${var.name}-igw" })
}

resource "aws_eip" "nat" {
  for_each = toset(var.azs)
  domain   = "vpc"
}

resource "aws_nat_gateway" "this" {
  for_each      = toset(var.azs)
  allocation_id = aws_eip.nat[each.value].id
  subnet_id     = aws_subnet.public[each.value].id
}

# route tables omitted for brevity — see infra/terraform/modules/network/

5.3 VPC endpoints

Create gateway endpoints for S3 and DynamoDB (free) and interface endpoints for ECR (api + dkr), CloudWatch Logs, Secrets Manager, STS, and MSK (interface, not gateway).

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private_app[*].id
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.this.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = values(aws_subnet.private_app)[*].id
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}
# repeat for ecr.api, logs, secretsmanager, sts

6. Phase 2 — Platform

6.1 ECR (one repository per service)

locals {
  services = [
    "api-gateway", "payment-service", "payment-query-service",
    "bank-adapter-service", "fraud-check-service", "notification-service"
  ]
}

resource "aws_ecr_repository" "svc" {
  for_each             = toset(local.services)
  name                 = each.value
  image_tag_mutability = "IMMUTABLE"
  image_scanning_configuration { scan_on_push = true }
  encryption_configuration { encryption_type = "AES256" }
  tags = var.tags
}

resource "aws_ecr_lifecycle_policy" "svc" {
  for_each   = aws_ecr_repository.svc
  repository = each.value.name
  policy = jsonencode({
    rules = [{
      rulePriority = 1, description = "Keep last 30 images",
      selection = { tagStatus = "any", countType = "imageCountMoreThan", countNumber = 30 },
      action = { type = "expire" }
    }]
  })
}

6.2 RDS for PostgreSQL

One db cluster per environment, multiple databases inside:

resource "aws_rds_cluster" "pg" {
  cluster_identifier      = "paymentgw-${var.env}"
  engine                  = "aurora-postgresql"
  engine_version          = "15.5"
  master_username         = "app_admin"
  master_password         = random_password.master.result
  backup_retention_period = 14
  database_name           = "platform"
  storage_encrypted       = true
  kms_key_id              = aws_kms_key.rds.arn
  vpc_security_group_ids  = [aws_security_group.rds.id]
  db_subnet_group_name    = aws_db_subnet_group.pg.name
  deletion_protection     = var.env == "prod"
  skip_final_snapshot     = var.env != "prod"
  apply_immediately       = false
}

resource "aws_rds_cluster_instance" "pg" {
  count              = var.env == "prod" ? 2 : 1
  identifier         = "paymentgw-${var.env}-${count.index}"
  cluster_identifier = aws_rds_cluster.pg.id
  instance_class     = var.env == "prod" ? "db.r6g.large" : "db.t4g.medium"
  engine             = aws_rds_cluster.pg.engine
  engine_version     = aws_rds_cluster.pg.engine_version
}

Per-service databases + roles are created with a small bootstrapping job (a Lambda or a one-shot ECS task). Each service gets its own user + password stored in Secrets Manager.

6.3 ElastiCache for Redis

For idempotency and rate limiting:

resource "aws_elasticache_replication_group" "redis" {
  replication_group_id          = "paymentgw-${var.env}"
  description                   = "Idempotency + rate limiting"
  engine                        = "redis"
  engine_version                = "7.1"
  node_type                     = var.env == "prod" ? "cache.m6g.large" : "cache.t4g.small"
  num_cache_clusters            = var.env == "prod" ? 2 : 1
  automatic_failover_enabled    = var.env == "prod"
  at_rest_encryption_enabled    = true
  transit_encryption_enabled    = true
  auth_token                    = random_password.redis.result
  subnet_group_name             = aws_elasticache_subnet_group.redis.name
  security_group_ids            = [aws_security_group.redis.id]
}

6.4 MSK (managed Kafka)

resource "aws_msk_cluster" "kafka" {
  cluster_name           = "paymentgw-${var.env}"
  kafka_version          = "3.6.0"
  number_of_broker_nodes = var.env == "prod" ? 3 : 2
  broker_node_group_info {
    instance_type   = var.env == "prod" ? "kafka.m5.large" : "kafka.t3.small"
    client_subnets  = values(module.network.private_data_subnet_ids)
    security_groups = [aws_security_group.msk.id]
    storage_info { ebs_storage_info { volume_size = 100 } }
  }
  encryption_info {
    encryption_at_rest_kms_key_arn = aws_kms_key.msk.arn
    encryption_in_transit { client_broker = "TLS", in_cluster = true }
  }
  client_authentication { sasl { iam = true } }
}

Cheaper option for dev: aws_msk_serverless_cluster — pay per partition.

6.5 Secrets Manager

For each service that needs secrets (DB password, API keys), create a secret:

resource "aws_secretsmanager_secret" "payment_db" {
  name        = "/paymentgw/${var.env}/payment-service/db"
  description = "Postgres credentials for payment-service"
  kms_key_id  = aws_kms_key.secrets.arn
}

resource "aws_secretsmanager_secret_version" "payment_db" {
  secret_id     = aws_secretsmanager_secret.payment_db.id
  secret_string = jsonencode({
    username = "payment_svc",
    password = random_password.payment_db.result,
    host     = aws_rds_cluster.pg.endpoint,
    port     = 5432,
    db       = "payment_db"
  })
}

ECS injects these as environment variables at task start using the secrets block in the task definition (no plaintext on disk).

7. Phase 3 — Compute

7.1 ECS cluster

resource "aws_ecs_cluster" "main" {
  name = "paymentgw-${var.env}"
  setting { name = "containerInsights"; value = "enabled" }
  configuration {
    execute_command_configuration { logging = "DEFAULT" }
  }
  service_connect_defaults { namespace = aws_service_discovery_http_namespace.svc.arn }
}

resource "aws_service_discovery_http_namespace" "svc" {
  name = "paymentgw-${var.env}"
}

7.2 The reusable ecs-service module

# infra/terraform/modules/ecs-service/main.tf
resource "aws_ecs_task_definition" "this" {
  family                   = var.name
  cpu                      = var.cpu
  memory                   = var.memory
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  execution_role_arn       = aws_iam_role.exec.arn
  task_role_arn            = aws_iam_role.task.arn

  container_definitions = jsonencode([{
    name      = var.name
    image     = var.image
    essential = true
    portMappings = [{ containerPort = var.port, name = var.name, appProtocol = "http" }]
    environment = [for k, v in var.env : { name = k, value = v }]
    secrets     = [for k, v in var.secrets : { name = k, valueFrom = v }]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        awslogs-group         = aws_cloudwatch_log_group.svc.name
        awslogs-region        = var.region
        awslogs-stream-prefix = var.name
      }
    }
    healthCheck = {
      command     = ["CMD-SHELL", "wget -q -O - http://localhost:${var.port}/actuator/health/liveness || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
      startPeriod = 60
    }
  }])
}

resource "aws_ecs_service" "this" {
  name             = var.name
  cluster          = var.cluster_arn
  task_definition  = aws_ecs_task_definition.this.arn
  desired_count    = var.desired_count
  launch_type      = "FARGATE"
  platform_version = "LATEST"

  network_configuration {
    subnets         = var.subnet_ids
    security_groups = [aws_security_group.svc.id]
    assign_public_ip = false
  }

  service_connect_configuration {
    enabled   = true
    namespace = var.service_discovery
    service {
      port_name      = var.name
      discovery_name = var.name
      client_alias { port = var.port; dns_name = var.name }
    }
  }

  dynamic "load_balancer" {
    for_each = var.attach_to_alb ? [1] : []
    content {
      target_group_arn = aws_lb_target_group.svc[0].arn
      container_name   = var.name
      container_port   = var.port
    }
  }

  deployment_controller { type = "ECS" }   # for blue/green use CODE_DEPLOY
  deployment_circuit_breaker { enable = true; rollback = true }
}

7.3 ALB + listeners

Only the api-gateway is exposed:

resource "aws_lb" "api" {
  name               = "paymentgw-${var.env}"
  load_balancer_type = "application"
  subnets            = module.network.public_subnet_ids
  security_groups    = [aws_security_group.alb.id]
  enable_http2       = true
  drop_invalid_header_fields = true
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.api.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = aws_acm_certificate.api.arn
  default_action {
    type             = "forward"
    target_group_arn = module.api_gateway.target_group_arn
  }
}

resource "aws_lb_listener" "http_redirect" {
  load_balancer_arn = aws_lb.api.arn
  port              = 80
  protocol          = "HTTP"
  default_action {
    type = "redirect"
    redirect { port = "443"; protocol = "HTTPS"; status_code = "HTTP_301" }
  }
}

7.4 Wiring services

module "api_gateway" {
  source            = "../../modules/ecs-service"
  name              = "api-gateway"
  cluster_arn       = aws_ecs_cluster.main.arn
  image             = "${aws_ecr_repository.svc["api-gateway"].repository_url}:${var.image_tag}"
  cpu               = 512
  memory            = 1024
  port              = 8080
  desired_count     = 2
  attach_to_alb     = true
  alb_listener_arn  = aws_lb_listener.https.arn
  service_discovery = aws_service_discovery_http_namespace.svc.arn
  env = {
    SPRING_PROFILES_ACTIVE = "aws-${var.env}"
    JWT_ISSUER_URI         = var.jwt_issuer
  }
  subnet_ids = module.network.private_app_subnet_ids
}

module "payment_service" {
  source        = "../../modules/ecs-service"
  name          = "payment-service"
  ...
  env = {
    SPRING_PROFILES_ACTIVE = "aws-${var.env}"
    BANK_ADAPTER_BASE_URL  = "http://bank-adapter-service:8092"
    FRAUD_CHECK_BASE_URL   = "http://fraud-check-service:8093"
    KAFKA_BOOTSTRAP        = aws_msk_cluster.kafka.bootstrap_brokers_sasl_iam
    REDIS_HOST             = aws_elasticache_replication_group.redis.primary_endpoint_address
  }
  secrets = {
    DB_USERNAME = "${aws_secretsmanager_secret.payment_db.arn}:username::"
    DB_PASSWORD = "${aws_secretsmanager_secret.payment_db.arn}:password::"
    REDIS_AUTH  = aws_secretsmanager_secret.redis.arn
  }
}
# … repeat for the other services …

7.5 Auto-scaling

resource "aws_appautoscaling_target" "svc" {
  service_namespace  = "ecs"
  resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.this.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = 2
  max_capacity       = 20
}

resource "aws_appautoscaling_policy" "cpu" {
  name               = "cpu-tt"
  service_namespace  = aws_appautoscaling_target.svc.service_namespace
  resource_id        = aws_appautoscaling_target.svc.resource_id
  scalable_dimension = aws_appautoscaling_target.svc.scalable_dimension
  policy_type        = "TargetTrackingScaling"
  target_tracking_scaling_policy_configuration {
    target_value = 60
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
  }
}

7.6 First deploy

# from infra/terraform/envs/dev/
terraform init
terraform workspace new dev || terraform workspace select dev
terraform plan  -var-file=terraform.tfvars -var image_tag=0.1.0
terraform apply -var-file=terraform.tfvars -var image_tag=0.1.0

To build & push images:

SERVICES=(api-gateway payment-service payment-query-service bank-adapter-service fraud-check-service notification-service)
for s in "${SERVICES[@]}"; do
  ./gradlew :services:$s:bootBuildImage --imageName=$s:0.1.0
  docker tag  $s:0.1.0 $ECR/$s:0.1.0
  docker push $ECR/$s:0.1.0
done

8. Phase 4 — Observability

  • Logs. ECS awslogs driver → CloudWatch Logs. One log group per service. Retention 30 days dev / 365 days prod.
  • Metrics. Container Insights (ECS-level), plus each service exposes Prometheus metrics on /actuator/prometheus. Use AWS Managed Prometheus + Grafana to scrape; OR use the CloudWatch agent as a sidecar.
  • Traces. X-Ray daemon as a sidecar OR OpenTelemetry collector forwarding to AWS Distro for OpenTelemetry (ADOT).
  • Dashboards. Per service: RED metrics, error rate, p95 latency. Cross-service: Kafka consumer lag, DB pool saturation.
  • Alarms. Examples:
  • payment-service 5xx rate > 1% for 5 min → page.
  • Bank-adapter circuit-breaker open > 2 min → ticket.
  • RDS CPU > 80% for 10 min → ticket.
  • Kafka under-replicated partitions > 0 → page.

See 04-observability.md for full setup.

9. Phase 5 — CI/CD

See 03-cicd.md. High-level:

  1. PR build: gradle :services:<changed>:test, contract verification, image scan.
  2. Merge to main: build all changed services, push to ECR, run Terraform plan in staging, auto-apply, run smoke tests.
  3. Manual approval to promote a tag to prod.

10. Phase 6 — Hardening

10.1 WAF in front of ALB

  • Managed rule groups: AWS Common, Known Bad Inputs, SQL injection, Linux OS.
  • Rate-based rule: 2000 requests/5 min per IP.
  • Geo-block list of countries you don't operate in.
  • Custom rule: require Authorization header on /api/payments.

10.2 Network ACLs

A second line of defence beyond security groups. Default-deny on data subnets; allow only ephemeral return + RDS/Redis/MSK ports from app subnets.

10.3 IAM least-privilege

  • Each ECS task has its own task role with policies scoped to only the resources it needs (e.g. payment-service task role can secretsmanager:GetSecretValue on its own secret only).
  • The deployer role used by CI is scoped to the resources it manages (no *:*).

10.4 PCI considerations

  • Even though we never store full PAN, we apply PCI-DSS-aligned controls:
  • Cardholder data environment (CDE) is the VPC; documented as such.
  • Quarterly access reviews.
  • Annual penetration test.
  • All RDS data encrypted at rest (KMS) and in transit (TLS).
  • 12-month retention of CloudTrail.

10.5 Backup & restore

  • RDS: 14-day point-in-time recovery in dev, 35-day in prod. Cross-region snapshot copy nightly to a second region.
  • S3 (logs, terraform state): versioning + cross-region replication.
  • Practice restore drills quarterly.

11. Day-2 operations

11.1 Deploys

Two patterns:

  • Rolling (default ECS): cheap, easy. Set minimum_healthy_percent=100, maximum_percent=200.
  • Blue/green via CodeDeploy. Two target groups; CodeDeploy shifts traffic on success. Auto-rollback on alarms.

11.2 Rollback

# rollback to a specific image tag
terraform apply -var image_tag=0.0.9
# or force a previous task definition
aws ecs update-service \
  --cluster paymentgw-prod \
  --service payment-service \
  --task-definition payment-service:42

11.3 Scaling events

  • Watch ECSServiceAverageCPUUtilization. If your scaling kicks in late, switch to a request-count-per-target metric on ALB target groups (more responsive).

11.4 Incident workflow

  1. Pager fires.
  2. Open the service's runbook (linked from the alarm).
  3. Run the diagnostic dashboard.
  4. Mitigate (rollback, scale, drain a bad node).
  5. Post-incident review within 5 working days; produce action items; track to closure.

12. Cost optimisation

  • Right-size tasks. Start small; let Container Insights tell you.
  • Use Fargate Spot for non-critical services (dev environment, batch).
  • Compress logs (set retention; export to S3 + Glacier for archive).
  • Single-AZ NAT in dev — savings of ~$70/month per AZ.
  • Reserved Instances / Savings Plans for stable production fleets.
  • Consolidate DBs. One RDS cluster with multiple databases is cheaper than one cluster per service.
  • MSK Serverless for low-traffic environments.
  • Schedule dev environments off at nights/weekends with EventBridge → ECS service desired_count=0.

A worked example of dev-only cost compression appears in §15.7 of the main guide.

13. Disaster recovery

Recovery objective Strategy
RPO: ≤ 5 minutes RDS automated backups + cross-region snapshot every 6 h.
RTO: ≤ 1 hour (prod) Hot-standby region with Route 53 failover and IaC pre-applied.
Service code Container images replicated to ECR in DR region.
Configuration Secrets Manager replication enabled on key secrets.
Topics MSK MirrorMaker 2 replicates topics to DR cluster.
State Documented "fail to DR" runbook, drilled twice a year.

14. Migration path: ECS → EKS

If you outgrow ECS (or want one platform across teams), migration is incremental:

  1. Stand up an EKS cluster in the same VPC.
  2. Deploy non-stateful services first (api-gateway, fraud-check-service) on EKS using the same images.
  3. Use a shared service discovery (Cloud Map for ECS, AWS Load Balancer Controller + ExternalDNS for EKS).
  4. Drain ECS services as their EKS equivalents reach parity.
  5. Decommission ECS cluster.

You don't need to rewrite Java code: it's all the same containers. Manifests change from Terraform aws_ecs_service to a Helm chart.