A complete, hands-on course for refactoring a Spring Boot monolith into a production-grade microservices system, and deploying it to AWS.

Source monolith: ../payment-gateway-challenge-java Target system: this folder (payment-gateway-microservices/)

Table of Contents

  1. How to use this guide
  2. Microservices in 10 minutes
  3. Reading the existing monolith
  4. Decomposition: from monolith to services
  5. Target architecture
  6. Project layout & build setup
  7. Service-by-service implementation
  8. Shared payment-events module
  9. discovery-server (Eureka)
  10. config-server (Spring Cloud Config)
  11. api-gateway (Spring Cloud Gateway)
  12. payment-service (write-side / orchestrator)
  13. payment-query-service (read-side / CQRS)
  14. bank-adapter-service (acquiring bank gateway)
  15. fraud-check-service
  16. notification-service
  17. Inter-service communication patterns
  18. Data management: DB-per-service, CQRS, outbox, saga
  19. Resilience: timeouts, retries, circuit breakers, bulkheads
  20. Security: JWT, mTLS, secrets, PCI-DSS hygiene
  21. Observability: logs, metrics, traces
  22. Testing strategy
  23. Local development with Docker Compose
  24. Deploying to AWS — full walkthrough
  25. CI/CD with GitHub Actions
  26. Operational runbook
  27. Common pitfalls and anti-patterns
  28. Glossary
  29. Further reading

1. How to use this guide

This document is both a textbook and a reference for the working code in this folder. Read it linearly the first time, then jump back to specific sections as you build.

The companion code is intentionally runnable end-to-end on your laptop (Docker Compose) and deployable to AWS (Terraform skeleton in infra/terraform). Every Java module mirrors the structure of the original monolith so you can compare side-by-side.

Audience Suggested reading path
Backend engineer new to microservices §2 → §4 → §5 → §7 → §14
Senior engineer / architect §4 → §5 → §8 → §9 → §10 → §15
DevOps / SRE §11 → §12 → §14 → §15 → §16 → §17
Interview prep §2 → §3 → §4 → §8 → §9 → §10 → §18

Convention. Throughout this guide, code blocks reference real files in this folder. When you see services/payment-service/src/..., you can open that file directly.

2. Microservices in 10 minutes

2.1 What is a microservice?

A microservice is an independently deployable process that owns:

  1. One bounded context of the domain (one cohesive piece of business capability),
  2. Its own data store (no other service reads its tables directly),
  3. A well-defined contract (HTTP/gRPC API and/or asynchronous events),
  4. Its own deployment lifecycle (you can deploy payment-service v3 without redeploying notification-service).

The architectural style assembles many such services behind a thin edge (API gateway) and connects them with synchronous calls and asynchronous events.

2.2 Why use microservices?

Driver What it buys you
Independent deployability Small blast radius, faster release cadence per team
Polyglot persistence Each service picks the right data store (RDS, DynamoDB, Redis)
Scale per workload Scale bank-adapter-service to 50 replicas without bloating others
Team autonomy One team owns one service end-to-end (you build it, you run it)
Failure isolation A leak or crash in notification-service does not stop payments

2.3 Why not to use microservices?

Microservices add real cost — don't pay it without reason:

  • Network calls replace method calls. Latency, partial failure, retries, idempotency all become first-class concerns.
  • Distributed data is hard. No more @Transactional across services — you need sagas or eventual consistency.
  • Operational tax. N services × M environments = a lot of pipelines, dashboards, alerts.
  • Cognitive load. Onboarding a new engineer means understanding service boundaries, contracts, and runbooks.

Rule of thumb: start with a well-modularised monolith. Split out a service only when one of these is true: (a) it has clearly different scaling needs, (b) a different team owns it, (c) it is in a different security/compliance perimeter, (d) it has a different release cadence.

The original payment-gateway-challenge-java is small enough that a monolith would be perfectly defensible in production. We split it here for teaching purposes — to show patterns you will need on a real project.

2.4 Twelve-Factor recap (the prerequisite mindset)

Before you split a service, your app should already be 12-Factor (https://12factor.net):

  1. One codebase tracked in version control, many deploys.
  2. Explicitly declare and isolate dependencies.
  3. Store config in the environment.
  4. Treat backing services as attached resources.
  5. Strictly separate build, release, and run stages.
  6. Execute the app as one or more stateless processes.
  7. Export services via port binding.
  8. Scale out via the process model.
  9. Maximise robustness with fast startup and graceful shutdown.
  10. Keep development, staging, and production as similar as possible.
  11. Treat logs as event streams.
  12. Run admin tasks as one-off processes.

The original monolith is almost there. We will fix the remaining gaps as part of the split.

3. Reading the existing monolith

Open ../payment-gateway-challenge-java. Its structure:

src/main/java/com/checkout/payment/gateway/
├── PaymentGatewayApplication.java
├── controller/PaymentGatewayController.java
├── service/PaymentGatewayService.java
├── repository/PaymentsRepository.java        # in-memory HashMap
├── model/                                    # request/response/bank DTOs + mapper
├── enums/PaymentStatus.java
├── exception/                                # global handler + domain exceptions
├── validation/                               # @ValidExpiryDate
└── configuration/                            # RestTemplate bean, request logger

It exposes:

  • POST /api/payments — create a payment (validates, sends to bank, stores result).
  • GET /api/payments/{id} — read a payment.

It calls a single downstream acquiring bank simulator at http://localhost:8080/payments.

3.1 Concerns mixed inside the monolith

Concern Where it lives today
HTTP transport PaymentGatewayController
Input validation PostPaymentRequest annotations + ExpiryDateValidator
Persistence PaymentsRepository (in-memory)
Bank integration PaymentGatewayService.sendPaymentRequestToBank
Mapping PaymentMapper (static methods)
Cross-cutting logging RequestResponseLogger
Error translation CommonExceptionHandler

Notice that PaymentGatewayService does three jobs at once: 1. Receive a request. 2. Talk to the bank. 3. Persist the result.

That is exactly the kind of method we want to break up — each job becomes its own service in §5.

4. Decomposition: from monolith to services

4.1 Domain-Driven Design refresher

A bounded context is a portion of the business where a particular language (the ubiquitous language) is consistent. Inside it, "Payment" means one thing; outside, it might mean something different (e.g. an Accounts team's "Payment" is a settlement, not an authorisation).

Steps to find boundaries:

  1. Event-storm the domain. List every business event ("PaymentSubmitted", "PaymentAuthorised", "FraudFlagged", …).
  2. Group events by aggregate. An aggregate is the consistency boundary inside one service — operations on it must be atomic.
  3. Identify commands and queries. Commands change state; queries read state. They might live in different services (CQRS).
  4. Identify external collaborators. The acquiring bank is a context outside our system — give it an anti-corruption layer (the bank-adapter-service).
  5. Apply the "two-pizza team" rule. A service should be small enough that one team can own it.

4.2 Candidate services for the payment gateway

Bounded context Service name Owns
Receive & orchestrate a payment payment-service payments write model, saga state
Read previously processed payments payment-query-service payments_view denormalised read model
Talk to the acquiring bank bank-adapter-service All HTTP plumbing for the bank
Pre-authorisation risk screening fraud-check-service Risk rules, blocked card list
Customer/merchant notifications notification-service Email/webhook delivery, retries
Cross-cutting: routing & auth at edge api-gateway Public REST surface, JWT, rate limit
Cross-cutting: service registry discovery-server Eureka registrations
Cross-cutting: centralised config config-server Externalised config (git-backed)

4.3 Why this split?

  • payment-service vs payment-query-service (CQRS). Reads and writes have very different scaling profiles. Separating them lets us cache/replicate the read side aggressively and keep writes consistent.
  • bank-adapter-service as an anti-corruption layer. The bank's API is outside our control. Wrapping it in a dedicated service means a bank-side change touches one codebase, not three.
  • fraud-check-service is a separate compliance perimeter. It often has different access controls and audit requirements; isolating it eases audits.
  • notification-service is async-first. It is allowed to be slow/retry without blocking payments.

4.4 What we are deliberately not splitting

  • We do not create a separate "validation-service" — input validation is a transport concern that lives where the request arrives (payment-service and api-gateway).
  • We do not split each entity into its own service (the "entity service" anti-pattern). One service owns one capability, not one table.

5. Target architecture

5.1 Logical diagram

                          ┌────────────────────┐
   merchant POST /payments│   API Gateway      │   JWT verify, rate limit, CORS
   GET  /payments/{id}   ─►│ (Spring Cloud GW)  │
                          └────────┬───────────┘
                                   │ /api/payments        /api/payments/{id}
                          ┌────────▼─────────┐    ┌────────▼─────────┐
                          │ payment-service  │    │ payment-query-svc│
                          │  (write-side)    │    │  (read-side)     │
                          └─┬────┬──────────┬┘    └────────┬─────────┘
                            │    │          │              │
              sync REST     │    │          │ Kafka        │ reads
                ┌───────────▼┐   │          │ payments.*   │ payments_view
                │bank-adapter│   │          │              │   (Postgres)
                │  service   │   │          ▼              │
                └─────┬──────┘   │   ┌──────────────┐      │
                      │          │   │  Kafka       │      │
                      ▼          │   │  (MSK in AWS)│      │
            external bank  sync REST └──────┬───────┘      │
                                            │              │
                                ┌───────────▼─────┐  ┌─────▼──────┐
                                │ notification-   │  │ Postgres   │
                                │   service       │  │ (RDS in AWS)│
                                └─────────────────┘  └────────────┘

           sync REST          ┌───────────────────┐
        ┌───────────────────► │ fraud-check-svc   │
                              └───────────────────┘

   Platform (cross-cutting):
   - discovery-server (Eureka)  - service registry
   - config-server              - centralised config
   - Redis                      - rate limit + idempotency cache
   - Prometheus + Grafana       - metrics
   - OpenTelemetry collector    - traces
   - ELK / CloudWatch Logs      - logs

5.2 Happy-path sequence: POST /api/payments

merchant ──► api-gateway ──► payment-service
                                  │
                                  ├─► fraud-check-service  (sync REST, ≤ 200 ms)
                                  │
                                  ├─► bank-adapter-service ──► acquiring bank
                                  │       (sync REST,            (external)
                                  │        circuit-broken)
                                  │
                                  ├─► Postgres (own DB)         insert payment row
                                  │
                                  └─► Kafka topic "payments.events"
                                         │
                                         ├─► payment-query-service  (project view)
                                         └─► notification-service   (deliver)

Key design choices on this path: - Idempotency key (Idempotency-Key header) is checked at payment-service against Redis to make retries safe. - Outbox pattern is used for the Kafka publish so the DB insert and the event publish cannot diverge. - Compensating action (refund / mark failed) runs if the bank declines after we already wrote the row in some advanced flows.

5.3 Service catalogue (cheat sheet)

Service Port (local) DB / store Speaks Spoken to by
api-gateway 8080 — HTTP merchants
discovery-server 8761 in-memory HTTP/Eureka every service
config-server 8888 git repo HTTP every service
payment-service 8090 Postgres payments HTTP, Kafka pub api-gateway
payment-query-service 8091 Postgres payments_view HTTP, Kafka sub api-gateway
bank-adapter-service 8092 (stateless) HTTP payment-service
fraud-check-service 8093 Postgres fraud HTTP payment-service
notification-service 8094 Postgres notifications Kafka sub, SMTP (subscribes to events)

6. Project layout & build setup

payment-gateway-microservices/
├── MICROSERVICES_TEACHING_GUIDE.md     ← you are here
├── README.md
├── settings.gradle                      ← multi-module project
├── build.gradle                         ← shared plugin/dependency mgmt
├── gradle.properties
├── docker-compose.yml                   ← run the whole platform locally
├── docs/                                ← deep-dive sub-guides
│   ├── 01-architecture-decisions.md
│   ├── 02-aws-deployment.md
│   ├── 03-cicd.md
│   └── 04-observability.md
├── shared/
│   └── payment-events/                  ← shared event/DTO library (versioned!)
├── services/
│   ├── api-gateway/
│   ├── discovery-server/
│   ├── config-server/
│   ├── payment-service/
│   ├── payment-query-service/
│   ├── bank-adapter-service/
│   ├── fraud-check-service/
│   └── notification-service/
└── infra/
    ├── docker/                          ← service Dockerfiles
    ├── imposters/                       ← bank simulator (Mountebank)
    ├── config-repo/                     ← git-backed config consumed by config-server
    ├── k8s/                             ← K8s manifests (alt to ECS)
    └── terraform/                       ← AWS Terraform IaC

6.1 Multi-module Gradle

A multi-module Gradle build keeps shared versions in one place but lets each service build/deploy independently. See settings.gradle and build.gradle at the root.

6.2 The shared/payment-events module

We version a small library that contains only the wire-level event payloads shared between producers and consumers (PaymentSubmittedEvent, PaymentAuthorisedEvent, etc.). Two rules:

  1. Never put internal domain types in the shared module — only the published contract.
  2. Keep it backward-compatible. Add fields, never rename or remove without a major version bump.

If you can avoid even this shared module by using JSON Schemas or Protobuf published to a registry — even better. We use a tiny JAR here for simplicity.

7. Service-by-service implementation

The remainder of this section walks each service. The full source lives next to this guide; what follows highlights the why and the non-obvious parts.

7.1 Shared payment-events module

Folder: shared/payment-events/

A thin library that exposes only POJOs (plus Jackson annotations). It must have zero Spring dependencies — anything can consume it.

Key event:

public record PaymentEvent(
    String eventId,
    String eventType,        // "PAYMENT_SUBMITTED" | "PAYMENT_AUTHORISED" | "PAYMENT_DECLINED" | "PAYMENT_FAILED"
    UUID paymentId,
    String merchantId,
    String currency,
    Integer amount,
    String cardNumberLastFour,
    String status,
    Instant occurredAt,
    String correlationId
) {}

Topics produced/consumed by services use this single envelope and switch on eventType. If you prefer separate types per topic, that's also fine — be consistent.

7.2 discovery-server (Eureka)

Folder: services/discovery-server/

A vanilla Spring Cloud Netflix Eureka server. Every other service registers here and discovers peers via logical names like bank-adapter-service.

Why have it?

  • Removes hard-coded URLs from your code and config.
  • Enables client-side load balancing via LoadBalancerClient.
  • Lets you do blue/green and rolling deploys without DNS dances.

AWS note. When you move to ECS or EKS, you can drop Eureka and use AWS Cloud Map / ECS Service Discovery instead. Code doesn't need to change if you use Spring's DiscoveryClient abstraction.

7.3 config-server (Spring Cloud Config)

Folder: services/config-server/

Centralises configuration. Each service downloads its config at boot (and can refresh at runtime via /actuator/refresh). Backed by a git repository (infra/config-repo/).

Why have it?

  • One place to rotate database credentials, feature flags, retry budgets.
  • Separates code from config (12-Factor §3).
  • Audit trail via git history.

AWS note. Replace this with AWS AppConfig or Parameter Store / Secrets Manager in production. The pattern is identical: pull config at boot, refresh on signal.

7.4 api-gateway (Spring Cloud Gateway)

Folder: services/api-gateway/

The single entry point. Responsibilities:

  1. Routing. /api/payments (POST) → payment-service, /api/payments/{id} (GET) → payment-query-service.
  2. AuthN/AuthZ. Verifies a JWT (issued by an external IdP — e.g. Cognito, Auth0). Forwards merchantId claim downstream.
  3. Rate limiting. Redis-backed (RequestRateLimiter filter), keyed by merchant + IP.
  4. Cross-cutting filters. Correlation-id injection, request/response logging, CORS.

Anti-pattern alert. Do not put business logic in the gateway. The gateway should be config-heavy and code-light. Anything that smells like a if (currency == "GBP") {…} belongs in a service.

7.5 payment-service (write-side / orchestrator)

Folder: services/payment-service/

The most behaviourally rich service. It owns the transaction-creation saga.

Endpoints

Method Path Purpose
POST /api/payments Create a payment

Steps for POST /api/payments:

  1. Read & echo the Idempotency-Key header.
  2. Validate the request (Jakarta Bean Validation + custom expiry validator).
  3. Idempotency check. Look up the key in Redis (or in the idempotency_keys table). If present and COMPLETED, return the stored response.
  4. Fraud check. Synchronous REST to fraud-check-service with a short timeout (e.g. 200 ms). On BLOCK, persist as REJECTED and return.
  5. Bank authorisation. Synchronous REST to bank-adapter-service (circuit-breaker wrapped).
  6. Persist the payment row (payments table) and an outbox row (payment_outbox) inside the same transaction.
  7. A scheduled poller drains the outbox table to Kafka topic payments.events.
  8. Return the API response.

Why an outbox? If you publish to Kafka after commit, a crash between commit and publish loses the event. Writing both inside one transaction and having a separate publisher poll guarantees at-least-once delivery without two-phase commit.

7.6 payment-query-service (read-side / CQRS)

Folder: services/payment-query-service/

Subscribes to payments.events and projects them into a denormalised read table payments_view. Serves GET /api/payments/{id}.

Why split read from write?

  • The read model can be denormalised (e.g. join merchant info, last-four card data) without complicating writes.
  • It can scale independently (often 10× more reads than writes).
  • It is naturally cache-friendly.

Eventual consistency. A merchant who submits a payment and immediately reads it might not see it for a few hundred milliseconds. Our API contract calls this out and provides Location header with the resource id — clients should retry with exponential backoff if they need read-your-writes.

7.7 bank-adapter-service (acquiring bank gateway)

Folder: services/bank-adapter-service/

A thin anti-corruption layer: external bank API ↔ internal canonical model.

Responsibilities:

  • Translate BankAuthorisationCommand ↔ bank-specific request/response.
  • Apply timeouts, retries with backoff, circuit breaker (Resilience4j).
  • Mask PAN/CVV from logs.
  • Surface a stable error taxonomy: BANK_TIMEOUT, BANK_DECLINED, BANK_UNAVAILABLE.

Multi-bank ready: a BankClient interface plus per-acquirer implementations + a routing strategy (e.g. by BIN, currency, merchant config). For the demo we ship one implementation talking to the same Mountebank simulator as the original challenge.

7.8 fraud-check-service

Folder: services/fraud-check-service/

Stateless rules engine for the demo:

  • BIN blacklist (in-DB list of card BINs we never accept).
  • Velocity check (more than N attempts from same merchant in last 60s → flag).
  • Currency-amount thresholds.

Returns { "decision": "ALLOW" | "REVIEW" | "BLOCK", "reasons": [...] }. The orchestrator (payment-service) decides what to do with REVIEW.

7.9 notification-service

Folder: services/notification-service/

A pure consumer. Subscribes to payments.events and:

  • Persists the event to notifications table for audit.
  • Sends an email/webhook (we use a stub LoggingNotifier in the demo).
  • Implements idempotent consumer — uses eventId as a unique constraint to deduplicate.

This service shows that microservices are not all about REST. Async eventing decouples the notification path from the payment hot path.

8. Inter-service communication patterns

8.1 Sync vs Async — when to use which

Choose sync REST/gRPC when Choose async events when
The caller needs the result now to proceed The caller can fire-and-forget
The downstream is fast (≤ a few hundred ms) The downstream is slow or bursty
You need request/response semantics for business reasons You need fan-out to many consumers
The callee owns authoritative data the caller needs You need temporal decoupling / replay

In our system:

  • payment-service → fraud-check-service: sync (need decision before bank call).
  • payment-service → bank-adapter-service: sync (need authorisation before responding).
  • payment-service → payment-query-service, notification-service: async (read model & emails can lag).

8.2 Synchronous: what to get right

  • Timeouts everywhere. Connect timeout and read timeout. No exceptions.
  • Bounded retries. Retries × per-attempt timeout < caller's overall budget.
  • Idempotency. Either the operation is naturally idempotent (GET) or you carry an Idempotency-Key.
  • Backpressure. Use bulkheads (separate thread pools) so one slow downstream cannot exhaust the caller.

We use Spring's RestClient (or WebClient) wrapped with Resilience4j for circuit breaker, retry, time limiter — see §10.

8.3 Asynchronous: Kafka topics

Topic Producer Consumers Key
payments.events payment-service payment-query-service, notification-svc paymentId
fraud.decisions fraud-check-svc (audit) payment-query-service paymentId
bank.responses (optional) bank-adapter payment-service paymentId

Use the partition key correctly. Keying by paymentId ensures all events for one payment land on the same partition → strict ordering per payment, parallelism across payments.

Schema discipline. Use a registry (Confluent / AWS Glue Schema Registry) in production. For the demo we keep schemas as Java records in shared/payment-events.

9. Data management: DB-per-service, CQRS, outbox, saga

9.1 Database-per-service

Each service owns its own schema (or its own database in production) and only that service writes/reads it. This:

  • Lets each team pick the right tech (Postgres for payments, Redis for idempotency, etc.).
  • Removes coupling at the data layer (the worst kind of coupling).
  • Forces well-defined APIs between services.

Forbidden: another service's batch job that "just reads" your tables. Wrap it in an API or expose an event stream.

9.2 CQRS in our system

payment-service is the command side. payment-query-service is the query side. They share no tables. The query side is rebuilt purely from events — meaning if the read model schema changes, you can replay events from the beginning to repopulate it. Powerful.

9.3 Transactional outbox

Inside payment-service, when we authorise a payment we want to (a) insert into payments and (b) publish to Kafka. We can't have a JTA transaction across Postgres and Kafka. So:

BEGIN;
  INSERT INTO payments (...);
  INSERT INTO payment_outbox (id, payload, created_at) VALUES (...);
COMMIT;

A separate scheduled job reads payment_outbox and publishes to Kafka. After Kafka acks, the row is marked published (or deleted). On crash, we re-read unpublished rows on restart — at-least-once.

Consumers must therefore be idempotent (use eventId for dedup).

9.4 Saga for multi-step flows

A saga is a long-running business transaction that spans services. Two flavours:

  • Orchestration saga. A central coordinator (here, payment-service) calls each step and reacts to outcomes.
  • Choreography saga. Each service reacts to events emitted by others.

Our payment flow is a small orchestration saga: fraud-check → bank-authorise → persist. If the bank call fails after the row is committed, a compensating action (REVERSE_AUTHORISATION event consumed by bank-adapter-service) is issued.

Pick orchestration when steps and compensation are well-defined; choreography when the workflow evolves and many services participate.

9.5 Schema migration

Each service owns its DB migrations (Flyway in our setup, under src/main/resources/db/migration/). Migrations run at service startup with spring.flyway.enabled=true. Forward-only — no down migrations in production.

10. Resilience: timeouts, retries, circuit breakers, bulkheads

10.1 The four levers

  1. Timeout. Cap how long any single call can take.
  2. Retry. Retry transient failures with backoff and jitter.
  3. Circuit breaker. When a downstream is on fire, stop hammering it.
  4. Bulkhead. Cap how many concurrent calls one part of the system can make to one dependency.

We configure all four with Resilience4j in each calling service. Example (YAML):

resilience4j:
  timelimiter:
    instances:
      bankAdapter: { timeoutDuration: 2s }
  retry:
    instances:
      bankAdapter:
        maxAttempts: 3
        waitDuration: 200ms
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - org.springframework.web.client.ResourceAccessException
  circuitbreaker:
    instances:
      bankAdapter:
        slidingWindowSize: 50
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 5
  bulkhead:
    instances:
      bankAdapter:
        maxConcurrentCalls: 25
        maxWaitDuration: 100ms

10.2 Idempotency

  • Sync endpoints that mutate state must accept an Idempotency-Key header. Store key + response; serve the stored response on retries.
  • Async consumers must dedupe by eventId.

10.3 Fail fast at the edge

The api-gateway enforces an end-to-end budget (e.g. 5 seconds). When a call hits that, every downstream sees its Deadline-Header and can short-circuit work that won't make it back in time.

10.4 Graceful degradation

If fraud-check-service is down, do you reject all payments or accept them risk-on? It's a business decision, not a technical one. We default to decision=REVIEW on fraud-check failure (configurable feature flag) so the payment doesn't auto-block, but is queued for later review.

11. Security: JWT, mTLS, secrets, PCI-DSS hygiene

11.1 Authentication at the edge

The gateway validates JWT bearer tokens issued by an external IdP. We support:

  • HS256 / RS256 signature validation against a JWKS URL.
  • Standard claims: iss, aud, exp, sub, merchantId.
  • Token-to-header propagation: forwarded as X-Merchant-Id to downstream services.

11.2 Authentication between services

Two acceptable patterns:

  1. Service mesh / mTLS. Each pod gets a SPIFFE identity; the mesh (Istio, Linkerd, AWS App Mesh) enforces mutual TLS.
  2. Token relay. Services use OAuth2 client credentials with a tiny scope per service. We illustrate this with Spring Security's WebClient filter.

For demos, we use plain HTTP between services on a private network. Never do that in production; you almost certainly need mTLS.

11.3 Secrets

  • Local dev: .env files (git-ignored), loaded by Docker Compose.
  • AWS: Secrets Manager (DB credentials, Stripe-style API keys), Parameter Store (non-secret config). Services read at boot via the AWS SDK or via Spring Cloud AWS.
  • Never bake secrets into Docker images.

11.4 PCI-DSS hygiene

Our system handles card data, so we apply minimum hygiene from the original monolith and tighten it:

  • Card PAN: never logged. Only last 4 stored (in DB and events).
  • CVV: never stored, never logged.
  • TLS 1.2+ end to end.
  • Scoped IAM: payment-service cannot touch notifications DB and vice versa.
  • Audit: a separate write-only log topic, immutable.

Full PCI-DSS scope reduction (tokenisation, Secure Acceptance) is out of scope for this guide but the architecture is friendly to it: insert a tokeniser-service between gateway and payment-service.

12. Observability: logs, metrics, traces

The "three pillars" — but in microservices, traces are the most valuable.

12.1 Logging

  • Structured JSON logs via Logback logstash-logback-encoder.
  • Inject correlationId (== traceId) into MDC on every request. We use Spring Cloud Sleuth / Micrometer Tracing — propagated via traceparent (W3C Trace Context) across HTTP and Kafka.
  • Ship logs to Elastic / CloudWatch Logs / Loki.

12.2 Metrics

  • Spring Boot Actuator + Micrometer exposes a Prometheus endpoint on /actuator/prometheus.
  • Standard service-level metrics:
  • RED: rate, errors, duration per endpoint.
  • USE: utilisation, saturation, errors per resource (CPU, mem, DB pool).
  • Domain metrics:
  • payments_authorised_total{currency=...}
  • payments_declined_total{reason=...}
  • bank_call_duration_seconds_bucket

12.3 Tracing

  • OpenTelemetry SDK → OTLP collector → Jaeger / Tempo / X-Ray.
  • Every span has tags: service.name, payment.id (never PAN), merchant.id.
  • Critical: trace across Kafka (extract context from headers in consumer).

12.4 SLOs and alerts

For each service, define:

  • Availability SLO: e.g. 99.9% successful POST /api/payments per rolling 30 days.
  • Latency SLO: p95 ≤ 500 ms.
  • Error budget burn alerts: page on fast burn, ticket on slow burn.

13. Testing strategy

13.1 Pyramid

                    e2e
                  ┌─────┐
              contract
            ┌───────────┐
        component / integ
      ┌────────────────────┐
        unit (the bulk)
   └──────────────────────────┘
  • Unit tests. Pure logic, no Spring. Lots of them. Fast.
  • Component / slice tests. @WebMvcTest, @DataJpaTest, @RestClientTest. One service in isolation, doubles for collaborators.
  • Contract tests. Pact or Spring Cloud Contract. Producer publishes a contract; every consumer's CI verifies it. Catches breaking API changes before runtime.
  • E2E. A small set running the full Docker Compose stack, hitting the gateway. Don't use as a substitute for the lower layers.

13.2 Useful tooling

Layer Tool
Unit JUnit 5, Mockito, AssertJ
MVC slice MockMvc, @WebMvcTest
HTTP doubles WireMock
Kafka Testcontainers (KafkaContainer)
DB Testcontainers (PostgreSQLContainer)
Contracts Pact / Spring Cloud Contract
Load k6, Gatling

13.3 Don't forget

  • Chaos tests. Kill a pod. Drop network. Toolkit: AWS FIS, Chaos Mesh, ToxiProxy.
  • Schema migration tests. Run Flyway on a fresh DB in CI.
  • Backward-compat tests. Old consumer reads new producer's events.

14. Local development with Docker Compose

The full stack runs with one command:

docker compose up --build

Services come up in dependency order (declared via depends_on and healthchecks, not just startup order). Useful URLs:

URL What
http://localhost:8080/api/payments Public API (gateway)
http://localhost:8761 Eureka dashboard
http://localhost:8888 Config server
http://localhost:9090 Prometheus
http://localhost:3000 Grafana (admin/admin)
http://localhost:16686 Jaeger UI (traces)
http://localhost:8025 Mailhog (notification-svc)

The bank simulator is the same Mountebank-based image as the original challenge, mounted from infra/imposters/.

See docker-compose.yml for the full topology.

15. Deploying to AWS — full walkthrough

A more detailed runbook (with copy-paste Terraform) lives in docs/02-aws-deployment.md. This section is the conceptual map.

15.1 Two viable platforms

Option When to choose
ECS Fargate Smaller team, no Kubernetes expertise, want managed
EKS (K8s) Larger team, multi-cloud, advanced scheduling needs

We use ECS on Fargate in this guide because it has the best managed-service ratio and the cheapest operational tax. Patterns translate to EKS — only the deployment manifests change.

15.2 Building blocks on AWS

Concern AWS service
Container registry ECR
Container runtime ECS Fargate (one Service per microservice)
Public ingress ALB in front of api-gateway Service
DNS / TLS Route 53 + ACM
Service discovery (private) AWS Cloud Map via ECS Service Connect
Database RDS for PostgreSQL (one instance per service or one cluster with multiple DBs)
Cache / idempotency ElastiCache for Redis
Async messaging MSK (managed Kafka) or Amazon MQ
Secrets Secrets Manager, Parameter Store
Config (non-secret) AppConfig (or Parameter Store)
Logs CloudWatch Logs
Metrics CloudWatch Metrics + Managed Prometheus
Traces AWS X-Ray or OpenTelemetry → Tempo
WAF / DDoS AWS WAF, Shield Standard
CDN (static) CloudFront (not needed for this API)

15.3 Network topology

  • One VPC per environment (dev, staging, prod).
  • Three AZs, public + private subnets in each.
  • NAT Gateway in each AZ for private-subnet egress.
  • VPC endpoints for S3, ECR, Secrets Manager, CloudWatch — keeps traffic on the AWS backbone, cheaper and more secure.
  • Security groups: each service has its own SG; allow only required ports between SGs (e.g. payment-service-sg → bank-adapter-service-sg:8092).

15.4 Phased plan

  1. Phase 0: Bootstrap (one-off). Create the Terraform state bucket + DynamoDB lock table.
  2. Phase 1: Network. VPC, subnets, route tables, NAT, endpoints.
  3. Phase 2: Platform. RDS, ElastiCache, MSK, ECR repos, Secrets Manager.
  4. Phase 3: Compute. ECS cluster, task definitions, services, ALB, target groups.
  5. Phase 4: Observability. Log groups, dashboards, alarms, X-Ray.
  6. Phase 5: CI/CD. GitHub Actions (or CodePipeline) to build → push to ECR → update ECS service.
  7. Phase 6: Hardening. WAF rules, GuardDuty, Inspector, IAM least-privilege review.

15.5 IaC (Terraform) skeleton

infra/terraform/ ships a small reference implementation:

infra/terraform/
├── main.tf                # provider, backend
├── variables.tf
├── outputs.tf
├── modules/
│   ├── network/           # VPC, subnets, NAT, endpoints
│   ├── ecs-cluster/       # cluster + ALB + Cloud Map namespace
│   ├── ecs-service/       # reusable: task def + service + target group
│   ├── rds/               # Postgres
│   ├── elasticache/       # Redis
│   ├── msk/               # Kafka
│   └── secrets/           # Secrets Manager wiring
└── envs/
    ├── dev/
    └── prod/

Each microservice deploys via the ecs-service module:

module "payment_service" {
  source            = "../../modules/ecs-service"
  name              = "payment-service"
  cluster_arn       = module.cluster.arn
  image             = "${var.ecr_repo}/payment-service:${var.image_tag}"
  cpu               = 512
  memory            = 1024
  port              = 8090
  desired_count     = 2
  env               = local.payment_env
  secrets           = local.payment_secrets
  alb_listener_arn  = module.cluster.private_listener_arn
  service_discovery = module.cluster.cloudmap_namespace_id
}

15.6 Step-by-step on a clean account

  1. Set up AWS account (root + IAM user with limited admin; enable MFA; turn on CloudTrail org-wide; enable GuardDuty).
  2. Create an OIDC connection between GitHub and AWS so CI doesn't use long-lived keys.
  3. Bootstrap state. Manually create an S3 bucket tfstate-paymentgw-prod and a DynamoDB table tfstate-locks.
  4. Run terraform init && terraform apply in envs/dev/.
  5. Build & push images. bash ./gradlew bootBuildImage aws ecr get-login-password | docker login --username AWS --password-stdin <ECR_URI> docker tag payment-service:latest <ECR_URI>/payment-service:0.1.0 docker push <ECR_URI>/payment-service:0.1.0
  6. Deploy services. terraform apply -var image_tag=0.1.0.
  7. Run smoke tests. Hit the ALB DNS with a sample POST /api/payments.
  8. Add custom domain. Route 53 record + ACM cert + ALB listener.
  9. Promote to staging/prod by re-running Terraform with the right tfvars and image tags.

15.7 Cost guidance (rough US-East-1 prices, 2025)

Item Quantity $/month (approx)
Fargate (0.5 vCPU / 1 GB × 8 services × 2 tasks) 16 tasks × 730 h ~ $250
RDS Postgres (db.t3.medium, multi-AZ) 1 cluster ~ $130
ElastiCache (cache.t3.small × 1) 1 node ~ $25
MSK (m5.large × 3 brokers, dev tier) 3 brokers ~ $400
ALB 1 + minor LCU ~ $25
NAT Gateway (3 AZ, light traffic) 3 ~ $100
CloudWatch Logs (10 GB ingest) — ~ $5
Total dev environment — ~ $935

Slim-down tip for dev: use a single-AZ MSK Serverless cluster and a single-AZ NAT — you can get to ~$300/month.

15.8 Production checklist

  • [ ] Multi-AZ for every stateful component.
  • [ ] Backups enabled and restore tested.
  • [ ] Auto-scaling configured (ECS service auto-scaling based on CPU + per-target ALB request count).
  • [ ] Read replicas for Postgres if read-heavy.
  • [ ] Blue/green deploys via CodeDeploy with rollback alarms.
  • [ ] WAF rules: rate limit, OWASP top 10, geo restrictions.
  • [ ] Run game-day drills: kill a task, fail an AZ, rotate a secret.

16. CI/CD with GitHub Actions

The repo includes (see docs/03-cicd.md for the complete files):

  1. .github/workflows/build-test.yml — runs on every PR. Builds, tests, contracts, SAST.
  2. .github/workflows/release.yml — on merge to main. Builds, pushes to ECR, deploys via Terraform.

Highlights:

  • Path filters so a change in services/notification-service/ does not redeploy everything.
  • Reusable workflows for the build/test/push pipeline (one service-pipeline.yml called per service).
  • OIDC to AWS instead of long-lived AWS keys.
  • Image scanning with Trivy in PR.
  • Manual approval gate before prod deploy.

A snippet:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gh-deployer
          aws-region: eu-west-2
      - name: Update ECS service
        run: |
          aws ecs update-service \
            --cluster paymentgw-prod \
            --service payment-service \
            --force-new-deployment

17. Operational runbook

Per service, the team owns a runbook. Minimum sections:

  1. What does this service do? One paragraph.
  2. Endpoints, topics, dependencies.
  3. SLOs and alerts.
  4. How to deploy / roll back.
  5. How to read logs / metrics / traces.
  6. Top 5 alerts and the procedure for each.
  7. Database access, backup/restore steps.
  8. Escalation matrix.

Sample alerts for payment-service:

Alert Threshold Action
http_server_errors_total 5xx rate > 1% over 5 min Page on-call
bank_call_duration_seconds p95 > 1.5 s over 10 min Investigate bank-adapter / circuit
Outbox publisher lag > 30 s Check Kafka, restart publisher pod
Consumer lag on payments.events > 10000 records Scale out consumer service
DB connection pool saturation > 90% for 5 min Increase pool, profile slow queries

18. Common pitfalls and anti-patterns

  1. Distributed monolith. Many small services that must be deployed together because they share DB tables or chatty sync calls. Cure: enforce DB-per-service and async events for non-critical paths.
  2. Shared library hell. A commons jar that depends on Spring and grows forever. Limit shared code to thin wire types.
  3. Per-table services ("entity services"). A customer-service whose only job is CRUD on the customer table. Group by capability, not by table.
  4. Synchronous chains > 3 hops. Latency multiplies, errors cascade. Break with async or denormalisation.
  5. Distributed transactions. No JTA across services. Use saga + outbox.
  6. Missing idempotency. Retries become double-charges.
  7. Versioning by accident. Add fields, don't rename or remove. Keep N-1 compatible with N.
  8. No observability. Without traces, debugging a 5-hop request is misery.
  9. Microservices too early. A 3-engineer team with 12 microservices will drown in pipelines.
  10. Letting the gateway grow business logic. It will become the new monolith.

19. Glossary

Term Meaning
Aggregate A cluster of objects treated as a single unit for data changes (DDD).
Anti-corruption layer A façade that translates an external model to your internal one.
At-least-once Delivery guarantee: each message is delivered once or more.
BFF Backend-for-Frontend. A gateway specialised per client app.
Bounded context A logical boundary inside which terms have a single, consistent meaning.
Bulkhead Isolating resources so a failure in one part does not sink the others.
Choreography saga Saga where services react to events without a central coordinator.
Circuit breaker Pattern that stops calls to a failing dependency for a cool-off period.
CQRS Command Query Responsibility Segregation. Separate write and read models.
DDD Domain-Driven Design.
Eventual consistency A system state that becomes consistent over time, not instantly.
Idempotency Property that performing an operation N times yields the same effect as once.
Orchestration saga Saga driven by a central coordinator service.
Outbox pattern Persisting events in the same DB transaction as state changes.
SLO Service Level Objective. Concrete target for availability/latency.
Twelve-Factor Twelve-principle methodology for SaaS apps (https://12factor.net).

20. Further reading

Books - Sam Newman — Building Microservices (2nd ed.) - Chris Richardson — Microservices Patterns - Eric Evans — Domain-Driven Design - Vaughn Vernon — Implementing Domain-Driven Design - Susan J. Fowler — Production-Ready Microservices - Nicole Forsgren et al. — Accelerate

Online - microservices.io — pattern catalogue (Chris Richardson) - martinfowler.com microservices articles - 12factor.net - AWS Well-Architected Framework - Spring Cloud reference

What's next

You've now read the high-level material. Open:

Then run docker compose up --build and step through each service in §7 with the code open.