Companion to ../MICROSERVICES_TEACHING_GUIDE.md §12.

Distributed systems are only as good as your ability to understand them. This document covers logs, metrics, traces, dashboards, and alerts.

1. The three pillars (and why traces are king)

Pillar Question it answers Source
Logs What exactly happened in service X at time T? application + Kafka + DB
Metrics What is the rate, error rate, and duration (RED) per endpoint? Prometheus / CloudWatch
Traces What was the path of one request across services? OpenTelemetry → Jaeger / X-Ray

In a microservices system, when something is wrong you usually start with a trace, then drill down to logs of the specific span, and use metrics for SLOs and dashboards.

2. Logs

2.1 Standards

  • JSON format, one log per line.
  • Required fields: @timestamp, level, service.name, trace.id, span.id, merchant.id, payment.id (when applicable).
  • Forbidden: PAN, CVV, full magstripe, JWTs, passwords. Apply masking in a logback layout.

2.2 Logback config (sample)

services/payment-service/src/main/resources/logback-spring.xml:

<configuration>
  <springProperty name="appName" source="spring.application.name"/>

  <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <customFields>{"service.name":"${appName}"}</customFields>
      <includeMdcKeyName>traceId</includeMdcKeyName>
      <includeMdcKeyName>spanId</includeMdcKeyName>
      <includeMdcKeyName>merchantId</includeMdcKeyName>
      <includeMdcKeyName>paymentId</includeMdcKeyName>
    </encoder>
  </appender>

  <root level="INFO">
    <appender-ref ref="JSON"/>
  </root>
</configuration>

2.3 Aggregation

Stack Notes
ELK Self-hosted, full control. Heavier to operate.
Loki + Grafana Cheaper for high cardinality.
CloudWatch Logs Default in our AWS deployment. Use Log Insights for queries.

CloudWatch Insights query for finding errors on a specific payment:

fields @timestamp, level, service.name, message, paymentId
| filter paymentId = "fa8b…"
| sort @timestamp asc

3. Metrics

3.1 Spring Boot Actuator + Micrometer

Each service exposes:

  • /actuator/health/liveness — for ECS health checks.
  • /actuator/health/readiness — for ALB target group / Service Connect.
  • /actuator/prometheus — Prometheus-format metrics.

application.yml (excerpt):

management:
  endpoints.web.exposure.include: health,info,metrics,prometheus,refresh
  endpoint.health.probes.enabled: true
  metrics:
    tags:
      service.name: ${spring.application.name}
      env: ${ENV:dev}
    distribution.percentiles-histogram.http.server.requests: true

3.2 Standard service-level (RED) metrics

Already provided by Micrometer:

  • http_server_requests_seconds_count{uri,status} — rate.
  • http_server_requests_seconds_count{status=~"5.."} — error rate.
  • http_server_requests_seconds_bucket{uri,le} — histogram for percentiles.

3.3 Domain metrics (write them yourself)

Inside payment-service:

private final Counter authorised;
private final Counter declined;
private final Timer   bankCallTimer;

public PaymentMetrics(MeterRegistry registry) {
  this.authorised = Counter.builder("payments.authorised").register(registry);
  this.declined   = Counter.builder("payments.declined").register(registry);
  this.bankCallTimer = Timer.builder("payments.bank.call")
      .publishPercentiles(0.5, 0.95, 0.99)
      .register(registry);
}

3.4 Resource metrics

In AWS, ECS Container Insights exposes:

  • ECS/ContainerInsights:CpuUtilized, MemoryUtilized, NetworkRxBytes, etc.
  • AWS/RDS:CPUUtilization, DatabaseConnections, ReadLatency, WriteLatency.
  • AWS/ElastiCache:Engine CPU, Evictions.
  • AWS/Kafka:UnderReplicatedPartitions, BytesInPerSec.

4. Tracing

4.1 Setup

Add to each service:

implementation 'io.micrometer:micrometer-tracing-bridge-otel'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
implementation 'net.ttddyy.observation:datasource-micrometer-spring-boot:1.0.5'  // DB spans

application.yml:

management:
  tracing:
    sampling.probability: 1.0   # 100% in dev, sample down in prod (e.g. 0.1)
otel:
  exporter.otlp.endpoint: ${OTLP_ENDPOINT:http://otel-collector:4317}
  resource.attributes: service.name=${spring.application.name},env=${ENV:dev}

4.2 Propagation

W3C traceparent is propagated automatically over HTTP. For Kafka, configure the producer/consumer interceptors:

@Bean
public ProducerFactory<String, Object> producerFactory(KafkaProperties props) {
  Map<String, Object> cfg = new HashMap<>(props.buildProducerProperties());
  cfg.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,
          List.of(io.opentelemetry.instrumentation.kafkaclients.v2_6.TracingProducerInterceptor.class.getName()));
  return new DefaultKafkaProducerFactory<>(cfg);
}

4.3 Backends

  • Local dev: Jaeger (in docker-compose.yml).
  • AWS: AWS Distro for OpenTelemetry (ADOT) sidecar exporting to AWS X-Ray or a self-managed Tempo cluster.

5. Dashboards (Grafana)

Each service gets a dashboard with:

  1. RED: rate (req/s), error rate (%), p50/p95/p99 latency.
  2. Resource saturation: CPU, memory, GC pauses, DB connection pool.
  3. Domain: payment outcomes per minute by status; Kafka lag (consumer side).
  4. Dependencies: bank-adapter circuit breaker state; fraud-check decision distribution.

Cross-service "platform" dashboard:

  • Kafka topic lag (per consumer group).
  • ALB request count and 5xx by target group.
  • RDS active connections, CPU, replica lag.
  • Top 10 slowest endpoints across all services.

6. Alerts

Alert rules should be:

  • Symptom-based, not cause-based ("error rate > 1%" not "DB pool full").
  • Tied to an SLO where possible.
  • Linked to a runbook in the alert annotation.

Example (Prometheus / Grafana):

- alert: PaymentService5xxRate
  expr: |
    sum(rate(http_server_requests_seconds_count{service_name="payment-service",status=~"5.."}[5m]))
      /
    sum(rate(http_server_requests_seconds_count{service_name="payment-service"}[5m]))
    > 0.01
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "payment-service 5xx > 1%"
    runbook: "https://runbooks/paymentgw/payment-service-5xx"

- alert: BankAdapterCircuitOpen
  expr: resilience4j_circuitbreaker_state{name="bankAdapter",state="open"} == 1
  for: 2m
  labels: { severity: ticket }

Alert routing (Alertmanager / PagerDuty):

  • severity=page → on-call PagerDuty.
  • severity=ticket → Jira queue.

7. SLOs

For each user-facing service, define an SLO and an error budget. Example:

Service SLI SLO target Window
api-gateway 2xx/3xx rate on /api/payments ≥ 99.9 % 30 d
payment-service p95 latency POST /api/payments ≤ 800 ms 30 d
payment-query-service p95 latency GET /api/payments/{id} ≤ 200 ms 30 d
payment events delivery time from authorise → query visible p99 ≤ 1 s 30 d

Error budget burn-rate alerts (multi-window, multi-burn):

  • 14.4× burn over 1 h → page (means 2 % of monthly budget burned in 1 h).
  • 6× burn over 6 h → ticket.
  • 3× burn over 24 h → ticket.

See Google's SRE Workbook, ch. 5 for the math.

8. The "golden signals" cheatsheet

For every service, in a meeting, anyone should be able to answer in 30 seconds:

  1. Latency — p95 of the main endpoint right now?
  2. Traffic — req/s right now? trend vs same time yesterday?
  3. Errors — % 5xx right now? top error message?
  4. Saturation — CPU, memory, DB connection pool, Kafka consumer lag?

If your dashboards make those four things hard to find, the dashboards need work.