Build an AWS Database Log Investigation Runbook

1. Incident intake

1. Capture incident ID, symptom, user impact, service owner, and first known event time.
2. Record AWS account, Region, environment, RDS/Aurora resource, engine/version, and application service.
3. Confirm timezone and select a narrow CloudWatch absolute time range with lead-up and recovery context.
4. Check service events, recent deployments, configuration changes, and alarm transitions.

2. Search and correlate

Open only the approved database and application log groups and start with a bounded query:

fields @timestamp, @ingestionTime, @logStream, @message
| filter @message like /ERROR|FATAL|timeout|deadlock|duration:/
| sort @timestamp asc
| limit 200

1. Identify the earliest relevant event and distinguish it from delayed ingestion.
2. Search the application trace/request ID and compare connection-pool wait with SQL duration.
3. For completed slow statements, search duration logs if enabled and parse the actual message format.
4. For a query that is running now, inspect PostgreSQL pg_stat_activity through approved database access.
5. Compare CPU, connections, storage latency, replication, and application error/latency metrics over the same time window.

3. Evidence and response

Record timestamps with timezone, log groups/streams, query text only when authorized and redacted, trace ID, relevant metrics, wait/transaction age, hypotheses, owner, and next action. Do not share sensitive log exports in public tickets. Use approved change control for cancellation, configuration changes, failover, or other production actions. Never kill a database process solely because it is old.

4. Verify resolution and prevent recurrence

1. Confirm error rate and user-facing latency recover.
2. Verify database connections, active waits, and service health return to the expected range.
3. Compare behavior with the same pre-incident metrics and logs.
4. Document root cause only when evidence supports it; mark unverified theories as hypotheses.
5. Create follow-up work for missing instrumentation, query optimization, alarm tuning, retention, or runbook gaps.
6. Close the incident with a timeline, impact, evidence, recovery check, and accountable owners.

Course completion checklist

• Can you find the database resource's exported log group and verify that events arrive?
• Can you search a bounded time range and explain event versus ingestion time?
• Can you distinguish a logged completed slow query from a currently active query?
• Can developers and DBAs correlate a request using a safe trace ID and timing breakdown?
• Does each alarm have a tested notification route, owner, and response procedure?