Analyze Database Errors and Incidents

1. Classify the error signal

Separate application exceptions, database errors, connection/pool failures, AWS resource events, and monitoring noise. Record the error code/severity, database and application, first and last event time, rate, request/process identifier, and user impact. Do not assume every message labelled ERROR means a database outage.

fields @timestamp, @logStream, @message
| filter @message like /ERROR|FATAL|PANIC|connection refused|timeout|deadlock/
| sort @timestamp asc
| limit 200

2. Find frequency and first occurrence

Count errors in short intervals to find the onset and whether a burst persists. Grouping fields depend on the actual message structure.

filter @message like /ERROR|FATAL|PANIC/
| stats count(*) as event_count by bin(1m), @logStream
| sort bin(1m) asc

Compare the graph with application error rate, request volume, database connections, CPU, storage latency, and deployment events. A rise in errors during a traffic spike may have a different cause than errors after a failover or configuration change.

3. Diagnose common database signals

• Authentication/permission errors: verify role, secret rotation, endpoint, and recent access changes; never paste credentials into logs.
• Connection timeouts: separate network/connect timeout from pool wait and SQL execution time; inspect connections and saturation.
• Deadlock: collect the database deadlock detail and involved statements; identify the application transaction paths and consistent lock ordering.
• Lock timeout/lock wait: inspect the blocking chain and transaction age using authorized database activity views.
• Disk or resource pressure: correlate database and storage metrics, log errors, write rate, and retention/backup growth.

4. Keep an auditable incident trail

1. Open or update the incident record with impact, evidence links, and timezone.
2. State verified observations separately from hypotheses.
3. Assign a DBA, application owner, and incident lead as needed.
4. Make changes only through approved, reversible steps; capture before/after evidence.
5. Confirm recovery from the application perspective and track follow-up work.