1. Establish a reliable incident window
- Record when users first reported the symptom and the time zone.
- Find the earliest matching error or latency change; note both event time and ingestion time.
- Include a lead-up and recovery buffer so slow buildup and delayed delivery are visible.
- Check clock synchronization and timestamp format for application hosts.
- Record the exact CloudWatch absolute time range and account/Region in the incident.
In Logs Insights, @timestamp is the event timestamp and @ingestionTime indicates when CloudWatch received it. A gap between them can indicate delayed delivery; it does not move the time the original event occurred.
2. Build an event timeline
fields @timestamp, @ingestionTime, @logStream, @message
| filter @message like /ERROR|FATAL|timeout|deadlock/
| sort @timestamp asc
| limit 200Export or record the small set of relevant events with timestamps, source, request or process identifiers, and sanitized message text. Add deploys, configuration changes, failovers, traffic spikes, and alarm transitions as separate timeline entries.
3. Separate symptom onset from cause
The first log error may be a consequence, not the initiating event. Compare database connection counts, CPU, storage, read/write latency, queue/pool wait, and application request duration. Look for a new deploy, scheduled job, connection-pool change, failover, or upstream dependency event before the first failure. Confirm the time zone and whether a plotted metric uses a period/aggregation that can hide a short spike.
Example: database timeouts at 14:05 UTC
- Search application logs for the first timeout and request IDs around 14:05.
- Check whether pool-acquisition time grew before database execution time.
- Inspect RDS/PostgreSQL events, database error logs, connections, and CPU for the same interval.
- Compare recent deployment and scheduled-job timestamps.
- Extend the window earlier if connection use or latency rose before the timeout burst.
State the earliest proven symptom separately from the current cause hypothesis; keep both tied to evidence.
Progress stays in this browser.