Postmortems

Capture learnings from incidents and prevent recurrence.

Overview

A postmortem is a written record of an incident that answers: What happened? Why did it happen? What are we doing to prevent it from happening again?

Runframe makes postmortems easy by pre-filling incident data and tracking follow-up items.

Why write postmortems?

When to write postmortems

Creating a postmortem

Automatic creation from incidents

When you resolve an incident, Runframe prompts you to start a postmortem:

/inc resolve

You’ll be asked:

1. Provide a resolution summary
2. Whether to start a postmortem
3. Options: Yes, Later, or Skip

If you choose Yes, Runframe creates a PM-### document with:

• Incident title and number
• Severity and customer impact flag
• Timeline of status changes and key events
• Assigned responders
• Resolution summary
• Pre-populated sections to complete

Manual creation

Create a postmortem from the dashboard:

1. Navigate to Incidents in the sidebar
2. Click on the resolved incident
3. Click Start Postmortem in the top right
4. Runframe creates the PM-### document

Creating standalone postmortems

For incidents that occurred outside Runframe:

1. Navigate to Postmortems in the sidebar
2. Click New Postmortem
3. Fill in the incident details manually

Postmortem template

Every postmortem follows a consistent structure for readability and searchability.

Header

Sections

Summary

A 2 to 3 sentence overview. Write this last—it should synthesize the full postmortem.

Impact

What happened and who was affected:

• Which services were down or degraded?
• How many users were affected?
• What functionality was broken?
• Was data lost or corrupted?

Timeline

Chronological sequence of events. Use specific timestamps:

Root cause

The technical explanation of what went wrong. Be specific:

• What code or configuration change caused this?
• What was the cascade of failures?
• What monitoring was missing?
• What made this hard to detect or fix?

Resolution and recovery

How you fixed it:

• What was the immediate fix?
• What was the permanent fix?
• How long did each step take?
• What was the rollback plan if the fix didn’t work?

Follow-up actions

Specific, actionable items to prevent recurrence. Each action should have:

Lessons learned

What went well and what didn’t:

What went well:

• On-call responded within SLA
• Hotfix was deployed quickly
• Communication to stakeholders was clear

What could be improved:

• Monitoring should have detected connection pool growth earlier
• Runbook for database issues didn’t cover this scenario
• Escalation to database specialist was delayed

Writing a great postmortem

Do’s

• Be specific – Exact times, specific services, precise error messages
• Be concise – Most postmortems should be under 1,000 words
• Include data – Metrics, screenshots, graphs from monitoring tools
• Assign actions – Every follow-up should have an owner and due date
• Share learnings – Distribute to the wider team, not just participants

Don’ts

• Don’t assign blame – Systems fail, not people
• Don’t be vague – “We’ll be more careful next time” is not a follow-up action
• Don’t bury the lede – Put the most important information in the summary
• Don’t forget the customer – Even internal incidents affect someone

Postmortem review process

After drafting:

1. Technical review – Have a senior engineer verify root cause analysis
2. Comms review – Ensure customer impact is accurate if sharing externally
3. Action item review – Verify all actions are specific and assigned

Managing follow-up actions

Runframe tracks follow-up actions from postmortems to ensure completion.

Creating actions

From the postmortem editor, add follow-up actions:

• Description – What needs to be done
• Owner – Who is responsible (assign to a user)
• Due date – When it should be completed
• Priority – P0 through P4

Tracking actions

View all open actions from:

• Postmortems → Action Items in the dashboard
• Individual postmortem pages
• Slack notifications for overdue items

Closing actions

When an action is complete:

1. Navigate to the postmortem
2. Mark the action as Complete
3. Add a brief note explaining what was done
4. Runframe records the completion timestamp

Overdue actions

Runframe notifies:

• The action owner when due date approaches
• The postmortem author when actions are overdue
• Team managers for critical overdue actions

Postmortem culture

A healthy postmortem culture accelerates learning and builds trust.

Blameless postmortems

Blameless doesn’t mean no accountability—it means:

• Accountability to systems – Did our processes, tooling, or training fail?
• Psychological safety – People feel safe sharing mistakes without fear of retribution
• Focus on improvement – Energy goes into preventing recurrence, not assigning fault

When something goes wrong, ask:

• “What system failed?” not “Who messed up?”
• “What process allowed this?” not “Why didn’t you catch this?”
• “How do we prevent this?” not “Who’s responsible?”

Making time for postmortems

Postmortems take time away from feature work, but pay dividends in reduced incidents:

• Schedule postmortem time – Block 1 to 2 hours within 48 hours of resolution
• Write while fresh – Details fade quickly; write soon after incidents
• Collaborative drafting – Two heads are better than one; write together

Sharing postmortems

Decide how broadly to share based on severity and customer impact:

Postmortem analytics

Track postmortem metrics to improve your incident response:

View metrics in the Analytics section of the dashboard.

Examples

Example postmortem: Database connection pool exhaustion

# PM-001: Database connection pool exhaustion
 
**Incident:** INC-042
**Severity:** High
**Date:** January 15, 2026
**Duration:** 58 minutes
**Customer impact:** Yes – checkout flows failing
 
## Summary
 
A database connection pool exhaustion caused 500 errors on checkout flows for 58 minutes. Root cause was a recent deployment that increased concurrent query capacity without adjusting connection pool size. Fixed by increasing pool size and adding monitoring for pool usage.
 
## Impact
 
- Checkout flows failed with 500 errors
- Approximately 1,200 users affected
- Estimated revenue impact: $15,000
- No data loss or corruption
 
## Timeline
 
| Time | Event |
|------|-------|
| 2:15 PM PT | Datadog alert: 500 error rate spike on checkout endpoint |
| 2:17 PM PT | On-call engineer paged via Slack DM |
| 2:22 PM PT | Incident acknowledged, investigating logs |
| 2:35 PM PT | Root cause identified: database connection pool exhaustion |
| 2:40 PM PT | Hotfix deployed: connection pool size increased from 10 to 50 |
| 2:45 PM PT | Error rates returning to normal |
| 3:13 PM PT | Incident resolved |
 
## Root cause
 
The deployment of v2.3.0 increased the number of concurrent worker processes from 5 to 20 to handle increased traffic. Each worker process maintains its own database connection pool of 10 connections, resulting in 200 total connections. The database connection limit was configured at 150 connections, causing 50 connections to be rejected.
 
The deployment process did not validate that connection pool sizing matched the new worker process count. Additionally, no monitoring existed for connection pool usage.
 
## Resolution and recovery
 
**Immediate fix:** Increased database connection limit from 150 to 500 and reduced worker process connection pool size from 10 to 5, allowing 100 total connections with headroom.
 
**Permanent fix:**
- Connection pool sizing now calculated automatically based on worker count
- Added monitoring alert for connection pool usage above 70%
- Added pre-deployment checklist item to validate connection limits
 
**Rollback plan:** If fix didn't work, would have reverted to v2.2.0 with 5 worker processes.
 
## Follow-up actions
 
| Action | Owner | Due date | Status |
|--------|-------|----------|--------|
| Add Datadog monitoring for connection pool usage | @alice | 2026-01-17 | Complete |
| Update deployment runbook with connection pool sizing | @bob | 2026-01-18 | Complete |
- Add connection pool check to pre-deployment checklist | @carol | 2026-01-20 | Open |
| Load test database with 2x traffic before next worker scale | @dave | 2026-01-25 | Open |
 
## Lessons learned
 
**What went well:**
- On-call acknowledged within 5 minutes
- Root cause identified quickly due to clear error logs
- Hotfix deployed and verified within 30 minutes
 
**What could be improved:**
- Monitoring should have detected connection pool growth before exhaustion
- Deployment process should validate infrastructure changes
- Runbook didn't cover database connection issues

Need more?

• Incidents – Incident lifecycle and severity
• Slash Commands – Complete /inc command reference
• On-Call – Scheduling and rotations
• Web Dashboard – Full postmortem management UI