High Availability & Disaster Recovery Planner

You are a site reliability architect. Design a high availability and disaster recovery strategy for my system.

System: [SYSTEM_DESCRIPTION]
Current availability: [CURRENT_AVAILABILITY]
Target availability: [TARGET_SLA] (e.g., 99.9%, 99.99%)
Current infrastructure: [INFRASTRUCTURE]
Data criticality: [DATA_CRITICALITY]
Regulatory requirements: [COMPLIANCE]
Budget for HA/DR: [BUDGET]

Design the HA/DR strategy:

**1. Availability Math**
- Target SLA → allowed downtime per year/month/day
  - 99.9% = 8.76 hours/year
  - 99.95% = 4.38 hours/year
  - 99.99% = 52.6 minutes/year
- Current failure modes and their individual availability
- Composite availability calculation (serial vs. parallel components)

**2. RTO/RPO Definition**
- **RPO** (Recovery Point Objective): Maximum acceptable data loss
- **RTO** (Recovery Time Objective): Maximum acceptable downtime
- For each tier of services (not everything needs the same RPO/RTO)

**3. High Availability Design**
- Redundancy at every layer (load balancers, application, database, storage)
- Multi-AZ deployment strategy
- Database HA: Primary-replica, multi-master, or distributed
- Session management (stateless services, externalized state)
- Health checks and auto-healing
- Graceful degradation plan (what to shed under extreme load)

**4. Disaster Recovery Plan**
- DR strategy: Active-Active vs. Active-Passive vs. Pilot Light vs. Backup & Restore
- Multi-region architecture (if required)
- Data replication strategy (sync vs. async, RPO implications)
- Failover automation (DNS, load balancer, database)
- Failback procedure

**5. Failure Scenarios & Runbooks**
For each scenario (single instance, AZ failure, region failure, data corruption):
- Detection: How do we know it happened?
- Response: Step-by-step recovery procedure
- Communication: Who to notify and status page updates
- Estimated recovery time

**6. Chaos Engineering Plan**
- Game day exercises to practice failures
- Automated chaos testing (Chaos Monkey, Gremlin, Litmus)
- Failure injection starting points (start small)

**7. Monitoring & Alerting**
- SLI/SLO definitions for each critical path
- Error budget tracking
- Escalation procedures
- On-call rotation design

**8. Cost Analysis**
| HA Level | Architecture | Monthly Cost | Downtime/Year |

Recommend the right trade-off for your budget and requirements.