Why Are Database Queries Timing Out During Peak Traffic?
An e-commerce platform suffers severe database latency during evening sales, risking checkout failures.
Context
You are a database specialist advising the infrastructure lead at a growing e-commerce app. Over the past two weeks, daily evening flash sales have triggered severe database query latency, leading to connection pool exhaustion and transaction timeouts at checkout. The head of engineering believes the primary relational database has outgrown its hardware capacity and wants to approve an immediate vertical upgrade before tonight's promotional campaign starts in four hours. You must analyze performance telemetry, diagnose the root cause, and determine whether to scale hardware or optimize system configuration.
Your Task
Provide your operational brief for the head of engineering covering:
- Recommendation — one concrete decision on addressing the database latency bottleneck.
- Impact — projected latency reduction, system resource utilization, downtime requirements, and cost savings.
- Implementation Plan — sequence of actions, operational owners, and execution timeline ahead of the peak traffic window.
Available Data
Database Performance During Flash Sales (8:00 PM – 9:00 PM)
| Metric | Baseline (Off-Peak) | Peak Flash Sale (Current) | Infrastructure Target |
|---|---|---|---|
| Active Connections | 45 | 495 (Max 500) | < 250 |
| CPU Utilization | 18% | 96% | < 65% |
| Disk I/O Wait (IOPS) | 420 | 11,800 (Max 12,000) | < 4,000 |
| Average Query Latency | 12 ms | 3,850 ms | < 50 ms |
| Failed Checkout Transactions | 0.01% | 8.4% | < 0.1% |
Checkout error rates have risen steadily over the last seven days, directly coinciding with the launch of a real-time order tracking feature in the mobile app. The infrastructure team has already prepared a purchase order to quadruple server RAM and CPU, which will require two hours of maintenance downtime.