W43 - Lessons Learned from the AWS Incident

Last week AWS suffered a major outage lasting 15 hours, interrupting services for more than 1,000 companies worldwide and causing billions of dollars in economic loss. The lesson was stark: no matter how elegant the engineering, a single-point dependency can undo it all.

The incident occurred in AWS’s most critical region, us-east-1. The root cause was a race-condition bug in the automation that manages DynamoDB’s DNS records, which mistakenly cleared the region’s endpoint IP entries and caused DNS resolution failures. us-east-1 is not an ordinary data center — it functions as the nervous center of AWS’s global infrastructure. Many public control planes in other regions depend on it, including key services like EC2 scheduling, NLB, and Lambda, which led to cascading failures.

More ironic was that 75 minutes after the outage began, the AWS status page still showed “all systems operational” — because CloudWatch itself depended on DynamoDB, so the monitoring system went blind along with everything else.

Recovery was not easy. DNS records could be repaired quickly, but DynamoDB was a core control node: the EC2 scheduler had to rebuild leases for tens of thousands of servers, the network manager had to clear congestion and redistribute delayed state, and load balancers repeatedly removed instances after misjudging their health. Ultimately AWS had to disable automatic health checks and failover; manual intervention was what finally stopped the ongoing cascade.

After watching the spectacle, I reviewed our own services. The most likely systemic single points of failure tend to be foundational services that receive less attention during product iterations. For example, information-entry gateways and authentication services: the logic of the gateway may be simple, but its attack surface is huge and it carries 80% of the traffic. Engineering these services should follow the principle of minimal dependencies and maximal entropy — keep them extremely clear and simple.

Components like the information-entry gateway can tempt ambitious engineers because they present interesting engineering challenges: supporting multiple versions across tech stacks, long release cycles tied to host pages. Looking back at what we did this year on the gateway, I think it’s worth replaying whether we chose the right direction. To enable dynamic updates, shorten release cycles, and increase release flexibility, we did a lot of engineering work, introduced more dependencies, and designed a more complex architecture, which also reduced rollback effectiveness. Following that path, once the gateway was made dynamic the next step would be to run dynamic components in containers and consolidate implementations for high dynamism and reuse. But after the iteration peak, those features start to cast the shadow of technical debt.

I reflect that this year my focus remained on training engineering skills and creating opportunities for the team to practice flashy techniques, rather than sufficiently shifting left to improve the quality of technical decisions. Decision-making lacked clear frameworks, evidence, and logical reasoning. The one thing I did get right, I believe, was introducing democratic, collective decision-making.

Every engineer’s growth includes a cognitive phase that dislikes redundancy, abstracts everything, and equates complexity with technical merit. We must be willing to abandon these mid-level habits and pursue clarity and simplicity arrived at through trade-offs. Like the Unix design philosophy, learn to appreciate higher-quality work.