{“lastseen”:”2025-10-22T00:05:09″,”description”:”_What can we learn from the recent AWS outage, and how can we apply those lessons to our own infrastructure?_\\n\\n## **What Happened?**\\n\\nOn **October 20, 2025** , AWS experienced a major disruption that rippled across the internet (and social media), affecting widely used services such as Zoom, Microsoft Teams, Slack, and Atlassian. The issue originated not in a single data center or customer workload, but in the AWS control plane, the management layer that coordinates how resources like EC2 instances, DynamoDB tables, and IAM roles operate.\\n\\nThe initial trigger appears to be a DNS failure around the DynamoDB API endpoint in the US-EAST-1 region, compounded by a malfunction in the subsystem that monitors network load balancers. Because those health-monitoring services also run in US-EAST-1, AWS throttled new EC2 instance launches while restoring the subsystem.\\n\\nThough AWS markets its regions as isolated clusters of data centers with independent power and cooling, this incident showed that core control-plane functions remain centralized, creating hidden dependencies that can cascade globally.\\n\\n## **Root Cause: A Single-Region Control Plane**\\n\\nAnalysts quickly identified that US-EAST-1 hosts AWS’s common control plane that supports a host of global services. Workloads running in Europe or Asia actually rely on API calls that route back through or to US-EAST-1, so the failure there had global consequences.\\n\\nWhen the region’s DNS and health-check subsystems failed, those control-plane calls stalled worldwide. The end result was a global slowdown in EC2 launches, configuration updates, and authentication, despite the other regions being technically \\”healthy.\\”\\n\\nAWS’s own design guidance encourages customers to spread workloads across availability zones for resiliency, but these customer-facing resiliency mechanisms ultimately depend on the same centralized control plane. In other words, data-plane isolation worked as designed, but control-plane isolation did not.\\n\\nThis pattern has surfaced before, not just at AWS. Cloudflare, Microsoft, and Google have all suffered outages triggered by control-plane or configuration failures that propagated globally. The lesson here is that in modern distributed systems, control-plane fragility can become a single point of failure.\\n\\n[Image: Timeline of Outages]\\n\\n## **Lessons Learned for Infrastructure Architects**\\n\\nUnderstanding and analyzing major outages like AWS’s is essential for infrastructure engineers. Each incident reveals gaps between design assumptions and real-world complexity, exposing weak points that might otherwise go unnoticed, and ultimately impact service availability. By studying what failed, why it failed, and how recovery proceeds, architects can refine their infrastructure and systems to be more fault-tolerant and resilient. This mindset of continuous learning ensures that when the next disruption happens, the impact on users and business operations is minimized. So what are the key lessons here? \\n\\n**1\\\\. Design for true multi-region, active-active operation.** A standby region isn’t enough if control traffic can’t reach it. Run in an active-active configuration so that the loss of one resource or region doesn’t disable the service. \\n\\n**2\\\\. Avoid single-region control planes:** It seems obvious to say now, but configuration and metadata services should be either fully local or replicated globally. Any system that depends on a single region’s DNS, load balancers, or other systems for coordination introduces a global risk.\\n\\n**3\\\\. Separate the control plane from the data plane:** The AWS incident began in the control plane but quickly cascaded to the data plane. Architect systems so runtime traffic can continue independently from configuration or provisioning systems.\\n\\n**4\\\\. Distribute DNS and caching layers:** Multi-provider DNS with long TTLs can reduce the impact of control-plane-related resolution failures. Cached or regional read replicas keep applications partially functional when the origin is unreachable.\\n\\n**5\\\\. Implement circuit breakers and bulkhead isolation: ** Systems should fail fast and gracefully. Circuit breakers can reroute or degrade functionality instead of hammering a failing endpoint. Bulkhead isolation limits the spread of failures between components.\\n\\n**6\\\\. Continuously test failure scenarios:** Regular \\”chaos\\” testing validates that redundancy, DNS failover, and RTO\/RPO objectives work in practice. AWS’s own Resilience Hub encourages these tests, but the lesson applies to any cloud or hybrid deployment. Check out Chaos Monkey, introduced by Netflix in 2011, as an example. \\n\\n**7\\\\. Plan for multi-cloud or hybrid resiliency:** Multi-availability-zone redundancy doesn’t protect against control-plane issues. Deploying across multiple cloud providers or keeping a minimal on-prem footprint prevents total dependence on one provider’s management systems.\\n\\n**8\\\\. Decouple capacity from failover logic** : AWS mitigated the outage by throttling new instance launches, buying time, not resilience. Reserve compute in secondary regions ahead of time, but ensure failover logic works autonomously from the control plane.\\n\\n## **Security Edge: Lessons Learned Applied**\\n\\nIn designing the architecture for Security Edge, we wanted to address many of these specific challenges. Today, Wallarm’s Security Edge embodies many of these principles by design. It was intentionally built to avoid the single-provider fragility exposed by the AWS outage. Security Edge includes: \\n\\n * **Active-active, multi-cloud architecture:** Security Edge runs enforcement nodes across AWS, Azure and other providers in an active-active configuration. Traffic can be instantly rerouted if one provider experiences degradation. This multi-cloud distribution eliminates the single-region or single-provider risk that impacted AWS. \\n * **Decoupled from customer infrastructure:** Security Edge operates as a managed, cloud-native security layer, not a component embedded in customer environments. Its filtering and enforcement nodes are positioned close to the API edge, but isolated from the customer’s own data plane. In other words, even if a customer’s cloud or provider fails, the Wallarm protection layer remains operational.\\n * **Always-on availability with automated failover:** Wallarm’s architecture provides automatic global failover and high availability independent of any one cloud provider or CDN. Customers benefit from security continuity even when the underlying infrastructure is disrupted.\\n * **Low-latency, edge-proximate enforcement:** By placing filtering nodes near the customer’s API endpoints, Security Edge maintains low latency while providing deep inspection and telemetry. The distributed footprint ensures that even during a provider outage, legitimate traffic continues to flow efficiently.\\n * **API-native and protocol-aware:** The platform supports REST, GraphQL, gRPC, Web Sockets and legacy protocols, ensuring that resilience doesn’t come at the cost of compatibility. \\n * **Secure by design.** Mutual TLS (mTLS) provides secure, authenticated connections across all nodes, essential for regulated industries.\\n * **Simplified deployment and management:** Because Security Edge is provisioned via a DNS change, teams don’t have to manage control-plane dependencies or infrastructure updates. The security layer remains operational even if customer environments or specific regions experience downtime.\\n\\n\\n\\n## **The Broader Pattern**\\n\\nAWS may be in the spotlight now, but looking across the industry, nearly every major cloud or CDN provider, AWS, Cloudflare, Microsoft, Google, has experienced control-plane-related outages in the past five years. These are rarely caused by attacks; more often, they stem from routine operational changes, misconfigurations, or centralized service dependencies.\\n\\nThe October 2025 AWS outage simply demonstrates that no cloud provider is immune. The best defense is architectural: distribute risk, decouple dependencies, and design for graceful degradation.\\n\\nWe\u2019re proud that Wallarm’s Security Edge demonstrates how these lessons can be applied proactively. By shifting API protection into a multi-cloud, active-active edge fabric, organizations gain resilience not just against attacks, but against the infrastructure failures that even the largest providers occasionally suffer.\\n\\n## **What\u2019s Next? **\\n\\n\\n\\nJoin Wallarm\u2019s Field CTO, Tim Ebbers this Thursday, October 23, for an in depth conversation on this AWS outage and architecting for resiliency.\\n\\nThe post AWS Outage: Lessons Learned appeared first on Wallarm.”,”published”:”2025-10-21T23:14:40″,”modified”:”2025-10-21T23:14:40″,”type”:”wallarmlab”,”title”:”AWS Outage: Lessons Learned”,”source”:””,”references”:””,”id”:”WALLARMLAB:F5396A571F99001F34FAAB7C22547DCA”,”bulletinFamily”:”blog”,”cwe”:null,”cvelist”:[],”sourceData”:””,”sourceHref”:””,”cvss”:{“score”:0,”severity”:”NONE”,”vector”:”NONE”,”version”:”NONE”},”cvss2″:{},”cvss3″:{“version”:””,”vectorString”:””,”baseScore”:0,”baseSeverity”:””,”attackVector”:””,”attackComplexity”:””,”privilegesRequired”:””,”userInteraction”:””,”scope”:””,”confidentialityImpact”:””,”integrityImpact”:””,”availabilityImpact”:””,”cvssV3″:{“version”:””,”vectorString”:””,”baseScore”:0,”baseSeverity”:””,”attackVector”:””,”attackComplexity”:””,”privilegesRequired”:””,”userInteraction”:””,”scope”:””,”confidentialityImpact”:””,”integrityImpact”:””,”availabilityImpact”:””}},”href”:”https:\/\/lab.wallarm.com\/aws-outage-lessons-learned\/”,”category_name”:”News”,”post_link”:””,”product”:””,”version”:””,”vendor”:””,”ai_description”:””,”ai_severity”:””,”ai_vendor”:””,”ai_product”:””,”ai_version”:””,”ai_score”:0}<\/p>\n","protected":false},"excerpt":{"rendered":"
{“lastseen”:”2025-10-22T00:05:09″,”description”:”_What can we learn from the recent AWS outage, and how can we apply those lessons to our own infrastructure?_\\n\\n## **What Happened?**\\n\\nOn **October 20, 2025**…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[6,8,12,13,33,7,11,5,105],"class_list":["post-22649","post","type-post","status-publish","format-standard","hentry","category-category_news","tag-cve","tag-cvss","tag-exploit","tag-news","tag-none","tag-security","tag-tapic","tag-vulnerability","tag-wallarmlab"],"yoast_head":"\n