972-383-9791 | contactus@t.digital

Lessons in Resilience: Building Platforms That Survive Cloud Failures

When AWS Blinked: What the US-EAST-1 Outage Revealed About Resilience as Architecture

Earlier this week, AWS experienced a failure in the US-EAST-1 region that rippled across the internet. DNS instability, control-plane failures, and service disruptions took major platforms offline for hours.

Some organizations lost full visibility into their environments. Others could not access administrative consoles at all. Recovery was uneven, and for many teams, the outage exposed architectural weaknesses they did not know they had.

For us at Thanawalla Digital, the incident was a real-world test of something we talk about often.

Resilience is not a feature. It is an architectural choice.

This week, that choice paid off.

What actually protected our client workloads.

Most companies, including large enterprises, deploy their applications directly on raw AWS primitives. That works until the AWS control plane becomes the point of failure, as it did this week. Our approach is different by design.

We run critical customer-facing workloads on abstracted, isolated platforms like Heroku Private Spaces: platforms that provide high availability, private networking, and automated failover without relying on a single AWS subsystem to stay healthy. This is not convenience. It is insulation.

Protection 1 Private DNS, not public DNS

Runtime traffic between our applications and databases flows entirely over protected private DNS. The public DNS instability that hit most of the internet never touched the transaction paths our clients depend on.

Protection 2 High availability across multiple zones

Our high-availability dynos are distributed across multiple AWS availability zones. Even when a subset of the underlying infrastructure degraded, runtime continuity held.

Did we feel impact? Yes. In one case, we temporarily lost access to an internal dashboard. That was not an accident. That dashboard sits on a simpler, non-critical stack. We design these distinctions intentionally. Customer-facing systems must survive chaos. Internal conveniences can wait.

Outcome Every production runtime stayed online. Every customer stayed supported.

Resilience is not improvised. It is engineered.

Events like this are reminders that resilience is not something you bolt on after a failure. You either architect for failure, or you are surprised by it. This is why we do not chase complexity for its own sake. We choose platforms that abstract infrastructure risks, enforce isolation, and provide built-in redundancy.

These design decisions are not theoretical. They buy real uptime when the rest of the internet blinks.

What happens next: hardening, validation, and forward motion.

As the AWS failure subsided, our team immediately moved into post-incident protocol across three tracks.

Integrity and trust boundary validation
  • Confirming no writes were dropped during the outage window
  • Verifying no transactions fell outside SLA windows
  • Validating data continuity spans the outage entirely
Behavioral and API-level review
  • Reviewing system logs for unusual request patterns
  • Identifying opportunistic probing during reduced visibility
  • Flagging any anomalies in load or timing across services
Accelerated modernization and risk reduction
  • Migrating deprecated components to supported runtimes
  • Restricting high-volume extraction endpoints
  • Tightening threat surface with stronger behavioral monitoring
  • Increasing penetration testing around failover behavior

Resilience is not static. It is iterative.

If this week was a wake-up call, you are not alone.

The AWS outage revealed a pattern we see too often: most systems are fragile not because teams are careless, but because the architecture beneath them was never designed to withstand real-world failures. Our job, and the core of our platform engineering practice, is to change that.

The questions this outage surfaces are not new. They are the same ones we explore in depth across our ongoing series on enterprise architecture: how speed, security, isolation, and polyglot design combine into systems that do not just perform, they endure. If those questions are relevant to your organization, that series is a useful starting point.

From the Thanawalla Digital content series Our six-part series on the architecture of enterprise agility explores the design principles that kept our environments stable this week: decision velocity, security as a design principle, real-time mapping, polyglot freedom, and the architecture of resilience. All installments are available on our site.

Build systems that stay online

If you would like to understand how we build systems that stay online when the underlying cloud does not, we are always happy to talk. Reach out directly or connect with us on LinkedIn.

Contact Thanawalla Digital