Yet, this utopian automation pipeline often fractures when applied to the beating heart of enterprise architecture: foundational platforms.
According to Ian Nowland, co-founder of Junction Labs and veteran engineering leader with a decade at Amazon and eight years spearheading AWS EC2 development, standard CI/CD frameworks are fundamentally misaligned with the realities of core infrastructure. Speaking on the unique engineering paradigms required to manage stateful, mission-critical systems, Nowland argues that the conventional push for rapid, automated deployments can spell disaster for foundational platforms.
Executive Overview
The engineering world frequently treats software development as a monolithic discipline, assuming that practices optimized for fast-moving frontend teams or stateless microservices will universally succeed. However, this one-size-fits-all approach overlooks a critical boundary in systems design: the line between non-foundational platforms (where a bug might frustrate a developer) and foundational platforms (where an outage brings trading desks to a halt, freezes data ingestion, or drops global availability).
Drawing from his extensive tenure managing massive-scale infrastructure at Amazon EC2, Two Sigma, and Datadog, Nowland outlines a stark reality. Foundational platforms are stateful, tightly coupled with physical hardware limitations, and deeply interconnected across a vast blast radius. For these systems, traditional CI/CD mantras—such as deploying multiple times a day or aggressively relying on feature flags—frequently introduce catastrophic instability.
Instead of chasing an unachievable ideal of single-button continuous delivery, platform engineering teams must pivot toward a rigorous discipline of safe progressive deployments. This operational philosophy trades raw velocity for deep observability, rigorous peer-reviewed change management, and comprehensive synthetic monitoring in production.
Detailed Chronology: Lessons from the Frontlines of Cloud Scale
To understand why traditional delivery models break down, it is necessary to examine the evolution of platform engineering through the lens of systems that defined modern cloud computing.
The Wild West of AWS EC2 (2010–2016)
When Nowland joined AWS in 2010, cloud architecture was still in its infancy. The engineering culture was defined by a radical "you build it, you own it" ethos, entirely devoid of dedicated testing or operations teams. In those early years, the infrastructure was notably volatile; transient failures—such as a 1-in-1,000 instance launch failure caused by undiscovered DHCP anomalies—were an accepted hazard of pioneering utility computing.
Despite this volatility, AWS scaled rapidly, transitioning from a startup playground to the backbone of Fortune 100 enterprises. To maintain stability amidst this staggering growth, EC2 engineering teams abandoned the illusion that manual human touchpoints could be entirely automated away.
Instead, they formalized a rigorous change-management lifecycle:
- Strict Change Review: Every production modification required meticulous planning, precise timing, phase-by-phase execution protocols, and pre-tested rollback scripts.
- The "Marc’s Law" Reality: Echoing distinguished engineer Marc Brooker, Nowland notes that complex distributed systems harbor failure modes that simply take time to manifest. A deployment could look pristine during its initial rollout, only to implode 24 hours later under peak global load due to subtle cascading failures, such as garbage collection pauses or thread-pool saturation.
- Geographic Rollout Phasing: Rather than adopting naive deployment sequences, EC2 learned to deploy changes first to low-sensitivity regions (such as South America) and subsequently to the heaviest-traffic zones, surfacing hidden anomalies before they impacted global availability.
Scaling and Cultural Collision at Datadog (2019–2023)
Moving to Datadog during its pre-IPO rocket ship phase, Nowland encountered a different kind of scaling challenge. The organization boasted a thriving product engineering culture optimized for speed, feature flags, and rapid customer feedback loops. However, as enterprise adoption accelerated, this velocity began to clash with the stability demands of core observability platforms.
New engineering leaders arriving from massive tech conglomerates frequently attempted to clone processes that only functioned at scales hundreds of times larger than Datadog’s footprint, leading to low-morale compliance rather than genuine operational excellence.
To bridge this cultural divide, Nowland bypassed top-down bureaucratic mandates. By instituting weekly post-mortem reviews and partnering with organizational champions among staff engineers, Datadog successfully integrated phased deployments without strangling the product teams’ agility. Furthermore, the engineering group mitigated the sprawl of complex feature flags by heavy investments in sandboxes and shadow deployments—routing real production data via Apache Kafka streams to test new logic asynchronously without risking downstream customer impact.
Supporting Context & Metrics: Why Platforms Differ
The friction between product development and platform engineering is rarely a matter of engineering competence; it is a structural divergence in system mechanics.
| Metric / Dimension | Application / Product Teams | Foundational Platform Teams |
|---|---|---|
| Statefulness | Stateless or lightly stateful; easy to isolate. | Deeply stateful; manages metadata and storage on behalf of the business. |
| Blast Radius | Isolated to specific UI features or user segments. | System-wide; a single bug can take down the entire enterprise infrastructure. |
| Failure Tolerance | Higher; minor UI bugs or instability often go unnoticed. | Extremely low; failures instantly trigger systemic outages. |
| Testing Environment | Easily mocked, staged, and tested locally. | Impossible to replicate production fidelity outside of production. |
| Deployment Cadence | Multiple times per day (Continuous Deployment). | Measured paces (weeks per deployment) with phased rollouts. |
The Myth of the Staging Environment
A recurring trap for platform teams is the persistent pursuit of a comprehensive staging environment. According to Nowland, staging environments for foundational platforms are a classic "tragedy of the commons."
When a handful of engineers utilize a staging cluster, coordination is manageable. However, as organizations scale past a dozen teams, the staging environment inevitably becomes a perpetually unstable battleground. Because platform teams are tasked with maintaining a pristine testing ground for everyone else, they can never use it to test genuinely risky, foundational architectural changes. Consequently, large-scale integration test frameworks become bloated, inherently flaky, and ultimately abandoned.
The antidote? High-coverage synthetic monitoring in production. By simulating end-to-end customer workflows (such as provisioning, utilizing, and tearing down resources via live APIs), engineering teams accept the reality of "testing in production" while maintaining a reliable, automated heartbeat that surfaces regressions immediately.
Official Statements and Industry Insights
Reflecting on the misconceptions surrounding infrastructure reliability, Nowland challenges the rigid interpretation of industry frameworks like DORA (Accelerate).
"When you write a book, you’re writing for the 80%, not the 20%. DORA metrics are incredibly important for product teams, but if a platform team reports a 15% deployment failure rate resulting in outages, they would be fired. The metrics must match the mechanical reality of the system."
Furthermore, Nowland addresses the executive aversion to the phrase "testing in production," noting that semantics often obstruct honest technical discourse:
"Your executives hate hearing ‘testing in production.’ But the reality is that complex distributed systems contain unknown unknowns. You are testing in production; the only question is whether you are doing it blindly with raw user traffic, or progressively through safe, observable canaries."
Future Outlook: The Next Era of Platform Engineering
As enterprises continue to migrate toward multi-cloud architectures, edge computing, and highly distributed microservices, the pressure on platform engineering teams will only intensify. The temptation to automate away operational friction using standard, app-tier CI/CD tooling will remain strong.
However, the industry is experiencing a necessary maturation. The future of foundational platform management lies not in pretending that infrastructure can be deployed like a simple web application, but in accepting its inherent complexity.
Organizations that succeed will be those that embrace safe progressive deployments as a first-class engineering discipline. By combining rigorous peer-reviewed change management, deep infrastructural observability, production-grade synthetic monitoring, and localized deployment automation, engineering leaders can reconcile the eternal tension between enterprise stability and digital velocity.
