Building Resilient AI-powered Software, And Planning For When Things Go Wrong
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Age 18–24?Offer from Amazon

Prime made for students and young adults

  • Fast, free delivery for dorm and study essentials
  • Prime Video and Amazon Music included
  • Member-only deals
Try Prime for Young Adults Free trial for eligible 18–24 year olds
As an affiliate, we earn on qualifying purchases.

The Pragmatic Engineer published excerpts from Sam Newman’s new book, Building Resilient Distributed Systems, covering disaster recovery and the reliability challenges of AI-powered software. The material discusses recovery targets, failover options, infrastructure constraints, model changes and vendor uncertainty; it does not report a specific outage or measure the effect of AI coding tools on incident rates.

The Pragmatic Engineer has published excerpts from Sam Newman’s newly released book, Building Resilient Distributed Systems, laying out disaster-recovery planning and reliability concerns for software that uses AI. The report focuses on practical choices for handling major infrastructure failures and on preparing for shifting model and vendor dependencies; it does not document a particular outage or establish that AI-generated code has increased incident rates.

In the excerpt from Chapter 10, Newman describes disaster recovery (DR) as the procedures used when a failure affects a large part of a system, such as a site-wide cloud outage, a power disruption, fire or flooding. He says organizations face trade-offs: reducing acceptable downtime or data loss generally raises cost and complexity. The report uses “site” as a broad term for the location of computing infrastructure, whether that is a cloud region or a physical data center.

Newman recommends setting two measurable targets before choosing a recovery approach. The recovery point objective (RPO) sets the maximum acceptable period of potential data loss, based on how recently data was backed up or synchronized. The recovery time objective (RTO) sets how long a system can remain unavailable, including the time to restore data, make computing resources available and redirect traffic to a replacement site.

The report also identifies topics covered in the book’s excerpts: backup-and-restore, pilot-light, warm-failover and multisite active-active recovery patterns; infrastructure availability amid reported GPU and CPU constraints; and resilience concerns specific to AI-powered software. These include dependence on model providers, changes to models over time, observability, using multiple models or vendors, making costs visible and reviewing vendors regularly.

At a glance
reportWhen: Published in 2026; the report describes…
The developmentThe Pragmatic Engineer has published excerpts from Sam Newman’s 2026 book on disaster recovery and resilience, including guidance for systems that use AI.

Recovery Targets Shape System Resilience

The guidance matters because a recovery plan involves choices about how much downtime and data loss an organization can tolerate, not simply whether it has a backup. RPO and RTO make those tolerances explicit and give engineering teams a basis for comparing recovery patterns against their service needs and budgets.

AI adds another set of dependencies to plan for. A product may rely on external model providers and changing model behavior, while its infrastructure needs may compete for limited compute resources. The report’s discussion points to observability and contingency planning as ways to detect failures and prepare alternatives. These are recommendations from the book excerpt, not evidence that any particular AI service has failed or that a multi-provider design is always the right choice.

Amazon

disaster recovery backup and restore software

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

From Site Outages to Model Dependencies

The Pragmatic Engineer report says its podcast guest was Sam Newman, whose book took three years to complete rather than the two originally planned. Newman told the publication he revisited and revised the material to address AI’s relevance to resilient-system design. The publication then shared selected material from two chapters, with permission from the publisher.

The source frames the book against a concern that software teams are seeing more outages and production bugs, and says increased code deployments, including code written by AI agents, are part of that picture. That is the publication’s framing, not a quantified finding in the supplied material: it provides no incident data or comparison baseline showing that AI coding tools caused an increase. The excerpt itself concentrates on planning for failures and managing dependencies.

“The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.”

— Sam Newman, in an excerpt from Building Resilient Distributed Systems

Amazon

high availability cloud failover solutions

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

AI Incident Claims Lack Metrics

The supplied report excerpt does not provide outage counts, incident-rate data or a comparison period to test its introductory concern about more deployments and production bugs associated with AI coding agents. That concern should not be read as a demonstrated causal finding. The available material also does not specify the right RPO, RTO or recovery pattern for a particular organization; those depend on its system and acceptable risk.

The report names model-provider resilience, changing models, compute availability and vendor uncertainty as issues, but gives no named provider failures, capacity figures or measured costs in the supplied text. It is also unclear from the excerpt how the book recommends testing each recovery pattern in practice or what safeguards are needed when moving between model providers.

Amazon

AI system resilience monitoring tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Teams Must Set Their Own Targets

The immediate next step for readers applying the excerpt is to define acceptable data loss and service downtime for each system, then compare recovery options against those targets, costs and infrastructure constraints. The report does not announce a deadline, industry standard or implementation plan; decisions remain specific to each engineering team and its service.

The Pragmatic Engineer says the selected passages cover less than a full chapter of the book’s 16 chapters and recommends the full book for engineers responsible for reliable systems. The book is published by O’Reilly Media. Further detail on the broader recommendations, including the chapter on vendor choices, would require the remaining book material or subsequent reporting.

Amazon

multi-region active-active cloud setup

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Key Questions

What is the news?

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, covering disaster recovery and resilience issues for AI-powered software.

What do RPO and RTO mean?

RPO is the maximum acceptable period of potential data loss, measured by the time since data was last backed up or synchronized. RTO is the acceptable time a system can remain unavailable after an incident.

Does the report prove AI coding tools are causing more outages?

No. The publication raises that concern, but the supplied material includes no incident statistics, comparison baseline or causal analysis establishing that AI coding tools have increased outages.

It points to model-provider resilience, changes to models, observability, compute availability and vendor uncertainty. The report describes these as planning considerations rather than documenting a specific provider failure.

Which disaster-recovery approaches are mentioned?

The report lists backup-and-restore, pilot-light failover, warm failover and multisite active-active patterns. It does not prescribe one approach for every system; the choice depends on recovery targets, cost and complexity.

Source: rss

FALL

Fall Picks

As an affiliate, we earn on qualifying purchases.

You May Also Like

AirPods 5

Speculation about Apple’s upcoming AirPods 5 has surged, with reports hinting at new features and design changes. Confirmed details remain scarce.

Tornado Watch

A tornado watch has been issued for several states in the US amid severe weather forecasts. Details on affected areas and next steps inside.

United Community Banks, Inc. Announces Quarterly Cash Dividend On Common Stock

United Community Banks, Inc. announces a quarterly cash dividend on its common stock, reflecting its ongoing financial stability and shareholder return strategy.

Will Bertha Become A Category 2 Hurricane?

Meteorologists are monitoring Hurricane Bertha as it intensifies, with some markets betting on it becoming a Category 2 hurricane. Development ongoing.