AI Rollback Playbook: What Happens When You Have to Revert a Model Update at 2 AM

The 2 AM Scenario (Why Most Teams Are Not Ready)

A vendor ships a model update with no warning. Within an hour, structured outputs start failing schema checks. P95 latency spikes. A customer-facing workflow returns errors. The on-call engineer is paged. They call vendor support. The answer: "Rolling back to the prior model version is not a self-service action."

This happens when a team has no rollback runbook. They fix it by guessing, then face the same problem next time. There are three ways a team finds out about model problems: a customer complaint, a monitoring alert, or an end-of-day review. Two of those are too slow. A customer complaint means customers saw the problem for minutes or hours before anyone knew. An end-of-day review means hours of SLO breach before anyone acts. Only a monitoring alert gives the team time to act before customers are hit. This article is the playbook for that action.

A rollback playbook is not a full incident response plan. It covers the rollback decision and the steps: when to revert, who decides, what must be in place first, and what to do when the vendor cannot revert. For the full AI incident response plan covering detection, escalation, and post-mortem, see the next article in this series.

Part 1: Pre-Conditions for Rollback (Before the Incident Happens)

In electrical engineering, you test a fault relay before a fault hits, not during one. If you wait until the fault arrives, you do not have a protection system. You have a hope. The same idea applies to AI rollback. Rollback is something you plan in advance. You cannot build it during an incident. Check each of the five pre-conditions below before any model update window opens.

The 12-Control AI Incident Readiness Audit covers rollback as control 9. The checklist below maps to that control's requirements.

Baseline established: you cannot detect degradation without a pre-update benchmark

Before any update, record your current numbers. That means schema validation pass rate, P95 latency, and task-success rate on a fixed eval set. Without this baseline, the on-call engineer cannot tell if new problems are outside normal range. Run your eval set 24 hours before any planned update window. Store the results with a timestamp and the current model version identifier.

Vendor rollback capability confirmed: the five questions to ask before any update window

The five questions in Part 4 show whether rollback is a self-service action, how long a restore takes, and how long the prior version stays available. Get written answers before the update lands. A vendor who cannot answer in writing before an update cannot be trusted for rollback after one. This maps to 10-Point AI Vendor Audit criterion 7: model-update cadence and rollback.

Fallback path operational: if rollback is unavailable, the fallback must be tested and ready

The fallback path is a pre-built route for when the main model fails. It is usually a prior-generation model, a cached-output layer, or a rules-based system for critical outputs. "Degraded mode" is an engineering design state, not a failure state. It keeps customer-facing systems running while the vendor fixes the model. Test the fallback path against your SLOs before the update window. Do not wait until after the alert fires. For the full treatment of fallback path design, see the article on AI fallback path production system design.

Rollback decision authority assigned: who has the authority to trigger rollback at 2 AM without a committee

Incident response gets worse under time pressure. Assign one named role with clear authority to trigger rollback or fallback. Note that on-call rotates, so this is a role, not a person. No committee approval should be required. Write the authority assignment in the runbook next to the trigger criteria. Those 35 minutes spent confirming what the team already suspects are almost always 35 minutes waiting for approval that was never assigned in advance.

Runbook location documented and accessible: not in a Slack thread from three months ago

The rollback runbook must be in the on-call engineer's main incident tool. Not buried in an email thread. Not in a Slack message that needs a search to find. Put the runbook location in the paging alert itself, or in the first document the on-call engineer opens when paged. If the runbook takes more than two clicks to reach from the alert, it will not be used during an incident.

The pre-conditions above work as a gate. All five must pass before any model update proceeds. The checklist below is formatted for screenshot or copy-paste into your pre-update procedure.

Part 2: The Decision Tree for the 2 AM Moment

The decision tree below is built to run at 2 AM when you are tired and stressed. Three yes/no gates decide what to do. Work through them in order. Do not skip gates. The time limits are engineering starting points, not contractual SLAs. They show the window where the on-call team can still act before the incident grows beyond their authority.

For AI production systems, AI system stability means recovery steps must be as clearly written as normal operating steps. The decision tree below is that specification for the rollback case.

Gate 1: Is the degradation confirmed by instrumentation, or only by a customer complaint?

If the only signal is a customer complaint and no monitoring alert has fired, do not treat the degradation as confirmed. Investigate first. Compare current output metrics against the pre-update baseline. If monitoring confirms the problem (schema validation failure rate above baseline, P95 latency above SLO threshold, or eval task-success rate below the pre-update benchmark), move to Gate 2. If monitoring does not confirm it, move to Hold and Monitor. Increase the monitoring frequency and set a re-check point within 15 minutes. The free Stability Auditor checks whether your pipeline has the monitoring layer this gate needs.

Gate 2: Is a vendor rollback available as a self-service action, or does it require vendor support?

If vendor rollback is a self-service action, and the vendor confirmed this in writing before the update window (pre-condition 2), proceed to Roll Back. If rollback requires a support ticket, or is not available at all, move to Gate 3.

Gate 3: Is the degradation causing active customer impact, or is it a quality regression below the SLO threshold?

If the degradation is causing active customer impact (errors visible to customers, SLO threshold breached), activate the fallback path right away. If the degradation is a quality drop that stays within SLO bounds, move to Hold and Monitor with a shorter re-evaluation window.

Part 3: What to Do When the Vendor Says Rollback Is Not Available

When vendor rollback is not available, the playbook does not end. It moves from Roll Back to Fall Back. It also moves from incident response to contract leverage. The three actions in this section assume the fallback path was built before the incident (pre-condition 3).

Activating the fallback path (the pre-built degraded-mode route)

Activate the fallback path as designed. Route traffic to the degraded-mode system. Notify customers per your SLO communication protocol. Open a vendor support escalation with a clear statement of customer impact. Do not improvise a fallback during an incident. If the fallback path was not built before the incident, the only option is to hold and monitor while the vendor fixes the issue on their own timeline. That puts the length of your SLO breach under the vendor's control.

For AI drift detection in production, the monitoring that triggers the decision tree is also the monitoring that tells you when the degradation is fixed. That applies whether the fix comes from a vendor rollback or a later update.

Documenting the vendor failure for contract leverage

Record these during the incident, not after: the timestamp of the vendor update, the timestamp of the first degradation signal, the timestamp of vendor contact, the vendor's stated reason for rollback unavailability, and the customer impact duration. This record is the evidence for the post-incident contract conversation. A vendor who cannot revert on demand has failed the commitment implied by criterion 7 of the 10-Point AI Vendor Audit: model-update cadence and rollback.

The post-incident requirement: add rollback availability to the vendor contract before the next update

The post-incident contract conversation has two goals. First, a written commitment to advance notice before model updates. Second, a written commitment to rollback availability with a stated maximum restore time. A vendor who cannot provide either in writing is a vendor whose model-update risk cannot be managed through a rollback playbook. The fallback path becomes the main risk control, not a backup one.

ISO/IEC 42001:2023, the AI Management System standard, requires organizations to set up corrective action processes for problems, including those caused by vendors. NIST AI RMF 1.0, in the MANAGE function, covers incident recovery and documented procedures for deployed AI systems. Neither standard mandates specific rollback SLAs. They set the process requirement for a documented and tested recovery path. Cite these frameworks in the post-incident vendor conversation as the governance basis for requiring written rollback commitments.

Audit criterion 7 mapping: model-update cadence and rollback

The 10-Point AI Vendor Audit criterion 7 covers model-update cadence and rollback. After the incident, use criterion 7 as the vocabulary for the vendor conversation. Ask: what is the update cadence, how much advance notice is given, and what is the vendor's rollback capability and SLA. A vendor who passes criterion 7 provides written answers to all three before any update window. A vendor who cannot provide these answers after an incident will not provide them before the next one either.

Five Questions to Ask Your Vendor About Rollback Capability

These questions are designed to find vendors who cannot revert on demand. A vague positive answer ("rollback is supported") to any of them is a red flag. A specific, written answer to all five is the minimum bar for vendor rollback capability. Send these before any model update window. Do not wait for an incident.

OWASP LLM Top 10 (2025) identifies LLM09 (Overreliance) as the risk that appears when a team has no rollback path and cannot revert a degraded model update. The five questions above are the operational controls that close LLM09 at the vendor-dependency layer. They work at that layer, not just the prompt or output layer.

How the 12-Control AI Incident Readiness Audit Covers Rollback

Control 9 (rollback) is one of 12 controls in the 12-Control AI Incident Readiness Audit. It is necessary but not enough on its own. A rollback playbook that runs cleanly needs at least three other controls in place: kill-switch (control 1), eval coverage (control 8), and failure-mode visibility (control 12).

Look at the dependency chain. Gate 1 asks whether degradation is confirmed by monitoring. If failure-mode visibility (control 12) is missing, Gate 1 cannot be answered from monitoring. The team falls back to customer complaints as the main signal. That is two of the three discovery paths that are too slow. If eval coverage (control 8) is missing, the team cannot confirm the rollback worked. The revert may restore the prior model version while the problem stays in a component the revert did not touch. If kill-switch (control 1) is missing, there is no fast way to stop all AI-assisted actions while the rollback runs. That means the degraded model keeps processing live traffic during the rollback window.

A reader who has built a rollback runbook covering control 9 has addressed one of twelve controls. The 12-Control AI Incident Readiness Audit covers the full control stack. That includes tool boundary docs (control 2), audit-trail completeness (control 3), sandbox separation (control 4), prompt-injection defenses (control 6), and production data isolation (control 10). Each control depends on the others. Rollback without kill-switch means the degraded model runs during the revert. Kill-switch without eval coverage means the team cannot confirm the revert fixed the degradation. The audit covers all twelve, with the dependency structure made clear.

Conclusion

Rollback is not something you figure out in the moment. It is a pre-condition you confirm before the update lands. The five pre-conditions in Part 1 are cheap to build before an incident. They are very expensive to assemble during one. The decision tree in Part 2 runs in under 15 minutes when the pre-conditions are in place. Without them, it degrades to guessing. The vendor questions in Part 4 are the written commitments that determine whether rollback is in the playbook at all, or whether the fallback path is the only real option.

The engineering background here is 7 years of electrical engineering experience and a BSEE from the University of South Florida. That background applies to AI systems that need the same pre-planned fault-protection discipline as any critical production system. Fault relays are not optional in power systems. Rollback pre-conditions are not optional in AI production systems.