AI Vendor Audit: What to Do After the Vendor Fails One of Your Criteria

A Criterion Failure Is Not the Same as a Deal Breaker

You ran the 10-Point AI Vendor Audit. The vendor failed one of the criteria. What you do in the next 48 hours matters. It decides whether the gap helps you negotiate, or hurts you in production.

Not every failed criterion is equally serious. A missing SLO definition (criterion 2) is a different problem from missing source-code ownership documentation (criterion 3). The first can be fixed in 30 days with a written promise. The second is more severe. If not fixed before you sign, you cannot leave or audit the system without the vendor. Your job now is to respond in a structured way. Classify the gap. Write it down. Get a written commitment. Apply a go/no-go gate before moving forward.

The four-step protocol below maps to the 10 criteria from the AI Vendor Audit. It gives you a clear action for each failure type. It also gives you the decision table you need to apply the gate with confidence.

Step 1: Document the Gap in Writing Before Anything Else

Before you email the vendor, write it down. Before you schedule a call, write it down. Before you talk to your own team, write it down. Record the criterion name and number. Record the evidence of failure: what the vendor said, or what they failed to produce. Record the vendor response, if any. Record the date you found the gap.

A verbal acknowledgment has no legal standing. A vendor who says "we are working on that" on a sales call is not committing to anything. If you keep talking without a written record, you have no negotiation evidence. You also have no audit trail for your legal team if the problem appears after launch. Criterion 3 from the 10-Point AI Vendor Audit covers source-code ownership and audit trail. It exists for this exact reason. Written records separate engineering accountability from sales talk.

Use the checklist below as your template for the written gap record. Fill it out before any further contact with the vendor.

Step 2: Classify the Failure Severity Against the 10 Criteria

The 10 criteria from the 10-Point AI Vendor Audit split into two classes: hard stops and remediable gaps. A hard stop means the gap puts you in a position you cannot recover from. It may also take away your ability to respond to an incident. A remediable gap is different. A written commitment and a 30 to 90 day deadline can close it without changing the contract structure.

Getting the classification right is the most important step in this protocol. NIST AI RMF 1.0 (GOVERN function, airc.nist.gov/RMF/1) says supplier risk monitoring is an ongoing requirement. It is not a one-time check. The severity classes below map directly to that requirement. Hard stops need a contract fix before you sign. Remediable gaps need a time-boxed written commitment.

Hard-Stop Criteria (Require Contractual Cure or Exit)

These four criteria are hard stops. A gap in any of them either locks you into the vendor without an exit, or takes away your ability to respond when something breaks in production. OWASP LLM Top 10 (2025, LLM06: Excessive Agency and LLM07: System Prompt Leakage) lists control gaps like these as the highest-severity production risks.

• Criterion 3: Source-Code Ownership and Audit Trail. If you do not own the source code, you have no way out. If you cannot produce an audit trail without the vendor, you have no way out. See the source-code ownership clause guide for the contract language this requires.
• Criterion 7: Model-Update Cadence and Rollback. A vendor with no documented rollback procedure leaves you exposed. If a model update breaks your pipeline, you cannot reverse it. There is no engineering reason to proceed without this.
• Criterion 8: On-Call and Incident Response Structure. The vendor must show you a written incident response structure. It must name owners and SLAs. If they cannot, the 12-Control AI Incident Readiness Audit shows exactly what is missing. A vendor without this is not ready for production.
• Criterion 10: Documented Handover and No Lock-In. A vendor that cannot provide a documented handover procedure blocks your exit. They keep you locked in by removing your options, not by delivering value. The Build vs Buy Framework covers switching cost and lock-in tolerance as a formal decision criterion.

Remediable Criteria (30 to 90 Day Window)

These six criteria can be fixed with a written remediation commitment and a specific deadline. The vendor must agree in writing. The deadline must carry a consequence clause. ISO/IEC 42001:2023 supplier relationship sections (iso.org/standard/81230.html) provide the governance framework for this. They treat these as corrective action items with documented closure requirements.

• Criterion 1: Monitoring on Every Critical Path. Fixable with a 30-day deadline. Require the vendor to provide monitoring dashboard access. Or require documented alert coverage before go-live.
• Criterion 2: Error Budgets and SLOs. Fixable with a 30-day deadline. Require written SLO definitions. Require a measurement method and a reporting schedule.
• Criterion 4: Drift Detection. Fixable with a 60-day deadline. Require a documented detection method. Require an alert threshold and a named owner.
• Criterion 5: Fallback Paths. Fixable with a 60-day deadline. Require a documented fallback procedure. It must include a tested activation sequence.
• Criterion 6: Cost-Anomaly Alarms. Fixable with a 30-day deadline. Require alert configuration and threshold documentation.
• Criterion 9: Data-Handling and Privacy Boundaries. Fixable with a 90-day deadline, depending on the regulatory context. EU AI Act deployer obligations (Regulation 2024/1689) and functional safety procurement standards both require documented data boundaries for high-risk AI systems. This gap is fixable only if the vendor can produce the documentation in time.

Step 3: Set a Written Remediation Deadline in the Contract

Once you have written down the gap and classified it as remediable, the next step is a written remediation clause in the contract. "Written" here means a contract amendment or an addendum. At minimum, it means a formal written notice that the vendor acknowledges in writing. An email thread is weak. A signed addendum is enforceable.

A remediation clause must have four parts to be usable. First: the specific criterion by name and number. Second: a measurable outcome that defines what "remediated" looks like. Third: a deadline date. Fourth: a consequence clause. The consequence clause states what happens if the deadline is missed (usually the right to exit the contract without penalty, or a fee reduction). Without all four parts, the clause cannot be enforced. There is nothing measurable to check at the re-audit date.

What if the vendor refuses to put the remediation commitment in writing? That refusal is itself a signal. A vendor who can meet the criterion has no reason to refuse a written commitment. Refusal means one of two things. Either the capability does not exist right now. Or the vendor is not willing to be held to a measurable standard. Both outcomes move the gap into the hard-stop class. This applies no matter what the original classification was. See the source-code ownership and exit clause guide for the contract language to use when the vendor will not commit in writing.

For high-risk AI systems, EU AI Act deployer obligations (Regulation 2024/1689) require documented monitoring of vendor performance. A remediation clause with a measurable outcome and a deadline is the minimum you need. ISO/IEC 42001:2023 (iso.org/standard/81230.html) supplier relationship sections require documented nonconformity correction. This step maps directly to that requirement.

Step 4: Apply the Go / No-Go Gate

The go/no-go gate is a three-row decision table. You apply it once you have three things: the written gap record from step 1, the severity classification from step 2, and the vendor's written response from step 3. The table removes guesswork from the decision. It replaces it with a documented process.

When you write the decision memo for legal and finance, it needs three things. First: the criterion that failed. Second: the severity classification and how you reached it. Third: the action taken and why. This memo protects you if the vendor later disputes the exit. It also protects you if the gap appears in a due-diligence review. The pre-signing questions checklist describes the questions that should have surfaced these gaps earlier. If those questions were not asked, the gap record is also evidence for improving the next vendor evaluation.

Re-Running the Audit After Remediation

The remediation deadline is not the finish line. It is the trigger for a re-audit. When the deadline arrives, run the 10-Point AI Vendor Audit again on the criteria that failed. Use documented evidence the vendor provides. Do not accept verbal updates. The re-audit uses the same tool on the same criteria. It is not a new evaluation. It does not replace the original audit record.

For remediable gaps, run the re-audit on this schedule. At 30 days after signing: run an interim check. Is the vendor making documented progress? At the deadline date: run a full re-audit of the failed criterion. At go-live: run a full re-audit of all 10 criteria, not just the ones that failed. Passing a re-audit on one criterion at one point in time does not cover all criteria. It does not cover future states of the system. A vendor that passes re-audit at go-live but then pushes a model update without a documented procedure has reopened criterion 7.

Engineering discipline here mirrors the audit-trail requirement from criterion 3. Every re-audit produces a dated record. The record must include the evidence reviewed, the outcome (pass or fail per criterion), and the next review date. A vendor that resists re-audit after a written remediation commitment is sending a clear signal. The capability does not exist.

When to Bring in an Independent Engineer

Three situations call for independent verification. Your internal team may not be able to provide it alone.

First: your internal team does not have production AI engineering experience. Checking that a rollback procedure actually works requires more than reading it. It requires someone who can judge whether the procedure maps to a real system state, not just a PDF. The same is true for audit-trail completeness (criterion 3) and incident response structure (criterion 8). Marketing language and engineering documentation look identical to a non-technical reader.

Second: the vendor relationship is high-stakes. A multi-year, multi-system contract creates a high switching cost if things go wrong. At that scale, the cost of an independent production audit is small. Compare it to the cost of finding a hard-stop gap 18 months into the contract. The Build vs Buy Framework treats vendor lock-in tolerance as a formal criterion. The switching cost grows over the contract term.

Third: the gap is in a technical criterion you cannot verify on your own. If the vendor claims to have fixed criterion 7 (rollback) by updating a runbook, you need to check that claim. Checking it means reading the runbook against the system architecture. It means confirming the rollback procedure would actually work during a production incident. Without production AI engineering experience, you cannot tell a real rollback from a description of one. The production AI audit service covers exactly this. You bring the vendor's documentation. The engineer applies the same production-engineering standard used in building sincllm-mcp v2.0.0 (12 tools) and the 99% pipeline reliability benchmark on sr-demo-ai.com (sincllm's own production benchmark, not a vendor SLA or client guarantee).

Conclusion

A criterion failure found before you sign is leverage. You have a written gap, a severity classification, and a contract tool to require a fix or exit cleanly. The same failure found six months after go-live is a liability. There is no audit trail. There is no remediation clause. There is no exit right. The vendor now has every reason to minimize the problem rather than fix it.

The four-step protocol in this article turns an unclear vendor gap into a documented engineering decision. The steps are: document, classify, set a written deadline, and apply the gate. This is not adversarial oversight. It is the same audit-trail discipline that criterion 3 from the 10-Point AI Vendor Audit requires of the vendor. Applying it to your own process is the engineering standard for production AI procurement.