10 Questions to Ask Before You Sign an AI Vendor Contract

Why the Contract Matters More Than the Demo

Every AI vendor demo shows the best case. The model answers well. Speed is good. The setup looks neat. But the demo cannot show you what happens at 3 AM when the model breaks. It cannot show you who is on call. It cannot show you whether you can roll back to an older version. It cannot show you what happens when you need to leave.

The contract controls all of that. Once you sign, your power to change those terms is gone. If you skip the SLO discussion, you have no written right to demand a fix when things break. If you never ask about data handling, you have no commitment to enforce when a breach happens. Your window to negotiate is before you sign. After you sign, you get whatever the vendor's lawyers wrote for the vendor's benefit.

The 10 questions below come from sincllm's 10-Point AI Vendor Audit. That is the same framework used to check production AI systems. You can send these questions before the vendor meeting and ask for written answers. Each question shows you what a good answer looks like and what a red flag answer looks like. That way you can score every vendor the same way.

Frameworks like the NIST AI Risk Management Framework's GOVERN function, ISO/IEC 42001:2023's supplier relationship requirements, and the EU AI Act's obligations for high-risk AI providers and deployers all say that checking vendors is the buyer's job, not a favor the vendor does. Asking these questions is not hostile. It is the standard.

The 10 Questions

Each question maps to one of the 10 audit criteria from the 10-Point AI Vendor Audit. A good answer is specific, measurable, and locked into the contract. A vague answer is a red flag.

1. Who monitors every critical path, and what is the SLO?

Why it matters: Monitoring every critical path (Audit Criterion 1) is the bare minimum for any AI system in production. If the vendor cannot name the endpoints they watch, the alert thresholds, and the SLO in the contract, you have no basis to demand a fix when things go wrong.

Good answer: "We watch inference latency, error rate, and availability on every endpoint in your setup. Our SLO is 99.5% availability per month. We have a written credit schedule when we miss that. Here is the dashboard you will see."

Red flag answer: "We have great infrastructure and strong uptime history." (No number. No list of what they watch. No contract promise.)

2. What are the error budgets, and what triggers an escalation?

Why it matters: Error budgets and SLOs (Audit Criterion 2) set the rules for how much failure is okay. Without them, every problem turns into an argument. You need to know what number triggers a human call, what that process looks like, and what happens if the budget runs out early.

Good answer: "Your error budget is 0.5% per month. When 50% of that is used, you get an automatic alert. When it is all gone, the on-call lead is paged and you get an incident notice within 30 minutes."

Red flag answer: "We will reach out if there are issues." (No threshold. No budget. No trigger defined.)

3. Do you own the source code, model weights, and audit trail?

Why it matters: Source-code ownership and audit trail (Audit Criterion 3) decide your exit options and your compliance stance. If the vendor owns the model weights and you cannot see the audit trail, you cannot prove compliance on your own. You also cannot move to a new vendor without starting over. The OWASP LLM Top 10 (2025) lists Excessive Agency (LLM06) and Overreliance (LLM09) as risks that only ownership and audit controls can limit, not vendor trust.

Good answer: "You own the fine-tuned weights for any model trained on your data. You can read the audit trail at any time. When the contract ends, we give you a full export of weights, prompt templates, and audit logs within 30 days."

Red flag answer: "The model is hosted on our platform. We handle that infrastructure." (No ownership claim. No audit trail access. No export promise.)

4. How does the vendor detect and alert on model drift?

Why it matters: Drift detection (Audit Criterion 4) tells you when the model that passed your tests is no longer the model running in production. Without it, quality gets worse and nobody notices until a user complains or a system fails quietly.

Good answer: "We run automatic quality checks against a test set every week. If quality drops past a set threshold, you get an alert within 24 hours with the specific cases that failed."

Red flag answer: "Our models are updated all the time for quality." (No detection system. No alert. No threshold defined.)

5. What fallback path activates when the primary model fails?

Why it matters: Fallback paths (Audit Criterion 5) are what turn a failure into a recovery instead of an outage. A production AI system with no fallback is one broken part away from going down. This question shows whether the vendor planned for failure or just hoped the main path would hold.

Good answer: "If the main endpoint is down, traffic moves to a backup endpoint within 30 seconds. If both are down, your system gets a safe fallback response that we define in the integration spec. We can show you the runbook."

Red flag answer: "We have high availability. Downtime is rare." (No fallback. No backup endpoint. No defined behavior when things go down.)

6. How does the vendor detect cost anomalies before they become a surprise invoice?

Why it matters: Cost-anomaly alarms (Audit Criterion 6) protect you when a model acts unexpectedly, when token use spikes after a prompt change, or when usage shifts in production. Without alarms, the first sign of a problem is a large bill.

Good answer: "We alert you when your usage hits 80% of your contracted volume in a billing period. You can set a hard spending cap. We send daily spend summaries and a webhook for spend events."

Red flag answer: "You can check usage in your dashboard." (No alert. No spending cap. No notice sent to you.)

7. What is the model-update cadence, and can you roll back to a prior version?

Why it matters: Model-update cadence and rollback (Audit Criterion 7) decide whether the behavior you tested is still what runs next month. A quiet model update can break downstream systems. It can cause errors in contexts that worked before. It can change the model's compliance stance with no warning.

Good answer: "We announce major updates 30 days ahead. You can pin your setup to a named model version. If a new version hurts your outputs, you can roll back to the last pinned version within 48 hours by opening a support request."

Red flag answer: "We update our models often to improve performance." (No schedule. No advance notice. No right to roll back.)

8. What does on-call and incident response look like for your integration specifically?

Why it matters: On-call and incident response (Audit Criterion 8) tell you what happens when your integration breaks in production and you need a real person, not a ticket queue. General SLAs about "support response time" are not the same as a named escalation path for your specific setup. See sincllm's AI Incident Readiness Audit to check whether a vendor has real incident controls.

Good answer: "For production incidents in your setup, we assign a named on-call engineer during onboarding. We issue an incident declaration within 30 minutes of detection. We deliver a post-mortem within 5 business days with the root cause and a fix timeline."

Red flag answer: "Our support team is available 24/7." (No escalation path. No named on-call person. No post-mortem promised.)

9. How is your data handled, where is it stored, and who can access it?

Why it matters: Data handling and privacy boundaries (Audit Criterion 9) set your regulatory risk. If the vendor stores your data in the wrong country for your rules, trains on your data without your consent, or shares it with third parties without telling you, those are contract and legal risks you must settle before signing. Use sincllm's free adversarial validator tool to test prompt-injection defenses before you sign. Prompt injection is one way data boundaries get broken at runtime.

Good answer: "Your data is stored in [named region] only. We do not train on customer data without your written consent. All subprocessors are listed in our DPA. You can request a data deletion report at any time."

Red flag answer: "We take security very seriously." (No region named. No training policy. No subprocessor list.)

10. What does a clean handover look like, and what does the exit clause say?

Why it matters: Documented handover and no lock-in (Audit Criterion 10) decide whether you can actually leave when you want to. A vendor exit is possible when the contract lets you do it: see proof that a vendor exit is engineering-feasible when the contract allows it. A contract that does not define how you exit, how long the export takes, or what documents you get creates lock-in no matter what the vendor says. This question makes vendors uncomfortable because it is the one that matters most.

Good answer: "When the contract ends, we give you: (1) a full export of your model weights and training data within 30 days, (2) all prompt templates and integration docs, (3) a runbook for moving to another provider or your own setup. Schedule B spells out the 30-day retention window and the export format."

Red flag answer: "We hope you will not need to leave, but we can talk about that later." (No export promise. No timeline. No documentation list.)

Red Flag Answers to Watch For

The table below is a quick reference for your vendor meeting. A red-flag answer does not cut a vendor automatically. It is a signal to ask for specifics. If a vendor cannot replace a red-flag answer with a clear, written commitment, that is a disqualifying finding.

How to Use This Checklist in a Vendor Meeting

These questions work best when you send them in writing before the meeting. Do not just read them aloud in the room. That gives vendors time to give polished off-the-cuff answers.

Step 1: Send the questions 48 hours in advance. Email the vendor all 10 questions. Label them "required written responses for contract evaluation." This makes clear that verbal answers are not enough. It also gives the vendor time to bring in the technical team, not just the sales team.

Step 2: Require written answers. Verbal answers in a meeting are not enforceable. They often do not match the contract anyway. Written answers give you a record to compare against the contract draft. If a vendor's written answer and their contract contradict each other, you must resolve that before signing.

Step 3: Score across vendors on the same rubric. Use the "What Good Looks Like" column in the table above as your standard. Give each answer a pass or fail for each vendor. Recency bias (the last vendor you talked to seems best) is a common buying mistake. A consistent rubric stops that.

Step 4: Push on every red-flag answer. A red-flag answer is not a deal-breaker on its own. Ask the vendor to replace it with a specific, measurable commitment. If they cannot, log that as an unresolved finding. An unresolved finding on Criterion 3 (ownership) or Criterion 10 (exit clause) is a contract risk. Take it to your legal team before signing.

Printable pre-signing checklist: Copy the list below. Share it with your team. Fill in the vendor response for each question.

What Comes After the Contract

Getting good answers to all 10 questions before signing is your moment of power. The terms you lock in now govern every future incident, every cost fight, and every exit. That is why this checklist exists. Written commitments on these points are non-negotiable for any AI deployment in production.

But the contract is not the end. It is the start. Once you sign, the question changes. It is no longer "what did the vendor promise?" It becomes "is the vendor keeping those promises in production?" That is a different question. It needs a different tool.

The 10-Point AI Vendor Audit is the ongoing check version of this pre-signing list. It uses the same 10 criteria. You can run it every quarter or after any big change: a model update, a new integration, a price renegotiation, or an incident review. The pre-signing list sets the standard. The audit checks whether the vendor is meeting it.

The audit is a free 16-page PDF. You get it by entering your email. It takes 15 minutes per vendor. If you just negotiated a contract with these 10 questions, the audit is your next step. It shows whether those contract promises are being kept in production.

Conclusion

The contract is the one moment when you hold all the power. Before you sign, you can negotiate SLOs, rollback rights, data handling rules, exit clauses, and incident response steps. After you sign, you are stuck with whatever the vendor's boilerplate says. That boilerplate is written for the vendor, not for you.

These 10 questions are not full legal advice. They are engineering-based due diligence checks. They are aligned with the NIST AI RMF GOVERN function, ISO/IEC 42001:2023 supplier requirements, and the EU AI Act's rules for high-risk AI deployers. Use them before every AI vendor talk. Ask for written answers. Score every vendor the same way. Then use the 10-Point AI Vendor Audit to check that those promises are kept after you sign.