How to Evaluate an AI Agency's Production Track Record Before You Hire Them

Why Agency Demos Tell You Nothing About Production Readiness

An agency demo shows the best case. The inputs are hand-picked. The server is idle, so it feels fast. Nobody asks what happens at 3 AM when the primary model returns a 500 and there is no fallback plan. The demo answers one question: can this agency build something that looks right? It does not answer the question that costs you money: can they build something that keeps working?

Most agency portfolios leave out four things that real production systems need. They leave out monitoring on every critical path. They leave out rollback documentation. They leave out on-call procedures for your specific setup. And they leave out a clear statement of who owns the source code, the model weights, and the audit trail when the work ends. These gaps are not accidents. Agencies gain from being vague about ownership and exit. The demo gains from hiding the failure modes.

Here is a common scenario. A founder picks an agency after a great demo. Six weeks after launch the pipeline starts returning broken outputs on some inputs. The agency has no automated drift alert. There is no rollback path to the prior model version. The prompt templates live in the agency's private repository under their own account. The founder's engineering team cannot diagnose the problem without access to the code. The agency's support takes five business days to respond. That gap was not in the contract because the founder checked demo quality, not production controls.

The criteria below help you close that gap before you sign, not after.

The left column is what most buyers check. The right column is what decides whether the system survives in production. The 10-Point AI Vendor Audit covers all ten items on the right with scored criteria you can use in any agency screening call.

The 10 Evaluation Criteria

Each item below maps to a named control from the sincllm.com 10-Point AI Vendor Audit. The names and descriptions come from that checklist. For each item, you will see what to ask, what a good answer looks like, and what a red-flag answer sounds like.

1. Do they monitor every critical path in production?

Audit Criterion 1: Monitoring on every critical path.

Ask: "Show me a sample of your monitoring dashboard for a production deployment. Which paths are watched, and what triggers an alert?" A good answer names specific things being watched (inference latency, structured output schema validation, API error rate, model response time), the alert threshold for each, and the on-call rotation that gets the alert. A red-flag answer: "We'll set up alerts once it's live." Monitoring planned after launch does not exist when the first incident hits.

2. Can they show you real SLOs from a deployed system?

Audit Criterion 2: Error budgets and SLOs.

Ask: "What SLOs did you define for your last production deployment, and how much of the error budget was used in the first 90 days?" A good answer gives specific numbers for SLOs (uptime target, latency p95, error rate ceiling) and a way to track error budget burn. A red-flag answer: verbal claims about reliability with no numbers, or "We aim for high availability." An agency that cannot name a past SLO cannot negotiate a real one for your system. The NIST AI RMF GOVERN function lists supplier capability documentation as a required due-diligence control, not an optional trust signal.

3. Who owns the source code, model weights, and audit trail?

Audit Criterion 3: Source-code ownership and audit trail.

Ask: "At go-live, who holds the repository, and what does handover look like? Who owns the fine-tuned model weights, prompt templates, and evaluation datasets?" A good answer: your organization owns the source code, the weights, and the audit trail from day one. They live in your own repository under your own account. The agency gets access rather than holding the primary copy. A red-flag answer: "proprietary stack," "our platform," or a handover pushed to project close. ISO/IEC 42001:2023 supplier relationship controls list procurement documentation requirements that match this item. You can exit an agency when the contract is set up right, as the engineering work at we-fine-tuned-a-7b-model-to-replace-an-api shows. You cannot exit when the agency keeps the weights.

4. How do they detect when the model starts behaving differently?

Audit Criterion 4: Drift detection.

Ask: "What automated signal tells you that model outputs have started to drift? What do you do when that signal fires?" A good answer names a specific detection method: output distribution monitoring, schema validation failure rate, semantic similarity scoring against a reference set, or something similar. The signal is automated, not a monthly human review. A red-flag answer: no automated drift signal, or "we review outputs periodically." Drift that is not caught automatically piles up silently until a user reports a problem.

5. What happens when the primary model fails?

Audit Criterion 5: Fallback paths.

Ask: "What is the documented fallback when the primary model returns a 500, a timeout, or a broken response?" A good answer describes a specific fallback: a secondary model endpoint, a rule-based handler, a cached prior response with a staleness limit, or a clear error state shown to the user. The fallback is documented, tested, and can be deployed without a human at 3 AM. A red-flag answer: "we'll handle it" with no documented fallback, or "the model is reliable enough that we haven't needed one." No model is reliable enough to run without a fallback in a production system.

6. How do they detect and alert on cost anomalies?

Audit Criterion 6: Cost-anomaly alarms.

Ask: "What alerts fire before a cost problem shows up in the invoice? What is the threshold, and who gets the alert?" A good answer names a specific cost monitoring setup: token usage per request tracked against a baseline, a daily or hourly budget ceiling with an alert threshold below the hard limit, and a named person who receives the alert (not just the agency's internal team). A red-flag answer: no cost monitoring before the invoice, or "we track it in the vendor dashboard after the fact." A cost problem that only shows up in an invoice has already been running the whole billing period. If you are deciding whether to build your own cost monitoring instead of relying on the agency's, the AI Build vs Buy Framework includes cost monitoring as one of its ten criteria.

7. What is their model-update cadence, and can they roll back to a prior version?

Audit Criterion 7: Model-update cadence and rollback.

Ask: "When you update the model or the prompt, what is the process, and how long does a rollback to the prior version take?" A good answer includes: updates go through a versioned pipeline with staging validation before reaching production, rollback takes under 15 minutes by running one command or pipeline trigger, and every model or prompt version in production has a documented changelog. A red-flag answer: no version control for the model or prompt, or "we update it as needed and haven't needed to roll back." A system with no rollback has no way to recover when an update breaks output quality.

8. What does on-call look like for your integration specifically?

Audit Criterion 8: On-call and incident response.

Ask: "What is the on-call rotation for the specific integration you will build for us? What is the escalation path, and what is the response time SLA for a P1 incident?" A good answer names the on-call engineer or rotation, gives a specific response time for a P1 incident (not "business hours"), and describes the escalation path when the first responder cannot fix the issue. A red-flag answer: generic "business hours" support for a system that runs 24/7, or "we'll define the support terms in the contract." Support terms set after the scope call are terms the agency will cut at negotiation time. OWASP LLM Top 10 (2025) LLM09 (Overreliance) names over-dependence on an AI provider without adequate incident controls as a production risk.

9. How is your data handled, where does it go, and who can access it?

Audit Criterion 9: Data-handling and privacy boundaries.

Ask: "Where does our production data go at inference time? Who at the agency can access it, and when is it used for fine-tuning or logged?" A good answer states the data residency region, names the third-party model provider if one is used, describes the logging policy (what is logged, how long it is kept, and who can access it), and says clearly whether production data is used for training. A red-flag answer: vague data-residency answers, "it stays with our platform," or confirmation that production data is sent to a third-party API under the agency's account instead of yours.

10. What does a clean handover look like, and what happens if you part ways?

Audit Criterion 10: Documented handover, no lock-in.

Ask: "Walk me through what a complete handover looks like. What do we receive, and what dependencies remain on your tooling, accounts, or infrastructure after the work ends?" A good answer includes: a full repository transfer, documentation of every third-party dependency and how to replace it, no license or API dependencies that require the agency to stay involved, and an exit clause in the contract that defines the handover timeline and deliverables. A red-flag answer: no exit clause, dependency on agency-proprietary tooling with no documented alternative, or a handover listed as "to be defined" in the proposal. Checking AI vendor lock-in and code ownership at the contract stage is the next step after this evaluation confirms the agency passes the production controls screen.

How to Use These Criteria Before the First Call

The most common mistake in agency evaluation is asking these questions live on a call. The agency answers verbally and moves on before you can probe. A better approach has three steps.

Step 1: Send written questions 48 hours in advance. Email each agency on your shortlist the 10 evaluation questions before the screening call. Frame them as your standard process, not an attack. A good agency will know these as standard engineering procurement questions. An agency that resists giving written answers is showing you how they handle documentation in production.

Step 2: Require written answers, not verbal explanations on the call. Verbal answers cannot be verified and cannot be compared across agencies. Written answers create a record, force specificity, and give your technical team time to assess each response before you talk again. A written answer that says "we will define monitoring thresholds after go-live" is the same red flag as a verbal one. The difference is that it is in writing where you can point to it later.

Step 3: Score each agency on the same rubric. Use the scorecard below to assign a 1 to 3 score per criterion for every agency on your shortlist. Score all agencies in the same pass, not one after another. Scoring one at a time causes recency bias: the last impressive demo pushes out the memory of a better answer from an earlier agency. The scorecard makes the comparison fair.

Score 3 for a specific, measurable, documentable answer. Score 2 for a partly specific answer that defers one item to post-contract. Score 1 for a vague or verbal-only answer. Any criterion scored 1 is a disqualifying flag unless the agency provides documentation within 48 hours of the call.

Red Flags That Disqualify an Agency Regardless of Price

The Difference Between Demo-Grade and Production-Grade Work

The criteria above are not theoretical. They reflect the controls needed to run a production AI pipeline reliably over time. Sincllm's own production benchmark on sr-demo-ai.com is 99% pipeline reliability across 500+ transcripts. That result did not come from building a good demo. It required monitoring on every inference call, schema validation on every output, a fallback path for malformed responses, and consistent structured output discipline across the full transcript set. The 55 hours per month recovered for one client engagement came from the same foundation: not prompt quality alone, but pipeline reliability that stopped silent failures from piling up into rework cycles.

My background is seven years of electrical engineering in Luanda and a BSEE from the University of South Florida. The framework in this article comes from engineering procurement discipline, not agency pitch culture. In electrical engineering, you check a supplier's production track record before writing the specification, not after. The component data sheet and the failure mode analysis are required inputs to the design. They are not optional. The same logic applies to AI agency selection: the production controls are the data sheet. The demo is the sales brochure.

The NIST AI RMF GOVERN function names third-party supplier due diligence as a required organizational control for AI risk management. ISO/IEC 42001:2023 supplier relationship requirements cover procurement documentation for AI management systems. Neither standard treats a demo and a portfolio as enough due diligence for a production AI engagement. OWASP LLM Top 10 (2025) names LLM06 (Excessive Agency) and LLM09 (Overreliance) as production risks that a poorly structured agency engagement directly introduces when controls like fallback paths, drift detection, and on-call are left undefined.

After You Choose an Agency: What the Contract Must Say

The framework above is the pre-hire screening layer. Once an agency passes the 10-criterion screen, the next step is making sure the contract captures the answers you got. An agency that gave good verbal answers about ownership and exit clauses must have those terms in the contract. An agency that deferred one item to "post-contract alignment" must resolve that before you sign.

Two articles cover the contract stage in detail: AI vendor contract questions (the specific questions to negotiate at the contract stage) and AI vendor lock-in and code ownership (the exit-risk dimension, including what happens to the weights and the codebase if the engagement ends early). Both cover the contract-signing stage. This article covers the pre-hire evaluation stage. They are designed to be used in order.

Conclusion

The question "what is your production track record?" is the right question to ask an AI agency before you hire them. The 10 criteria in this article make that question answerable before you sign: monitoring on every critical path, error budgets and SLOs, source-code ownership and audit trail, drift detection, fallback paths, cost-anomaly alarms, model-update cadence and rollback, on-call and incident response, data-handling and privacy boundaries, and documented handover with no lock-in.

A good agency will not be surprised by these questions. They will have specific, documented, verifiable answers for each one. An agency that cannot answer them has not built production systems. They have built demos. The difference between those two things is the cost you pay when the system fails at 3 AM and there is no rollback, no alert, and no on-call engineer who knows your specific integration.