AI Build vs Buy: The 10-Criteria Checklist Engineering Teams Actually Use

Why Most Build vs Buy Advice Fails Engineering Teams

Most build vs buy articles on the web fall into one of two groups. The first group is vendor-sponsored content. It pushes you toward Buy no matter what. The second group is generic software frameworks written for SaaS deals. Neither group covers the three things that actually decide AI sourcing. The first is model drift: the model changes behavior over time without a code deploy, so someone has to catch it and respond. The second is data residency: where inference runs decides your regulatory risk. A managed API may process your data in a place you cannot audit. The third is failure-mode visibility: who owns the response plan when the model produces wrong output at 3 AM. Standard make-or-buy guidance, documented in the IEEE Software Engineering Body of Knowledge Section 10 and referenced by Gartner's Build, Buy, or Partner framework, covers cost, time, and strategic fit. It does not cover these three. This checklist fills that gap.

A wrong sourcing decision does not show up on day one. A vendor gives you speed at signing and lock-in at renewal. An in-house build gives you control at month six and a runbook gap at month twelve. The real cost of the wrong decision is not a dollar amount. It is the engineering time you spend reversing it while your AI work stalls. The NIST AI Risk Management Framework (MAP function) requires teams to weigh AI acquisition against development. Criteria include risk ownership, auditability, and operational control. This checklist maps those requirements to 10 specific engineering signals.

How to Use This Checklist

Each of the 10 criteria below points toward Build, Buy, or Hybrid. Check each one against your situation. Note which way it leans. The total is a directional input, not a final answer. If seven criteria lean toward Build and three lean toward Buy, that is a Build-leaning decision. You have known trade-offs on the three Buy criteria. It is not a mandate to do one or the other.

The full scoring template is in the AI Build vs Buy Framework. It includes numeric weights per criterion and a one-page decision matrix for board presentations. This article gives you the criteria and the reasoning. The download gives you the scored format.

The 10 Criteria

1. Time-to-Value: How Fast Does the Initiative Need to Deliver?

Most teams check this criterion first. It is also the one most likely to give you a wrong answer when checked alone. If your project needs results in eight weeks, a managed API or SaaS AI platform has a real advantage. The inference infrastructure is already running. The API is documented. Integration starts on day one. That speed is real. The question is what you pay for it later. Tilt toward Build: the project has a twelve-plus month horizon, ML iteration is expected, and the team can accept a slower start in exchange for long-term ownership. Tilt toward Buy: the project needs results in under three months, the use case is clear, and speed matters more than long-term cost control. Hybrid signal: buy a foundation model API now, plan the move to a self-hosted or fine-tuned model before month twelve.

2. Strategic Differentiation: Is This AI a Core Competency or a Utility?

If the AI capability sets your product apart in the market, building it yourself is a defensibility argument. If the capability is a commodity (document summarization, customer service routing, translation), buying is a cost argument. The difference matters. A vendor can copy your integration but not your fine-tuned model or your proprietary training data. Tilt toward Build: the AI capability is a product moat and your training data is proprietary. Tilt toward Buy: the capability is commodity AI and your competitive edge is elsewhere in your stack. Hybrid signal: buy the base model, build the fine-tuning layer and the proprietary data pipeline.

3. Data Sensitivity and Residency: Where Does Your Data Have to Live?

Generic frameworks skip this criterion entirely. If inference runs on patient records, financial data, or any data under regulatory rules, where the vendor's servers are located is not a preference. It is a compliance requirement. A managed API that routes data through a place you cannot audit may disqualify itself before you weigh any other criterion. NIST AI RMF MAP function guidance requires teams to evaluate data-handling boundaries as part of AI acquisition decisions. Tilt toward Build: data must stay on-premises or in a specific cloud region, inference must be auditable, or data is subject to privacy rules that block third-party processing. Tilt toward Buy: data is non-sensitive, publicly available, or already covered by a cloud data-processing agreement that includes the vendor. Hybrid signal: buy the model weights and host them in your own infrastructure. This removes the data-residency risk without the cost of training from scratch.

4. In-House ML Talent: What Can Your Team Actually Maintain?

An in-house build that nobody on the team can maintain after the contractor leaves is not really a build. It is deferred technical debt. ML talent is scarce. "We have software engineers" is not the same as "we have engineers who can retrain, evaluate, and deploy a production model." If your team cannot own the model lifecycle, the Build path creates a dependency on whoever built it first. Tilt toward Build: you have ML engineers on staff. They can own training, evaluation, and deployment without outside help. Tilt toward Buy: your team is strong on software engineering but has no ML training or fine-tuning experience. Hybrid signal: buy a pre-trained model, hire or contract one ML engineer to own fine-tuning and evaluation, and build the integration and serving layer yourself.

5. 3-Year Total Cost: What Does Each Path Cost Over a Real Horizon?

The six-week cost comparison almost always favors Buy. The three-year comparison is more complicated. A managed API charges per token or per call. Those costs grow with usage. An in-house model has higher upfront infrastructure and talent costs but a lower cost per additional request at scale. Neither path is always cheaper. The answer depends on your usage volume, your infrastructure costs, and whether you count the engineering time needed to maintain each path. Evaluate both paths over three years, not six weeks. Include the cost of reversing the decision if the chosen path fails. Tilt toward Build: usage volume is high enough that per-call API costs exceed infrastructure plus talent costs over three years. Tilt toward Buy: usage volume is moderate and the per-call cost stays below the infrastructure-plus-talent breakeven. Hybrid signal: run the cost model at your expected usage growth rate and set a trigger point where you would move from Buy to Build.

6. Vendor Lock-in Tolerance: What Is Your Exit Cost if the Vendor Changes Terms?

Vendor lock-in in AI has a specific shape. It differs from traditional SaaS lock-in. The API is not just a data format. It is a prompt schema, a context window size, a model version, and a pricing tier. When the vendor deprecates a model version, raises per-call pricing, or changes their SLA, your integration breaks or your cost model breaks. The exit cost is the engineering time to reconnect to a new vendor or move to a self-hosted model. Tilt toward Build: your tolerance for vendor-driven pricing or capability changes is low, or your use case makes reconnection expensive. Tilt toward Buy: your use case is simple enough that switching vendors is a two-week project, not a six-month one. Hybrid signal: build an abstraction layer over the vendor API so that a model swap is a configuration change, not a full reconnection.

7. Regulatory and Audit Requirements: What Does Your Industry Require You to Prove?

Regulated industries require you to prove what your AI system did, when it did it, and on what data. A vendor that cannot give you an audit trail of model inputs, outputs, and decisions is a risk in a compliance audit. NIST AI RMF MAP function guidance requires AI system acquisition decisions to account for auditability and accountability. If your industry requires an audit trail on demand, this criterion is binary. The vendor either supports it or they do not. Tilt toward Build: your industry requires a full audit trail of AI decisions that a vendor cannot provide. Tilt toward Buy: your use case is not subject to AI-specific regulatory requirements, or the vendor provides the required audit trail as part of their service. Hybrid signal: buy the model, build the audit instrumentation layer yourself.

8. Integration Depth: How Tightly Coupled Does This Need to Be?

A chatbot that calls an API is loosely connected. An AI system that reads from your internal database, writes back to your CRM, triggers downstream workflows, and runs as a component in a larger agent mesh is tightly connected. Tight connection means the AI system needs to understand your data schema, your business logic, and your failure modes. A generic vendor cannot provide that context. Your team can. Tilt toward Build: the AI system reads and writes to internal systems. It operates as part of a larger workflow and requires deep knowledge of your data and business logic. Tilt toward Buy: the AI system is a standalone capability with a clean API boundary and no dependency on internal data or logic. Hybrid signal: buy the inference layer, build the integration and orchestration layer yourself.

9. Iteration Cadence: How Often Do You Need to Change the Model or Behavior?

If your use case requires weekly retraining, prompt schema updates, or behavior changes driven by new data, you need control over the full model lifecycle. A managed API gives you control over the prompt, not the model. If the prompt is the only control you need, Buy is enough. If you need to retrain on new data, adjust model behavior at a deeper level, or run A/B tests on model versions, you need the Build path. At minimum you need a hybrid that gives you fine-tuning access. Tilt toward Build: iteration cadence is weekly or faster, retraining on new data is expected, or you need to run controlled experiments on model behavior. Tilt toward Buy: the use case is stable, model behavior is fixed, and prompt-level control is enough. Hybrid signal: buy a foundation model with fine-tuning access (not just prompt access), so you control iteration at the fine-tuning layer without running pre-training infrastructure.

10. Failure-Mode Visibility: Who Owns the Runbook When It Breaks?

Every AI system in production will degrade, produce wrong output, or fail. The question is not whether it will happen. The question is who responds at 3 AM and what their response plan says. With a vendor, the response plan is a support ticket and an SLA. With an in-house build, the response plan is owned by your team and can be as specific as your use case requires. Failure-mode visibility is not about distrusting vendors. It is about operational control. If a production failure costs you more than the vendor SLA compensates, you need to own the response. Tilt toward Build: production failures are high-cost, the vendor SLA does not cover your exposure, or your team needs to own failure detection and response. Tilt toward Buy: the vendor SLA covers your exposure, on-call responsibilities are acceptable to delegate, and your use case can tolerate the vendor response time. Hybrid signal: buy the model. Build your own monitoring, alerting, and fallback layer so you own failure detection even when you do not own the model.

The table below summarizes each criterion for quick reference. The full scoring template with numeric weights is in the downloadable Build vs Buy Framework.

Quick Self-Assessment: 10 Build Signals

For each question, a "Yes" answer is a Build signal. A "No" answer means look again with a Buy or Hybrid lens.

A Real In-House Build Example: sincllm-mcp v2.0.0

sincllm-mcp v2.0.0 is sincllm's own internal production tooling. It runs 12 tools in production. It was built in-house rather than assembled from managed API providers. It is not a client deliverable. It is the production AI infrastructure that powers sincllm.com's own systems.

Four criteria pushed the decision toward Build. On C3 (Data Sensitivity and Residency), inference runs on content and logic that could not be routed through a third-party API without creating data-handling exposure. On C8 (Integration Depth), the tools are tightly connected to internal workflows. They require deep context about system state that a generic managed API cannot hold. On C9 (Iteration Cadence), the tooling is updated and extended regularly. That requires control over the full model lifecycle, not just the prompt layer. On C10 (Failure-Mode Visibility), production failure response needs to be owned by the team. It requires a specific response plan, not delegation to a vendor support queue. For the technical details of what an in-house build looks like in production and a concrete example of when replacing a vendor API with an in-house model made sense, see the linked posts. The decision was not ideological. On C1 (Time-to-Value) and C4 (ML Talent), the decision carried real cost. The build took longer than a managed API integration would have. It required engineering investment that a Buy path would have deferred. The framework does not say Build is always better. It says you need to score all 10 criteria before the decision is defensible.

Common Mistakes in Build vs Buy Decisions

• Underestimating the 3-year maintenance cost of a build. The initial build is a one-time cost. Retraining, evaluation, deployment, monitoring, and drift response are recurring costs. Teams that look only at the upfront build cost consistently underestimate the total cost of ownership over three years.
• Overestimating vendor flexibility after signing. Vendor demos show best-case behavior. The contract governs worst-case behavior. Terms that seem flexible during the sales process (API versioning, pricing tiers, data-handling clauses) tend to tighten at renewal. Evaluate the contract, not the demo.
• Ignoring ML talent attrition as a build-side risk. An in-house build that depends on one or two ML engineers creates a key-person risk. If those engineers leave, the team inherits a production system they cannot maintain. Account for attrition risk in the C4 (ML Talent) criterion before committing to a full build.
• Skipping regulatory mapping before vendor selection. Teams often select a vendor and then discover a data residency or audit trail requirement that the vendor cannot meet. The regulatory mapping (C7) should happen before vendor evaluation, not after signing.

What to Do After the Decision

If your criteria tilt toward Buy: the next step is a structured vendor evaluation before you sign. Most vendor contracts look standard until you check them against production engineering criteria: source-code ownership, audit trail, SLO terms, fallback paths, and exit clauses. Before you commit to a vendor, run a pre-signing vendor audit using the 10-Point AI Vendor Audit. Evaluate the vendor against the same criteria you just applied to the sourcing decision. A vendor who cannot answer those questions in writing is a vendor whose contract governs your worst-case behavior.

If your criteria tilt toward Build: set your evaluation criteria before you ship, not after. Define what success looks like on C10 (Failure-Mode Visibility) before the first production deployment. A build without a response plan is not a production system. It is a prototype that happens to be running in production.

Either way: download the scoring template from the Build vs Buy Framework to run the numeric version of this evaluation. It produces a one-page decision matrix you can present to a board or investor. The criteria in this article are the same criteria in the framework. The download gives you the scored format.

Conclusion

The build vs buy decision is not permanent, but the contract is. A vendor agreement signed without evaluating data residency, lock-in terms, and audit requirements governs your production system for the length of the contract, often longer. An in-house build committed to without evaluating ML talent depth and 3-year maintenance cost becomes a production system your team cannot maintain or exit. The 10 criteria in this checklist are the engineering-specific signals that make the decision defensible rather than a gut call. Run them against your specific situation. Score them. Download the framework to produce a decision you can present to a board. If your criteria tip toward Buy, run the vendor evaluation before you sign. Every production system deserves the same engineering rigor. This decision is no different.