When to Distill a Vendor Model Into a Local One: The Engineering Case for Self-Hosting

The Decision Is Not About Capability, It Is About Control

Most engineers ask the wrong question first. They ask: "Is the local model good enough?" That is the second question, not the first. The first question is: "Has the vendor created a dependency that now puts the business at risk?" Four signals answer that question: cost, data residency, update cadence, and in-house ML talent. None of them measure capability. All of them are engineering limits. They either make the vendor relationship safe to keep, or they make self-hosting the smarter choice.

This article follows the real 10-criterion AI Build vs Buy Framework. Four of those criteria matter most for distillation: criterion 3 (data sensitivity and residency), criterion 4 (in-house ML talent), criterion 5 (3-year total cost), and criterion 7 (regulatory and audit). Each of the four signals maps to one of those criteria. The article also shows the full 10-criterion matrix scored for distillation versus staying vendor. If you do not yet know your per-call cost, start with the 9-Question AI Spend Audit. It finds the cost and volume numbers you need for the comparison in Signal 1.

Distillation is a behavior and ownership decision, not the default way to keep knowledge current. The RAG vs fine-tuning vs long-context guide separates those architecture goals before you commit to training.

The Four Signals That Trigger the Distillation Decision

Signal 1. Inference Cost Crosses the Engineering Threshold

The cost case for self-hosting only works when you compare the right numbers. Take the per-call API price and multiply it by your projected monthly volume. That is one side. On the other side, add up the GPU hardware cost spread over 36 months, plus the time an ML engineer spends on maintenance. If the 36-month total favors self-hosting and the payback period is under 12 months, the cost signal is real. If you are only comparing this month's bills, or using today's volume without a growth projection, that is not a cost signal. That is a guess.

Build vs Buy criterion 5 (3-year total cost) is the right tool for scoring this signal. If you do not have a per-call cost baseline yet, run the 9-Question AI Spend Audit first. The audit finds idle infrastructure waste, model-tier mismatch, and cache-miss costs that the vendor invoice does not break out on its own.

Signal 2. Data Residency or IP Boundaries Cannot Be Met by the Vendor

This signal fires when the vendor cannot clearly answer three questions: Where does my data go during inference? Who owns the outputs? Can I guarantee data stays inside a specific jurisdiction? For regulated industries like financial services, healthcare, and government contracting, these are not theoretical questions. They are legal and contract requirements.

The NIST AI Risk Management Framework (GOVERN function, at airc.nist.gov/RMF/1) covers supply-chain risk and vendor dependency controls. The EU AI Act (Regulation 2024/1689, at eur-lex.europa.eu) sets rules for third-party provider dependencies in high-risk AI systems. ISO/IEC 42001:2023 (at iso.org/standard/81230.html) covers supplier management and AI system lifecycle requirements around model provenance and update control. None of these frameworks tell you to self-host. They do require you to document how you control the vendor dependency. If the vendor cannot satisfy criterion 3 (data sensitivity and residency) in the Build vs Buy Framework, self-hosting is the engineering answer, not a personal preference.

Output IP ownership is a separate problem that often gets ignored. Some vendor terms leave it unclear who owns the model's outputs, or allow the vendor to use them. For companies that sell the outputs themselves (generated content, structured reports, trained classifiers), this is a real business risk, not a footnote.

Signal 3. The Vendor Update Cadence Is Breaking Your Outputs

This is the most overlooked signal. Vendor model updates change how the model produces tokens, and they happen without warning. A JSON parser, an entity extractor, or a downstream classifier tuned against one model version can break quietly when the vendor rolls out a new checkpoint. The breakage is often slow. Outputs that were 98% parseable might drop to 87%. You find out three weeks later when a support ticket arrives.

Self-hosting gives you control over when updates happen. You run an eval suite before any update goes live. You roll back if the eval score drops. This connects directly to criterion 7 (regulatory and audit) in the Build vs Buy Framework and to audit criterion 7 (model-update cadence and rollback) in the 10-Point AI Vendor Audit. If your downstream system is sensitive to how the model produces tokens, losing the ability to roll back is a real operational risk, not a minor inconvenience.

Signal 4. In-House ML Talent Can Absorb the Maintenance Burden

This signal is required, but it is not enough on its own. A distilled 7B model needs engineers who can re-tune it when the task distribution drifts, keep eval coverage current, and run rollback paths. "We can hire for it" is not a yes. The real question is whether a specific engineer on the current team has run fine-tuning workflows before and has time set aside for ongoing maintenance. Build vs Buy criterion 4 (in-house ML talent) is the right scoring tool. If it scores "Favors Vendor," distillation is not viable, no matter how well cost and data signals score.

Maintenance has three concrete parts: re-tuning when the task distribution drifts, eval coverage that catches regressions before they reach production, and a documented rollback path back to the vendor API if something fails. A self-hosted model without all three is more fragile than the vendor dependency it replaced.

What Distillation Actually Produces (and What It Does Not)

Distillation produces a model that handles the specific tasks it was trained on. It does not produce a replacement for a frontier model. This matters for the decision. If your use case is a well-defined structured-output task (classification, entity extraction, templated generation, schema-constrained JSON), a distilled 7B model can do it with speed and cost that a vendor API cannot match at volume. If your use case needs broad reasoning, cross-domain generalization, or task types that change often, a distilled model will fail on cases it never saw in training.

The evidence from sincllm's documented distillation of Claude Haiku into Qwen-7B shows this boundary clearly. That distillation targeted a specific structured-output task class. It did not try to replace frontier reasoning or cross-domain generalization. That is the right scope for a 7B distillation project. The project documenting replacing a vendor API with a fine-tuned 7B model in production also shows this. Production replacement at the 7B scale works when the task distribution is well-defined and eval coverage is in place before cutover. Neither project claims the distilled model equals the frontier model. Both show production viability within a bounded task scope.

The honest engineering frame is this: distillation is a narrowing, not a cloning. You are not replacing the vendor model. You are replacing the vendor's handling of one specific task distribution with a model you own. Your model handles that distribution with similar or better accuracy, at lower cost, with no external dependency. The remaining use cases outside that task distribution either stay with the vendor or are out of scope.

The 10-Criterion Build vs Buy Matrix Applied to Self-Hosting

The table below maps all 10 criteria from the AI Build vs Buy Framework to how they typically score for distillation versus staying vendor. Four criteria push toward self-hosting. Four typically favor staying vendor. Two depend on context. The table shows the overall shape of the decision. The full scored matrix for your specific system is in the downloadable framework.

Four criteria (C3, C4, C5, C7) decide whether distillation is viable. If any of the first three score "Favors Vendor," and the fourth (ML talent) scores "not viable," distillation is not ready, no matter what the other six criteria say.

What the Engineering Setup Looks Like

This section is a checklist for before you commit, not a tutorial. Three engineering decisions must be settled before starting a distillation project.

Inference stack. Self-hosting runs on either local GPU hardware or a cloud GPU instance. Local hardware costs less per token at steady volume, but you must buy it, maintain it, and plan for failures. A cloud GPU instance (on-demand or reserved) costs more per token, but it has less operational overhead and gets you to first inference faster. For an initial distillation proof, a cloud GPU instance is usually the right place to start. The evidence from running a local LLM as a production website backend shows that local inference can work in production, but the infrastructure choices are not simple and must meet your latency target before you cut over.

Eval coverage before cutover. Sending a distilled model to production without an eval gate is the top failure mode for self-hosting projects. The eval suite must cover every task type the distilled model will handle. Set pass/fail thresholds before any inference traffic is routed. Incident Readiness control 8 (eval coverage) from the 12-Control Audit is the right checklist for this gate. Do not cut over until the eval suite passes at your production quality threshold.

Rollback path. Keep the vendor API live as a fallback during the transition and for the first 90 days of production. The rollback path is not an emergency measure; it is a planned operational state. The routing logic (what percent of traffic goes to the distilled model, what percent goes to the vendor API) should be controlled by a feature flag, not a code deploy. This matches the kill-switch and rollback controls in incident readiness planning.

When Self-Hosting Is the Wrong Call

Three cases make self-hosting the wrong choice, no matter how cost or data signals score.

No in-house ML talent. If no one on the team has run fine-tuning or distillation workflows before, the project will stall in training, or produce a model that cannot be maintained when the task distribution drifts. Build vs Buy criterion 4 is not optional. "We will hire for it" is a project risk, not a current capability.

Payback period exceeds 12 months. A distillation project that does not reach cost-parity with the vendor API within 12 months of cutover is a losing investment in most cases. ML engineer time, hardware or cloud GPU spend, and eval infrastructure all cost money upfront. If your volume does not justify that cost within 12 months, the total cost comparison does not support the decision.

Task distribution is too broad for fine-tuning scope. A distilled 7B model works when the task distribution is stable and narrow. If the product needs the model to handle many different task types, domain switches, or prompt patterns that change often, the distilled model will need constant re-training. At that point, the maintenance cost exceeds the cost and control benefits of self-hosting. Stay vendor and put the effort into better API cost management and contract terms.

Getting this wrong costs more than a failed project. A fragile self-hosted model with no fallback path and no eval coverage is worse than the vendor dependency it replaced. You have added hardware risk, lost the vendor's model quality improvements, and built a system that breaks quietly when the task distribution drifts. At least the vendor dependency had a clear failure mode and a support contract.

The Decision Framework (Download)

The four signals and the 10-criterion matrix in this article show you the shape of the decision. Scoring your own system against all 10 criteria needs the full framework. It includes the complete scoring rubric, the load-bearing criteria thresholds, and a one-page matrix you can use in a board presentation. If you are at the threshold and want a production engineer to stress-test the decision before you commit resources, you can book a 30-minute production review after you download the framework.