Skip to content
sinc-LLMBook a 30-minute call

Engineering reference

From Haiku to a Local Model: Prompt Scatter With a 7B GGUF

A first-person account of moving prompt transformation to a local model, covering training data, model setup, and operating tradeoffs.

The services, methods, and examples described here do not establish an expected performance result for your workload.

Mario Alexandre  ·  March 26, 2026  ·  open-source local-model token-savings

I moved the scatter step from Haiku to a model on my own GPU. This article describes the training data, LoRA setup, GGUF export, and checks for the local workflow.

PropertyValue
Base modelQwen2.5-7B
Training approachLoRA fine-tuning
HardwareRTX 5090
GGUF size4.7GB
Inference enginellama.cpp
Inference routeLocal model
Training dataPrompt/JSON pairs
sinc JSON Output Format
x(t) = Σ x(nT) · sinc((t - nT) / T)

How I Generated the Training Data

The training pairs contain a raw prompt and a sinc JSON output. The described filtering checks JSON validity, all six fields, and the CONSTRAINTS field.

# Training data format (JSONL)
{"input": "fix the auth bug",
 "output": "{\"formula\": \"x(t) = ...\", \"T\": \"specification-axis\",
  \"fragments\": [{\"n\":0,\"t\":\"PERSONA\",\"x\":\"...\"},
  {\"n\":1,\"t\":\"CONTEXT\",\"x\":\"...\"},
  {\"n\":2,\"t\":\"DATA\",\"x\":\"...\"},
  {\"n\":3,\"t\":\"CONSTRAINTS\",\"x\":\"...\"},
  {\"n\":4,\"t\":\"FORMAT\",\"x\":\"...\"},
  {\"n\":5,\"t\":\"TASK\",\"x\":\"...\"}]}"}
# ... additional prompt/JSON pairs

I used Unsloth with Qwen2.5-7B, LoRA adapters, and 4-bit quantization on an RTX 5090.

# Simplified training config
model = FastLanguageModel.from_pretrained("Qwen/Qwen2.5-7B")
model = FastLanguageModel.get_peft_model(model,
    r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    max_seq_length=2048,
    num_train_epochs=3,
    per_device_train_batch_size=4,
)
trainer.train()  # Record training duration for the selected hardware

After training, I exported to GGUF using llama.cpp at Q5_K_M quantization. The file is described as 4.7GB. Review the exported model’s outputs for the task.

Output Checks for Local Scatter

Check local scatter output for valid JSON, all six fields, and task-specific constraints, especially for short prompts. Compare those checks with the API-based option.

The article notes that local constraints can be more generic. Review the proposed fields before using them with the main model.

Compare local hardware and operating costs with API token usage, scatter overhead, responses per prompt, and output quality checks for the same workload.

Local scattering uses a model on your own hardware. Include the hardware, operating, and quality checks when comparing it with an API-based scatter call.

The article describes a local model and a GitHub release. Leave a comment if you want the repository link and training data.

// Production AI Engineering

Build AI systems that hold up in production.

sinc-LLM designs, audits, and stabilises production AI infrastructure: from vendor evaluation and cost accountability to incident controls and MCP architecture.

See what we do →
Additional context

sinc-LLM About Blog Services

DOI: 10.5281/zenodo.19152668 · © 2026 sinc-LLM · Mario Alexandre