Gary Engineering Lab #002: Ornith V7 — 12 Measured Runs, Real TTFT, and a 160-Token Trap

Engineering log · October 11, 2026 · Test ID FPR-ORNITH-BENCH-V7-20261011 · Status: V7 single-task evidence collected, wider model qualification pending

Four locally installed Ornith 9B profiles entered the same controlled coding task. Each ran three sequential trials with the same prompt, context allocation, output-token limit, random seed, and deterministic scoring rules. The experiment exists because our previous V6 benchmark misclassified a valid solution, truncated stored responses, and reported a fabricated approximation of time to first token.

The question we tested

A Python condition uses if len(password) > 5:. It mistakenly rejects a password containing exactly five characters. A valid correction is if len(password) >= 5:. The scorer parses the final candidate as Python syntax and tests three boundaries: lengths 4 (reject), 5 (accept), and 6 (accept). Merely explaining the old defect does not count as providing executable code.

Bench hardware and controls

• Host: Ryzen 9 3950X, 96 GB RAM, RTX 5060 Ti with 16 GB VRAM, Windows 11 Pro.
• Model family: Ornith 1.5, 9B local Ollama profiles; four explicitly named variants.
• API: Ollama /api/generate with streamed events and structured terminal counters.
• Same for every run: num_ctx=65536, num_predict=160, temperature=0, seed=42.
• Evidence: complete raw response retained privately; terminal termination reason, model digest/quantization, prompt and decode token counts, native load/evaluation times, genuine streamed first-content latency, and GPU-board snapshots.

Results: correctness is not just speed

Ollama model	Executable fix	Output limit hit	First load	Warm median	Decode rate
fpr-ornith-9b-64k-gpu:latest	3/3 PASS	0/3	6.88 s	0.85 s	68.29 tok/s
ornith-1.5:9b	3/3 PASS	0/3	6.85 s	0.85 s	68.37 tok/s
ornith-1.5:9b-64k	0/3 completed fix	3/3	57.14 s	45.28 s	3.55 tok/s
ornith-1.5:9b-64k-cpu	0/3 completed fix	3/3	56.53 s	45.66 s	3.52 tok/s

Timing definitions: first load is full initial wall time; warm median excludes initial unloaded-model startup; decode rate comes from Ollama’s structured generated-token count and evaluation duration. Figures are descriptive of this single task, not universally representative throughput benchmarks.

Why the CPU results deserve a closer look

The existing 64K and 64K-CPU model profiles carry a CPU-only setting (num_gpu=0). At roughly 3.5 generated tokens per second, the two profiles spent substantially longer generating text. All six responses hit the configured num_predict=160 cap (done_reason=length). Their complete saved text described the right >= 5 concept, but never finished with the requested executable condition. These are INCOMPLETE_OUTPUT_BUDGET outcomes—not proof that CPU inference is incapable of the fix.

Separately, all six GPU-profile responses included the correct executable predicate but also unwanted explanation/thinking text. They passed functional correctness and failed a strict “only the line” output-format criterion. That distinction matters when using an LLM to generate code consumed by another system.

What changed from V6 to V7

1. Deterministic correctness: 14 scorer regression tests passed, including answers that quote the defective expression before giving the correct fix.
2. Complete evidence: no artificial 300-character truncation. The exact raw output is retained in private per-trial JSON, not pasted into a public blog.
3. Real timings: first-content latency is measured from streaming; token counts, model-load time, prompt evaluation, and decode duration come from Ollama’s API.
4. Provenance: per-model digest, options, quantization and SHA-256 evidence manifests were recorded and independently replayed without modifying V6 files.
5. Safety: no model downloads, production routing switches, server restarts, or intentional interruption of running workloads were performed.

What remains unverified

One coding condition is insufficient to recommend a permanent coding model. The test did not validate tool calling, complex debugging, 65K-token retrieval, multiturn agent autonomy, or energy per useful result. Whole-GPU readings are not process-level energy measurements. Next experiments must test diverse tasks and investigate how output budgets affect the CPU profiles.

Routing verdict: the GPU-capable Ornith profiles are promising for fast interactive work under this task. All production routing remains unchanged. CPU-only fallback still requires an explicit latency and completion policy. Agent orchestration, long-context quality, and energy efficiency remain UNVERIFIED.

This is the standard for the Gary Engineering Lab and our early FPR Mechanic project: benchmark claims are tied to saved evidence, and limitations are reported alongside successful measurements.