Harness engineering
Wrapping an LLM in scaffolding so it runs reliably in production
awesome-ai-stack · concept slides
What it is
A single LLM call is smart but unstable on its own. The same prompt passes yesterday and misses today.
flowchart LR
subgraph harness [Harness]
loop[Agent loop]
loop --> tools["Tools·web"]
loop --> sandbox[Code sandbox]
loop -.-> guard[Guardrails]
loop --> obs[("Tracing·evals")]
end
task[Task] --> loop
loop --> model[(LLM)]
model --> loop
loop --> out[Result]The model is one part; the harness is everything else that makes it trustworthy.
Why it matters
The gap between demo and production is mostly the harness, not the model. These problems don’t yield to a bigger model — they’re structural.
| Common failure | Fixed by the harness |
|---|---|
| Runaway loop — endless retries | a bounded agent loop |
| Dangerous side effects — deleting files, arbitrary requests | code sandbox |
| Unsafe / off-topic output | guardrails |
| Confident wrong answers | evaluation · tracing |
| ”It worked yesterday” — silent quality drift | observability |
Orchestration — the spine that drives the cycle
The loop calls reason → act → observe in order, decides at each gate whether to stop, retry, or pass, and records every step to tracing.
flowchart LR
task["Task"] --> reason["Reason"]
reason --> tool["Tool call"]
tool --> obs["Observe"]
obs --> gate{"Over the cap?"}
gate -- no --> reason
gate -- yes --> stop["Stop · report"]
reason -.log.-> trace[("Tracing")]langgraph · openai-agents-sdk · crewai · agno
How to approach it
Don’t build it all at once. Add one layer at a time, in the order risk appears.
- Loop + model first — the simplest reason→act loop
- Running code? add a sandbox — isolate side effects
- Output reaching users? add guardrails — check before it lands
- Iterating? add evals + tracing — you can’t improve what you can’t measure