An agent harness supplies the context, tools, memory, permissions, and control flow that turn a model response into action. Two applications using the same model can behave very differently because their harnesses expose different information, retry failures differently, or stop at different moments.
A benchmark that records only the final answer misses this system behavior. It cannot tell us whether the agent used an unsafe shortcut, received hidden help, repeated an expensive action, recovered from a transient failure, or produced the correct result by accident.
Start with a harness map
Document the components that can influence a run:
- Instructions: system prompts, role descriptions, examples, and dynamically assembled context.
- Tools: schemas, descriptions, permissions, timeouts, return formats, and side effects.
- State: conversation history, plans, scratch data, retrieved documents, checkpoints, and long-term memory.
- Control flow: routing, delegation, concurrency, budgets, termination rules, retries, and escalation.
- Environment: files, services, credentials, network access, and other state outside the harness.
- Human control: approvals, corrections, intervention points, and visibility into pending actions.
Version this map with the test results. A prompt change or a new retry rule is an experimental change even when the model remains constant.
Define success at three levels
1. Outcome
Did the task reach an acceptable final state? Evaluate correctness, completeness, user constraints, and side effects. For state-changing tasks, inspect the external state rather than trusting the agent’s summary.
2. Trajectory
Was the route to the outcome acceptable? Record tool calls, arguments, results, state transitions, handoffs, errors, retries, and approvals. A correct answer reached through forbidden data access is a failure.
3. Operation
Was the run practical? Measure time, model calls, tool calls, tokens or compute, human attention, and cleanup required. A harness that eventually succeeds after uncontrolled retries may be less useful than one that stops clearly.
Build a scorecard that resists one-number thinking
| Category | What to inspect | Example signal |
|---|---|---|
| Task result | Final artifact and resulting state | Requirements satisfied; tests pass |
| Instruction fidelity | User and system constraints | No prohibited action or dropped requirement |
| Tool fidelity | Tool choice and arguments | Valid calls; appropriate idempotency |
| State integrity | Memory and checkpoints | No stale, leaked, or corrupted state |
| Recovery | Response to injected failures | Bounded retry, safe stop, useful escalation |
| Observability | Ability to reconstruct the run | Linked events with stable identifiers |
| Human control | Approval and correction surfaces | Intervention occurs before material side effect |
| Efficiency | Time, calls, compute, attention | Cost distribution, not only average |
Teams may combine these categories for a release gate, but the individual values should remain visible. An aggregate score can hide a severe permission failure behind several small efficiency gains.
Preserve an event trace
A useful trace does not need to expose private chain-of-thought. It needs enough structured events to reconstruct observable behavior. Each event can include a run identifier, agent or component, event type, timestamp, input and output references, tool or model version, permission decision, error class, and parent event.
Store sensitive inputs by reference or controlled digest when raw content cannot be retained. The evaluation should still be able to answer which context was available, whether the correct artifact was used, and where the run diverged.
Test controlled failure, not only happy paths
Inject failures that a production harness will eventually encounter:
- A tool times out after accepting a request.
- A tool returns malformed, stale, partial, or contradictory data.
- A credential expires during a multi-step task.
- An agent receives an instruction that conflicts with its permissions.
- A delegated agent finishes late or returns no result.
- The run exceeds its time, call, or cost budget.
- The user changes or cancels the request while work is active.
Observe whether retries are bounded and idempotent, whether partial state remains consistent, and whether the user receives a truthful description of what did and did not happen.
Evaluate multi-agent systems as networks
Adding agents creates communication boundaries. Measure the information lost or distorted at each handoff, duplicated work, disagreements, waiting time, and the authority each agent inherits. More agents do not automatically mean more independent verification; they may share the same prompt, source, or failure mode.
For each role, ask whether it contributes unique capability or a verifiable check. Compare the multi-agent configuration with the simplest credible baseline, including a single model with tools. Complexity should earn its place through better outcomes, safety, or inspectability.
Separate deterministic checks from judgment
Use deterministic checks wherever possible: schema validation, unit tests, file diffs, policy rules, resource counts, and exact state assertions. Use human or model-based judgment for qualities that genuinely require it, such as clarity or usefulness.
When a model grades another model, version the grader and rubric, randomize presentation when appropriate, and retain examples of disagreement with human reviewers. A convenient judge is still part of the harness and can introduce its own bias.
A minimum reproducible harness report
Publish the harness map and versions, task suite, baselines, scoring rules, trace schema, failure injections, outcome distributions, cost distributions, representative trajectories, and known limitations. Include both successes and failures so readers can see what the aggregate values conceal.
The purpose of harness evaluation is not to prove that an agent looked intelligent. It is to determine whether the system behaved acceptably, for understandable reasons, under conditions that resemble real use.
Open questions
- What is the smallest trace that preserves accountability without collecting unnecessary private data?
- How should evaluation represent a helpful human intervention: harness failure, designed collaboration, or both?
- Which fault injections best predict real incidents?
- How do we compare harnesses when tool access and permission models differ substantially?