Why Do AI Agents Fail? Fix the Agent Harness or Upgrade the Model?

Why do AI agents keep failing at tasks even when the underlying language model seems capable? The answer is often not “buy a smarter LLM.” Missing API pages, truncated tool output, ambiguous tool schemas, unsafe retries, approvals and unsupported “done” claims can break an otherwise capable agent. An agent harness is the runtime layer that controls tools, context, state, permissions, recovery and verification. First identify whether the failure comes from reasoning or execution; then compare a model upgrade with a harness fix on the same tasks. This article combines published research with five deterministic mock-tool checks, not a fabricated commercial-agent benchmark.

Your model can answer coding questions, summarize documents, and explain a plan. Yet when an agent must paginate through APIs, edit files, handle a timeout, request permission, and prove that the final result is real, task completion becomes unpredictable. Should you pay for a stronger model or change the software that orchestrates its actions? The answer depends on where the failure occurred. Reasoning and interpretation are model problems; tool boundaries, state transitions, recovery, and independent verification are often harness problems. Optimizing the wrong layer may buy more tokens without buying a successful task.

On November 3, 2026, IBM Research published a striking comparison. With the same GLM-5.3-flash model and 90 AppWorld tasks, IBM reported 73 passes with CUGA (81.1%), 42 with Hermes (46.7%), and 27 with Deep Agents (30.0%). That is strong evidence that an execution harness can change what one fixed model accomplishes. It is not proof that CUGA wins all workloads, or that improving a harness always beats upgrading a model. IBM explicitly disclosed differences between configurations, a single benchmark, one attempt per task, and the absence of policy-enforcement tests in those runs. IBM Research, November 3, 2026.

This article is about the decision the benchmark cannot make for you: what should your team optimize first? XBSTACK already has a DeepSeek model and harness guide and an OpenAI Agents API versus Agents SDK comparison. Rather than reproduce those introductions, we examine the evidence for model–harness interaction, demonstrate five execution-control failures with a tiny reproducible script, and propose a 2×2 trial that distinguishes a model upgrade from a harness change. We label vendor evidence, independent research, our own small demonstration, and untested proposals separately.

What the IBM result does—and does not—measure

AppWorld is a benchmark of multi-step workflows across nine simulated apps and hundreds of APIs, including email, file storage, and payments. IBM evaluated 90 tasks across 30 scenarios under the same GLM-5.3-flash model. Every task was attempted once under each harness and graded using Harbor. The relevant comparison is not a hidden change of model weights. It is a change in the operating environment around those weights.

Figure 1 | The same model can follow different execution paths under different harness controls. Concept art, not an IBM benchmark trace.

Harness in this comparison	Task passes	Scenarios with all three variants passed	Important context
CUGA	73/90, 81.1%	20/30, 66.7%	AppWorld-specific decomposition, focused tools, pre-execution checks and reflection
Hermes	42/90, 46.7%	8/30, 26.7%	Default Harbor setup, AppWorld tools exposed through MCP
Deep Agents	27/90, 30.0%	2/30, 6.7%	Middleware for tool search, summaries, to-dos, retries and limits, plus two format-handling fixes

Task Goal Completion counts individual task successes. Scenario Goal Completion demands that all three variants of a scenario pass. The second metric matters because an agent that succeeds at one formulation but fails nearby variants may not be a robust executor. However, all three harnesses were configured differently. CUGA was optimized for AppWorld-style multi-app work; Hermes is better known as an always-on assistant, while Deep Agents targets long multi-step work in broader settings. The authors caution against interpreting the comparison as an overall ranking.

The mechanism is plausible and operationally important. An agent with access to hundreds of APIs may need only a few of them at one step. A harness that narrows the tool set, detects unfinished pagination, verifies a proposed state-changing action before execution, and returns a structured failure signal can make the same model’s next decision easier. But because the context, middleware and verification steps differ, the benchmark does not isolate one feature’s causal contribution. A reliable product decision would still require ablations: turn specific controls on and off while holding the rest constant.

There is also a security trap. CUGA documents runtime-enforced Tool Approval and ToolGuard mechanisms, but the 90-task evaluation did not run with policy enforcement enabled. A high completion percentage does not prove protection against unauthorized payments, customer-data exposure, prompt injection, or unsafe deletion. Functional completion, policy enforcement, and recoverability must each receive separate tests.

Other controlled studies complicate a simple “harness wins” story

Composio published approximately 1,800 harness, planning and connector trials. In its fixed-model DeepSeek V4 Pro experiment, harness success ranged from 30% to 60% across a 30-task suite. With GPT-6 Astra and 29 comparable cases, the six harnesses ranged from roughly 66% to 72%. These are substantial but very different spreads. Some reported cost gaps were large, but usage coverage differed across certain arms, and some cost values were estimates rather than vendor invoices. You should not import the largest headline cost multiplier into your own budget forecast without checking the methodology. Composio study.

A further August 2026 preprint, Same Model, Different Harness: Different Coding-Agent Results, comes closer to isolating a harness intervention: the authors held weights and tasks fixed, mechanically shortened older tool traces as context filled up, and detected recurring stalled work. In a 169-task tight-context SWE-bench Verified cohort (20,480-token budget and a 480-second per-task endpoint), fully solved tasks rose from 43 to 72. Under wide-context settings, some differences were small. This supports a conditional benefit from context and recovery management, not a guarantee that the same intervention wins on every workload. These are the preprint authors’ measurements, not an XBSTACK reproduction.

A separate paper, Finding the Right Fit: Model-Harness Interactions across Agent Tasks, evaluated 66 configurations spanning several harnesses and models. Its key finding is not that one harness always wins. Model ordering can change with the harness and task. On one Terminal-Bench 4 comparison, the relative performance of Claude and GPT reversed between OpenHands and PI. That is an interaction effect: a model’s ability to use tools, interpret errors, recover from a failed patch, and work with a particular interface changes the score of the combined system.

The strongest counterexample to a universal harness preference comes from Harness or Model? Isolating the Harness Effect in Agentic Coding. In the reported matched coding comparisons, the average observed advantage of native over neutral harnesses was small, with confidence intervals that did not establish a definitive winner. For GPT-5.5, the reported task pass rates were 55.6% versus 54.4%; for an Opus comparison, they were 48.8% versus 50.0%. That study also exposes important limitations in usage records, elapsed-time cutoffs and post-hoc task partitions. Different task families can favor different execution systems even when the overall average looks similar.

An October 2026 paper, When Agents Fail: Model or Harness—Who’s to Blame?, argues for measuring Valid Completion Rate independently from task correctness. An agent may terminate cleanly and produce a gradeable artifact that is wrong. Conversely, a run stopped by a time limit may already contain a passing patch. Treating every timeout, invalid artifact, unauthorized side effect and incorrect answer as one undifferentiated “failed run” destroys the information needed to decide what to fix.

These studies point to a more useful unit of evaluation: model × harness × task family × operating constraints. The model cannot be judged outside its execution environment, and neither can the harness. “Always optimize the harness first” is as simplistic as “always buy the strongest model.”

Diagnose the failing layer before replacing anything

Consider six recurring situations. When the agent cannot find the correct tool, start with tool discoverability, names, descriptions, parameter schemas and which capabilities are actually visible in context. When the agent stops at the first page of a paginated API, inspect pagination metadata, cursor handling and whether the harness detects a full page without follow-up. When a tool response was truncated, determine whether the model was told it was incomplete. These are interface and orchestration failures; a stronger model cannot infer records that were never made available.

Figure 3 | A failure-triage checklist. Diagnose the layer before upgrading the model or replacing the harness.

When a read request times out, a retry can be reasonable if it is safe, bounded and observable. The same rule cannot blindly govern a payment, order creation or production write: the remote side may have completed the operation even if the local request timed out. A robust harness must account for idempotency keys, observed remote state, transaction boundaries and the user’s authority to perform the action. Retrying is not merely a prompt preference.

When an agent proposes deleting a customer record, the model’s description of the user’s intent is not authorization. The trusted application boundary needs to check the resource, principal, action, scope, policy and approval event before executing. It should be impossible for a prompt injection or an enthusiastic model to treat a proposed action as approval.

When a coding agent says “fixed and deployed,” verify which files changed, which tests ran, whether the build passed, whether deployment actually completed, and whether the live route serves the intended content. A run can feel complete in chat while leaving the external system unchanged. This failure matters in publishing workflows as much as in coding.

Observed symptom	First layer to inspect	Objective evidence	When a model upgrade is justified
Wrong tool or argument	Tool search and typed schema	Correct tool exposed, validated arguments	Same interface still yields reasoning mistakes
Missing pages or truncated outputs	Pagination, cursor and result metadata	Complete response count and next-page state	Full evidence is available yet interpretation fails
Duplicate effects after timeout	Idempotency and recovery state	Replayed interrupts and remote-state checks	Safe transitions but bad action decisions persist
Unapproved write	Runtime authorization and human gate	Protected tool never executes without approval	Do not substitute a model upgrade for permission checks
Unverified “done” claim	Build/test/result verifier	Logs, diffs, output receipts, live status	Complete verification still identifies bad reasoning
Bad reasoning despite sound inputs	Prompt, task decomposition and model	Same evidence and task across candidate models	The model is likely a real bottleneck

Some “AI agent keeps failing” symptoms originate in the provider adapter rather than in either model reasoning or a sophisticated orchestration design. In its November 2026 agent model invocation guidance, Volcano Engine documents multi-turn degradation when required reasoning state is not returned to the provider, truncated outputs when token limits are too low, and unstable cache behavior when model or tool configuration changes within a session. Diagnose these with the provider-specific API contract and raw request/response traces before replacing a harness or upgrading the model. Reasoning-state transport differs across providers and has privacy implications; do not copy a vendor’s settings indiscriminately. Anthropic’s long-running harness design report also describes why compaction alone may fail to prevent premature wrapping-up and why explicit handoffs or context resets can help. These are attributed engineering reports, not XBSTACK cross-provider benchmarks.

A small XBSTACK replay: five checks, no model calls

To turn the engineering distinction into something executable, I added a minimal script to the XBSTACK workspace: experiments/agent-harness-vs-model-20261104/harness-replay.mjs. It takes five scripted task intents with deterministic mock tool responses. A baseline path accepts the model-proposed action or first result immediately; a guarded path adds specific checks. Both are graded by the same assertions.

This is deliberately not an LLM benchmark. It does not call any model API. It does not install or evaluate CUGA, Hermes, Deep Agents, Codex or Claude Code. It does not measure latency, token use, memory, safety in production or model accuracy. The only claim it can substantiate is that the chosen harness controls have a visible, reproducible effect on these five synthetic conditions.

The first condition provides three invoice identifiers over two pages. Without cursor follow-up, the baseline reports two; the guarded path collects all three. The second presents a deletion call without user approval. The baseline executes the simulated side effect; the guarded path requires approval and does not execute. The third makes the first read-only request time out and the second succeed. Only the guarded path performs the bounded safe retry. This says nothing about retrying payment or write operations.

The fourth returns truncated:true with an incomplete result. The guarded path fetches a complete response; the baseline treats the excerpt as final. The fifth supplies a drafted 99% success claim with evidence for only 93%. The guarded path returns the supported figure with its evidence reference, rather than authoritatively restating an unsupported number.

node experiments/agent-harness-vs-model-20261104/harness-replay.mjs
# {"cases":5,"baselinePassed":0,"guardedPassed":5}
# All assertions passed. Read replay-results.json for per-case evidence and limits.

Do not cite “0/5 versus 5/5” as the success rate of real agents. These are five intentionally selected fixtures for which the guarded implementation was explicitly written to meet the test’s contract. There is no statistical sample, model reasoning, or independent task selection. The replay is useful because it converts vague recommendations—“handle errors better,” “respect permissions,” “verify completion”—into executable assertions. It is not useful for predicting which commercial harness or foundation model will win your tasks.

A website operations example clarifies why the distinction matters. A search engine accepting a submitted URL is not the same as indexing it. A rich-text editor holding an article draft is not equivalent to a published public page. A successful build does not prove all public routes work. These are facts that can be checked by dedicated systems, rather than inferred from a model’s fluent final response. A harness that requires those receipts reduces false completion claims. But the local synthetic replay does not prove that any particular production website has implemented the entire chain.

The experiment that can answer the actual business question

To compare harness optimization with a model upgrade, treat them as separate experimental factors. Choose your current model A and a candidate model B; your current harness H0 and a revised harness H1. Evaluate four combinations: A/H0, A/H1, B/H0 and B/H1. Keep the task pool, initial repository or database snapshot, available tools, authorization, budgets, deadlines and grading rules as consistent as possible. Publish any unavoidable differences.

Figure 4 | The diagram uses “Valid Completion Rate” colloquially for independently validated outcomes. It is NOT the formal paper definition of VCR (normal, evaluable completion irrespective of correctness); report the two metrics separately.

Figure 2 | A proposed 2x2 controlled evaluation, not completed cross-model experimental results.

Cell	Model	Harness	Comparison enabled
A/H0	Current	Current	Baseline
A/H1	Current	Revised	Harness effect at fixed model
B/H0	Candidate	Current	Model-upgrade effect at fixed harness
B/H1	Candidate	Revised	Combined effect and model–harness interaction

Build the task pool around your real workload. For an engineering organization, include a repository change that must pass tests, an API workflow with pagination and a controlled write, and an interrupted multi-step task that resumes safely. For a local RAG product, include evidence retrieval, citation fidelity, deletion or revocation, and explicit cloud-data egress conditions. For an operations assistant, include a draft-versus-published distinction, a final verified external state and an authorized-side-effect check.

Each arm should run several times using frozen version identifiers and logged seeds or replay conditions where available. Do not tune H1 repeatedly against your final evaluation tasks. Maintain held-out task families so improvements do not merely overfit a public suite. Compare correctness separately from Valid Completion Rate, unauthorized side effects, manual corrections, retries, elapsed time and cost. A good economic measure is cost per validated task—all relevant cost divided by the number of independently verified correct completions. The denominator must not include incomplete runs simply because they generated a convincing answer.

There is a second confounder: an evolving harness may win by using more test-time reasoning and feedback, not by learning a reusable strategy. Rethinking the Evaluation of Harness Evolution for Agents argues for matching feedback and inference budgets and checking generalization to unseen tasks. If the new harness runs five internal probes where the baseline runs none, that is a real operating-mode difference, but you must include the additional cost and avoid claiming a free design improvement.

Which harness changes are worth trying first?

Start with irreversible risk and false completion, not with the most fashionable orchestration package. A safe tool authorization gate should verify principal, resource, allowed action, approval scope and expiring permission before any protected call. The final result gate should distinguish generated text from independently observed state. A coding agent may propose a patch, but only a passing test can justify “tested”; a publisher may populate an editor, but only a public page and its live assets can justify “published.”

Next reduce ambiguity at the tool interface. Prefer smaller, well-scoped tools with typed parameters, explicit errors, retryability labels and pagination state. Returning an error code, partial-result flag, cursor and idempotency semantics gives the model a far more useful next step than “something went wrong.” A compact, relevant set of tools can also reduce the decision burden without changing model weights.

Then address context and long-run state. Keep durable task facts, user approvals, selected evidence and executed side effects in structured records rather than trusting a summary to preserve every distinction. Compress oversized logs and tool results in a way that retains references to full artifacts. Make cancellation and resumption explicit: an interrupted network call may have executed remotely, and a saved checkpoint may reflect a different state than the external system.

Only after those foundations should you invest heavily in subagent architectures, automatic skill selection and self-modifying harness configurations. Otherwise each new component creates another failure boundary and another place to misattribute success or cost.

Cases where changing the model is the correct investment

Harness work is not a universal remedy. If the agent sees the correct repository files, tool descriptions, test failures and constraints, yet repeatedly produces logically invalid patches, a stronger reasoning model or better task decomposition may help. Evaluate that claim at a fixed harness and task input. The correct baseline is not a screenshot of one poor answer; it is a repeatable, independently scored task set.

The same is true for document intelligence. Suppose retrieval, chunk selection, citation references and evidence boundaries are stable, but a model still misreads exclusions or makes inconsistent conclusions from the supplied text. That may be a generation-model limitation. If the retrieval system omitted the governing clause in the first place, swapping generation models will not restore missing evidence. A local-first app should therefore distinguish recall failure, reranking failure, evidence-selection failure and generation failure before deciding which layer to upgrade.

Finally, an already efficient and reliable harness can become a ceiling: adding more middleware and extra model calls may increase latency and operating expense with little improvement. The empirical question is whether the next dollar spent on model capability or on harness engineering yields more verified correct outcomes for your own task mix. The answer may change after either component improves.

Self-improving harnesses are promising—and easy to overclaim

Google Research’s Harness² proposes refining a harness within a task using Propose–Probe–Compose, then carrying lessons forward to future tasks. Its public repository includes executable research code and benchmark support. Reported improvements on the evaluated professional tasks suggest a meaningful research direction, not a guarantee that adaptive harnesses generalize to unseen business environments. More probing also consumes compute, may expose new permission risks, and requires held-out verification.

Another direction, JevAgentHarness, delegates certain decisions to a smaller specialized decision model and reserves difficult cases for a larger model. The paper reports savings under particular task and confidence-threshold conditions. If an incorrectly confident router triggers a privileged action, however, the cost of failure can outweigh inference savings. The runtime still needs guardrails that are enforced outside both models.

These are reasons to watch the next generation of harness design, not reasons to add another half-maintained tool page or an untested production dependency. The durable trend is toward selecting the right model, context, control point and verification procedure for each task—and measuring the combination rather than any component in isolation.

The decision: fix the failing layer, then compare

If your failures are missing pagination, truncated results, ambiguous tool schemas, unsafe retries, weak approval boundaries, lost state or unsupported “done” claims, optimize the harness first. If the evidence, tool use, recovery and verifiers already work but reasoning remains persistently incorrect, compare models at a fixed harness. If both dimensions are uncertain, run the 2×2 matrix rather than changing both and crediting a single factor.

The IBM AppWorld result is important precisely because a fixed model generated very different task outcomes under different configurations. Independent research shows why that is not a universal ranking. XBSTACK’s five deterministic checks demonstrate the behavior of a few concrete controls; we have not run a real cross-model A/B, a live CUGA/Hermes/Deep Agents comparison, or a production-security audit. That boundary is part of the conclusion. A useful agent is not the one that produces the most confident closing message—it is the one whose actions, permissions, recoveries and outputs can be verified against the user’s actual goal.

Sources

• IBM Research: Same Model, Different Harness (November 3, 2026)
• Composio: Harness and connector choices across 1,800 trials
• Same Model, Different Harness: Different Coding-Agent Results
• Finding the Right Fit: Model-Harness Interactions across Agent Tasks
• When Agents Fail: Model or Harness—Who’s to Blame?
• Harness or Model? A Contamination-Controlled Coding-Agent Study
• Rethinking the Evaluation of Harness Evolution for Agents
• Harness² by Google Research
• OpenAI: Harness Engineering
• XBSTACK local reproducible experiment: experiments/agent-harness-vs-model-20261104/harness-replay.mjs (workspace-only asset without a verified public URL; not presented as a clickable reader resource)

Related reading

• DeepSeek V4 Pro and DeepSeek Harness
• Agents API vs Agents SDK vs Responses API
• Agent tool authorization and policy gate
• LangGraph checkpointer storage and recovery

Topic path / AI Agents

Continue from one agent pattern to the complete production system

The AI Agent hub organizes architecture, memory, tool use, evaluation, security, deployment and multi-agent coordination into a single learning path.

Open AI Agent hub → Review the architecture guide Subscribe