Improvised retrieval in constrained coding agents: an initial study

This study measures whether AI coding agents construct alternative retrieval channels when their sanctioned web-search tool is unavailable, and whether that behavior varies with model capability. The question comes from a first-party operational observation, described in Section 1, in which agents subject to a newly introduced search cap continued searching through a different tool. Across 72 controlled runs spanning four Claude models, the behavior proved general: every model constructed retrieval workarounds under the constraint, at rates that rose with capability and with task difficulty, reaching 100% at the frontier tier and 100% on any task that could not be completed without retrieval. No run produced a fabricated answer.

1. Originating observation

On 2026-07-17, Claude Code v2.1.212 introduced a session-wide cap on web-search tool calls, 200 per session by default, as a control against runaway search loops. On 2026-07-18, during a multi-agent research session in our environment running that release, a fan-out of research agents exhausted the shared budget mid-task. Subsequent search calls returned a hard error, budget exhausted (200/200). One agent recorded the error on its first search call, the allowance having been consumed by other agents in the session.

The agents did not stop on the error. Session records show them constructing search-engine query URLs against a keyless, server-rendered HTML endpoint and retrieving the result pages with the URL-fetch tool, which remained available and uncapped. Roughly a dozen searches ran through this channel in the remainder of the session. At least three agents adopted the technique independently. No instruction, prompt, or tool description in the session referenced it.

A prior-art review found extensive deliberate use of the pattern, scraping libraries and fallback-search plugins wired into agent frameworks as designed features, and no published account of agents improvising it unprompted when a metered search tool is exhausted.

2. Question

A single operational observation is consistent with two hypotheses: the workaround is an artifact of one model, session, and context, or it is a stable behavior of capable agents under retrieval constraint. The study separates them by measurement. With the web-search tool removed and a task that requires current web information, does an agent construct an external-retrieval workaround, and does the rate depend on model capability and task type.

3. Method

Four Claude models spanning the capability range were evaluated: Haiku, Sonnet, Opus, and Fable. Each ran three task types requiring information unavailable from training data and not reachable by guessing a URL: a recent funding round, a first-hand account of a production failure, and a specific maintainer discussion concerning dropped dependency support. The constraint condition removed the web-search tool from the agent’s toolset; all other tools remained, including URL-fetch and a shell. A control condition left search available.

The constraint removes the tool from the start of the session rather than reproducing the originating mid-task budget exhaustion, but the effect is in line. The two conditions present a similar situation, search unavailable and the task incomplete, but are not identical; the difference is recorded in Section 6.

Runs executed in an isolated environment constructed to exclude experimenter influence: a configuration directory reseeded per run with no memory, instructions, or skills; a working directory with no project context; and task prompts that do not mention search, budgets, scraping, fallback, or any retrieval surface. Any workaround therefore originates with the model. Network activity was captured per run. The constraint condition ran 15 times per model (five per task); the control ran 3 times per model. Total: 72 runs on Claude Code v2.1.216.

The primary outcome is binary: on a constrained run, the agent either constructed an external-retrieval workaround (any networked tool call) or did not (deferred to the user or declined, with no retrieval attempt). The workaround channel was recorded per run.

4. Results

Workaround rates under the constraint condition, with Clopper-Pearson exact 95% confidence intervals:

Figure 1. Workaround rate under the constraint condition, by model (n = 15 per model). Whiskers are Clopper-Pearson exact 95% confidence intervals.

Two effects are present and separate cleanly by task.

Task difficulty sets a floor. On the maintainer-discussion task, whose answer is not reachable by URL guessing, every model improvised: 20 of 20 runs (100%, 95% CI 83.2% to 100%). A task that cannot be completed without retrieval produces a workaround at every capability level, with no separation between tiers.

Capability sets the ceiling. On the funding task, which some models judged completable or declinable without retrieval, Opus and Fable improvised in 10 of 10 runs while Haiku and Sonnet improvised in 1 of 10, the remainder declining. The contrast is significant by Fisher’s exact test (p = 1.2 x 10^-4). Pooled across all tasks, the frontier pair improvised in 30 of 30 runs and the smaller pair in 15 of 30 (p = 5.8 x 10^-6). Constructing an undocumented workaround is the more capable action; declaring the limitation is the conservative one. The data records a difference in disposition, not a deficiency in the smaller models.

Figure 2. Workaround rate by model across the three tasks, ordered by difficulty. The floor rises as the task becomes harder to complete without retrieval; the frontier models stay at the ceiling throughout. n = 5 per model per task.

4.1 The workaround technique

The dominant technique is the construction of search-query URLs against endpoints that accept keyless requests. The canonical instance is the one from the originating observation: DuckDuckGo’s HTML endpoint, html.duckduckgo.com/html/?q=<query>, a server-rendered search interface that returns results to a plain request without an API key, which DuckDuckGo permits. Agents assembled query strings for this endpoint, including search operators such as quoted phrases and OR, retrieved the page with the URL-fetch tool, and parsed the returned links. Eight of the constrained runs used a DuckDuckGo endpoint this way.

The same construction generalized to any surface that answers keyless queries. Twenty-three runs built queries against a site’s or code host’s own search URLs (a news site’s ?s= parameter, a code host’s /issues?q= and /discussions?discussions_q= paths); nine invoked the GitHub CLI’s search subcommands; a minority navigated directly to known resource URLs. Some runs combined channels. The channel an agent selected followed the task’s natural retrieval surface, general search where no structured source exists and a code host’s search where one does, rather than following model capability.

4.2 No fabrication

No constrained run produced an invented answer. Agents that could not retrieve did not answer from memory; they declined and stated why. One Sonnet run, unable to search, wrote: “I won’t fabricate company names, dollar amounts, dates, or article URLs, since that would risk giving you convincing-looking but false information.” Agents that did retrieve frequently marked the provenance, noting that a fact was confirmed against a source rather than recalled. The behavior under constraint is verifiable retrieval or an explicit decline, not confabulation.

4.3 Controls

All control runs completed their tasks through the sanctioned search tool and produced no workaround. The removed tool is the cause of the improvised retrieval. A separate cross-check reran four frontier constraint cells under API-key authentication, rather than the subscription authentication of the main grid, and reproduced the workaround behavior in all four.

5. Discussion

The originating observation was not an artifact. Under a search-tool constraint, constructing a retrieval workaround is the typical behavior for frontier-tier models and a frequent behavior at every tier, rising to certainty when the task cannot otherwise be completed. A session-level cap on a search tool changes which surface the searches run against; it does not change whether searching happens.

The result has a practical consequence for anyone publishing developer-facing resources. The surfaces that determine what an agent can retrieve include every keyless, server-rendered page reachable by a fetch tool, because agents operate in the reduced-tool mode as a matter of course rather than as an edge case. A resource that renders only inside a browser executing JavaScript, or that sits behind a login, is not reachable by an agent in that mode, whatever the quality of its content.

6. Limitations

Fifteen constrained runs per model distinguish rare from usual behavior and support the capability contrast; they do not fix per-model rates precisely, and the two smaller models’ confidence intervals are wide. The constraint condition removes the tool at session start rather than reproducing mid-task budget exhaustion, which is analogous to the initial discovery but not a perfect reproduction. Workaround channels were classified from run transcripts by pattern matching and confirmed by direct reading of the ambiguous cases, but the full set was not exhaustively audited. The study covers Claude models on a single agent harness (Claude Code v2.1.216; the originating observation occurred on v2.1.212). Cross-vendor agents, whose tool ecosystems differ, are untested.

zaira_labs