Does the hub make an agent find the right code?

The harness

Three setups, one variable at a time

Every run is a headless claude -p session with Claude Sonnet, started in a throwaway git worktree pinned to a fixed commit. The prompt is identical across setups except for one sentence that tells the agent where the other repositories are — or that they are not on this machine.

1. 01

   Fresh, pinned, isolated

   Each run gets its own detached worktree at a fixed commit, removed when the run ends. Runs execute three at a time, interleaved by repetition so a slow hour hits every setup alike. Unrelated MCP servers on the machine are disabled in all setups.

2. 02

   No run can teach the next one

   In setup B every hub tool that writes — knowledge, memory, notes, tasks, session summaries, re-indexing — is disallowed. Every run starts from the same hub state, so repetitions are independent and a good answer cannot be cached for the run after it.

   read: search · grep · read · contextallowedwrite: store · notes · tasks · reindexdisallowed
3. 03

   Scored from the transcript, not by eye

   The run's stream-json transcript is kept. Cost and turns come from its result event; tool calls are counted by kind — hub calls, file searches (the Grep tool plus any rg, grep, find or git grep in a shell), and file reads. Recall is computed from the final answer only, as below.

The tasks

Real questions from a real product

The product is one system split across seven private repositories: a server, a web front end, internal tools, two generations of a client, an older upstream codebase, analysis tools and protocol documentation. It grew over many years and several teams, so the same idea often lives in four places under four names. The model has never seen this code.

// shape only — paths invented, the real ones are private
{ id: 'x-wire-crypto', kind: 'find',
  repo: '<docs repo>',     // where the agent starts
  gold: {
    server:   /net\/crypto\/\w+Cipher\.cs/,
    tools:    /codec\/\w+Packet\.cs/,
    docs:     /crypto\/\w+\.cs|protocol\.md/,
    upstream: /net\/Session\.cs/,
    notes:    /decoder\.py|cipher\.md/,
  } }

const hit = Object.entries(task.gold)
  .filter(([, re]) => re.test(finalAnswer))
recall = hit.length / Object.keys(task.gold).length

Results

Cross-repo questions: as good as “everything on disk”, for less

Three find tasks. Setup A shows the problem: from one repository the agent finds about a quarter of the answer. Putting all seven repositories on disk (AA) finds nearly all of it, but doubles the cost and the time. Setup B is the hub's job: get to AA's recall from A's footprint. In the latest run it did — 0.90 against 0.88, at 77% of the cost and 65% of the time.

Searches = Grep tool calls plus shell rg/grep/find/git grep. In B most lookups go through the hub instead (16.1 hub calls per run on average). Cost is what the CLI reports for the run, at list price.

How B got from 0.59 to 0.90

Every miss was read in its transcript, the cause fixed in the hub, merged, deployed, and the same three tasks run again. Nothing in the tasks, prompts or scoring changed between rounds.

Single-repo bug fixes: no difference, and we say so

On the six fix tasks the bug lives in the repository the agent starts in. Both setups changed the right files every time; the judge scored B's diffs slightly closer to the real fix, within noise. The hub is not what makes these tasks easier — it neither helps much nor gets in the way.

Retrieval

Underneath: how well the index ranks

The agent benchmark tests the whole loop. These test the search layer alone, with no model in the loop, and anyone can rerun them from benchmarks/.

Limits

Read these before quoting a number

1. Small n. Three find tasks and six fix tasks on one product. The numbers are evidence of direction, not a leaderboard.
2. One model, one agent. Claude Sonnet through Claude Code. Other agents read the same hooks and tools, but are not measured here.
3. The repositories are private. The agent runs cannot be reproduced outside the team; the retrieval benchmarks can.
4. Recall is strict. A correct explanation that does not name the file scores zero for that repository. Some of B's remaining misses are exactly that.
5. The hub was tuned on these tasks. Each round fixed a cause seen in their transcripts. The fixes are general — exact search, file listing — but fresh tasks are the honest next test.
6. Next-day memory is not measured yet. Hub writes are off in these runs so each one is independent. Whether yesterday's decisions and dead ends reach today's fresh session is the next benchmark.
7. Cost is list price for one run. Hub hosting and indexing are not included; on a small server the index costs no tokens to query.