NeuroArxiv: your agent should read the paper first

This had been bothering me for months. Every time I asked my coding agent to build something, it started every single piece from scratch. Cache layer: invented. Retrieval strategy: invented. Scheduling: invented. Confidently, and in one pass.

That is not how any of us built things before agents. There is a whole GitHub for the ordinary parts — app layout, API scaffolding, workflows, the standard algorithms. And there is a whole arXiv for the genuinely hard parts:

• how do you cut RAG fatigue?
• how do you raise an agent's response deliverability?
• which SLM architecture actually fits the constraint you have?

Nobody sane derives that from first principles at 2am. You read what already exists, you find the approach that survived contact with reviewers, and you build on it. Then an agent writes all of it itself instead, and you get the predictable result: laggy, brittle, quietly vulnerable, and rebuilt again next sprint.

So I took the way I actually research and turned it into a system-design engine. It is called NeuroArxiv, it is free, and it is open source.

What it does

Ask an agent to build something non-trivial, and before it writes a line:

PROBLEM
  │
  ▼
0. CATEGORIZE  — map the problem onto 3-5 arXiv categories + search terms
  │
  ▼
1. FETCH       — real HTTP against export.arxiv.org, category by category
  │               (no LLM call — deterministic, courtesy-rate-limited)
  ▼
2. DIVERGE     — one isolated read per paper, in parallel
  │               (each sees ONE abstract, never the others)
  ▼
3. SCORE       — relevance / practicality / rigor, per paper
   + CLUSTER   — group by underlying architectural angle
  │
  ▼
4. CONVERGE    — pick ONE cluster, synthesize, cite, name the first step,
                 name the risk, list pitfalls from EVERY paper's limitation

One line to install it:

npx github:UditAkhourii/neuroarxiv install

Restart Claude Code and /neuroarxiv "<problem>" is live. It drops the skill into ~/.claude/skills/neuroarxiv — no clone, no build step of your own.

Fetching is not divergence

The step people skip is step 2, and it is the whole point. Search hands you sources. If you then pour all of them into one context window and ask for a recommendation, the first abstract in the buffer frames how every later one gets read. You have not done a literature review; you have done one biased read with citations attached.

So each paper gets its own pass, seeing one abstract and none of the others. Only after every paper has been read on its own terms do the reads get scored and clustered by the architectural angle underneath them — not by keyword.

This is the same discipline as adhd, aimed at a different target. There it is candidate solutions that must not anchor each other; here it is sources. I wrote the general version of this argument up in Shared context is shared bias.

The eval, and why the headline is not a ratio

The easy eval is research versus no research, and it proves nothing. The real question is whether the isolate-then-converge discipline beats a plain, capable agent that already has web and arXiv access and just uses them sensibly.

Three conditions, same model, five previously untested cross-domain problems — decoherence mitigation in superconducting qubits, a memory-constrained sparse solver for 3D FEM, protein-ligand binding affinity from few labels, KV-cache compression without losing needle-in-haystack accuracy, and real-time regime-change detection under a low false-positive budget. Run independently, no visibility into each other.

Before scoring anything I pulled a sample of the web-search condition's citations and checked them against the real arXiv API. All of them were real and matched. That condition is not a strawman — it produced genuinely grounded answers across twenty real sources.

Then every transcript was re-read for one specific behaviour: does the answer name a source it just cited and flag a real limitation in that source's own claim — not a generic domain risk?

Zero, zero, seven.

It caught a withdrawn proof it had cited and declined to rely on it. It caught a benchmark result validated at only one context length and flagged that before recommending the approach. It noticed that neither change-point paper it cited reports an absolute false-positive rate — on a problem whose entire constraint was a false-positive budget.

I am reporting this instead of a ratio on purpose. A ratio between two scores on a bounded scale inflates whenever the denominator sits near the floor, which is the mistake I already wrote about. A behaviour that happened in five of five runs for one condition and zero of five for both others has nothing to round up.

Where it loses

On raw answer quality — specificity of the recommended mechanism, quality of the named risk — NeuroArxiv beat the web-search condition by roughly $1.1\times$ to $1.3\times$. Modest, not a dramatic multiple, and I would rather say so than pick the flattering framing.

It also lost on citation breadth in two of five problems. arXiv-only search casts a narrower net than general web search, and on the KV-cache and regime-change problems that cost it real, relevant sources. That is a direct consequence of a design decision, not a fluke: no PNAS, no PMC, no bioRxiv, no paywalled venue.

The eval itself is small and self-graded — n=5, one session, chosen and judged by me, not blind. Reads are abstract-only, so a recommendation is bounded by what an abstract can support. Treat it as a first data point.

The eval found a real bug

The useful part of running it across physics, applied math and quantitative biology instead of comfortable CS problems: the curated category taxonomy in src/categories.ts has no slot for quant-ph, cond-mat, q-bio, or even math.NA — mainstream numerical analysis, not a niche corner.

For three of the five problems the agent had to guess a category id from general arXiv knowledge, unverified by any tool. All three guesses happened to be right. There is currently no fallback if one is wrong — it would fail silently into an empty result set with no self-correction signal. That is an open issue, and it is filed as one.

When not to run it

The skill is expensive: a real fetch plus roughly one isolated read per paper, typically ten to twenty, plus scoring and convergence. It has a pre-flight gate that aborts unless three things hold — there is an actual technical mechanism to research, you are about to commit real effort to it, and you have not already decided on the approach.

Naming a variable, wiring a CRUD form, gluing two documented SDKs together: abort, build it directly. Designing a caching strategy, a coordination scheme, a retrieval approach, an inference technique, anything where the naive version breaks at scale: that is what it is for.

Try it

It is MIT, it is free, and everything above is in the repo — the full scorecard, the raw transcripts, and the skill itself if you want to read the prompts before you run them.

If you have architectural questions, or you want to argue about category coverage, eval design, or where the isolation discipline breaks down — the Discord is where that happens, and I answer there.