The First Peer-Reviewed AI Co-Scientist Gets Nearly Half Wrong. 10,000 Labs Use It Anyway.

Stanford's Biomni just became the first AI co-scientist to clear peer review, publishing in Science on July 9 — and its own benchmark score is 57%. More than 10,000 labs are already running it in production.

What actually got published

"Autonomous biomedical research with an artificial intelligence agent" (Huang et al., Science, July 9) is the peer-reviewed version of Biomni, a general-purpose biomedical research agent out of Jure Leskovec's lab at Stanford, built with Genentech, the Arc Institute, UCSF, and Princeton. It isn't a chatbot bolted onto a literature search — it reads papers, selects datasets and tools, writes and runs code, interprets results, and proposes next experiments, citing its sources and logging its own work as it goes. Across 25 biomedical subdomains, it wires together 150 specialized tools, 105 software packages, and 59 databases, and the paper reports it handling causal gene prioritization, drug repurposing, rare-disease diagnosis, microbiome analysis, and molecular cloning without task-specific retraining.

The number that matters: 57%

On its own held-out benchmark — 443 questions spanning those 25 subdomains — Biomni averaged 57% accuracy. That's a 30-percentage-point lead over general-purpose LLMs and a 25-point lead over specialized drug-discovery agents, plus a clear win over a 43%-accuracy programming-assistant baseline and a 6-to-12-point edge on the benchmark's harder reasoning subsets (per reporting on the published figures). Read one way, that's a commanding margin over every system anyone tested it against. Read the other way: the single most rigorously vetted AI co-scientist that exists today still gets nearly half its own test wrong.

Where it actually earns its keep

The more convincing number in the paper isn't an accuracy score — it's a time comparison. In one real-world case cited by Stanford Report, Biomni analyzed more than 450 files of glucose-monitoring, dietary, and activity data in 40 minutes; the researchers estimate the same analysis would have taken a human 60-plus hours. That's the pattern across the demonstrated tasks — wearable-data analysis, multi-omics integration, wet-lab gene-cloning design, protein thermal-stability optimization, even driving a liquid-handling workstation — mechanical, multi-step work that used to eat a researcher's week, compressed into an afternoon. A reinforcement- learning variant built on the same stack, R0-32B, reportedly improved from a 0.35 to a 0.67 score on an internal eval, enough to pass Claude 4 Sonnet on that same measure — a sign there's real headroom left in the 57%, not a ceiling.

10,000 labs, and no autonomy claim in sight

A prototype of Biomni is already, per Stanford, in use across more than 10,000 academic and industry labs — making it, by that measure, the most widely deployed AI co-scientist in biomedicine. Notably, nobody behind it is selling it as autonomous. Leskovec told Stanford Report that Biomni "will not replace humans," and that it frees scientists to focus on "the value of the scientist — ideation and judgment." Kexin Huang, the paper's first author and now CEO of a startup commercializing the tool, has been explicit that this is a research assistant, not a research replacement. That framing matches the benchmark: 57% is a strong showing for a generalist agent across 25 subdomains, and a genuinely bad number for anything meant to run unsupervised.

What this means for reproducible science

Biomni's own design gets one thing right that a lot of AI-for-science tools skip: every run logs full citations and a record of the steps it took, so a result can in principle be checked. That matters more, not less, once 10,000 labs are running a system that's right 57% of the time — at that scale and that error rate, independent verification isn't a nice-to-have bolted on afterward, it's the only thing standing between "the agent found something" and "the agent found something and someone confirmed it." It's the same conclusion we keep landing on in this newsletter from a different angle each week: the model doing the reasoning is no longer the bottleneck. Whether you can trust and reproduce what it hands back is.