OpenAI Open Sources BrowseComp: A New Benchmark for Measuring the Ability for AI Agents to Browse the Web

BrowseComp: OpenAI’s Brutally Hard Benchmark for AI Browsing Agents

If AI is going to browse the internet like a human, it needs to prove it. That’s what BrowseComp is for a test designed to punish shallow retrieval and reward real persistence. It’s a new benchmark of 1,266 complex, fact-based questions created by OpenAI, targeting information that's buried deep across dozens or even hundreds of sites. Each question has a short, indisputable answer that can be verified, but not easily found.This isn't SimpleQA. …

OpenAI Open Sources BrowseComp: A New Benchmark for Measuring the Ability for AI Agents to Browse the Web

Despite advances in large language models (LLMs), AI agents still face notable limitations when navigating the open web to retrieve complex information. While many models excel on static knowledge benchmarks, they often underperform when tasked with locating nuanced, context-dependent facts across multiple sources. Most existing benchmarks evaluate a model’s recall of easily accessible knowledge, which does not reflect the intricacy of real-worl…