Perplexity AI Releases WANDR: An Open Benchmark Evaluating Research Agents That Must Search Wide And Deep

Research agents already handle real knowledge work today. Teams delegate competitive mapping, due diligence, and literature review to them. However, most benchmarks test a single answer, not large evidence-backed collections. Perplexity targets that gap with a new open benchmark.

Perplexity released WANDR (Wide ANd Deep Research). It is an open benchmark and evaluation harness. It is built around 500 realistic, challenging data-collection tasks for knowledge work. WANDR is the wide sibling of Perplexity’s DRACO benchmark for deep research. DRACO asks whether an agent produces an accurate, complete, objective long-form report. WANDR instead asks whether it can build a large collection with evidence.

What is WANDR

At its core, WANDR tests two demands together. Wide means discovering a large, often open-ended set of qualifying entities. Deep means investigating every entity enough to support each claim with evidence. Combining both changes the problem for agents. A few compelling examples are not enough here. A polished narrative built on incomplete research also falls short.

To capture this, WANDR uses a composable qualification key hierarchy. One task might request company(n) -> employee(m) -> url(k). This means n qualifying companies, m employees each, and k supporting pages each. Every complete path through the tree gets validated independently. The same structure can represent a flat list, nested search, or matrix.

A Concrete Task Example

To ground that hierarchy, consider the released ceo_cfo_appointments task. It asks for at least 70 US-based companies. Each must have a CEO or CFO appointment first announced between March 1 and April 30, 2026. For each, the agent supplies one authoritative appointment page. A subtask adds a listing-authority page per company. Together, the task requires 140 source-backed records.

Concretely, the two hierarchies and one submitted record look like this:

# Task hierarchies
company(70) -> company_appointee(1) -> url(1)   # 70 appointment records
company(70) -> url(1)                           # 70 listing records

# One record the grader re-fetches and re-checks (values are illustrative)
{
  "item":     "Example Corp - new CFO",
  "url":      "https://issuer.example.com/press/cfo-appointment",
  "excerpts": ["Example Corp today named Jane Doe as Chief Financial Officer, effective April 2026."],
  "answer":   "Jane Doe appointed CFO; announced April 2026"
}

Realistic Tasks, Generated At Scale

Beyond single examples, WANDR builds its tasks from real usage. It starts from de-identified patterns seen in production, not synthetic prompts. A semi-automated pipeline then turns those patterns into tasks. The pipeline runs four stages: seeding, authoring, admission, and curation. It uses an interleaved author-critic loop with mechanical linting.

As a result, the median task asks for 50 members and 245 records overall. Across all 500 tasks, WANDR calls for 170,495 source-backed records. Tasks split into 167 lower, 166 middle, and 167 higher difficulty. Difficulty depends on per-record work, not scale alone.

How WANDR Grades Agents

Unlike fixed answer keys, WANDR grades each claim against cited evidence. Every record contains an item, URL, selected excerpts, and answer. The grader re-fetches the page during evaluation. It checks whether the page is usable and in scope. It then verifies the excerpts truly appear and support every requirement.

These binary record verdicts then roll up through the hierarchy. Precision measures the quality of what a system submitted. Recall measures quality-adjusted completion, filling any shortfall with zeros. Soft scores give partial credit to incomplete members. Hard scores count only members whose full subtree is correct.

The Benchmark Results

Using that method, Perplexity ran six production systems on all 500 tasks. Its own Search as Code (SaC) system leads. Still, no system comes close to solving the benchmark.

With more effort, Perplexity reaches 0.447 soft F1 at the xhigh setting. Cost across settings spans more than four orders of magnitude. It ranges from $0.03 per task up to $324.83 per task.

Beyond the leaderboard, four findings stand out. First, partial progress is common, but complete coverage is not. Every system shows soft recall below soft precision. Second, scale compounds the problem sharply. Deeper hierarchies hurt most, since each branch adds a failure point. Third, discovery is the first structural bottleneck. Top-level discovery completion ranges from 0.611 to 0.951 across systems. Under-delivery, not duplicate merging, explains most missing volume. Fourth, finding a usable page is usually easy. Turning it into complete evidence is the hard part. For Perplexity, 41.4% of pages miss a substantive requirement. Also, 57.5% of excerpts fail to support the full claim. Its soft F1 falls from 0.531 under a retrieval-only check to 0.363 under the full verdict.

Notably, Search as Code fits this task shape well. An agent can express retrieval, filtering, fan-out, joins, deduplication, and stopping logic as a program. Deterministic compute then handles repeated operations outside the model context.

Use Cases With Examples

Practically, WANDR maps to jobs teams already automate. A market analyst needs every qualifying competitor, with matching evidence for each. A due-diligence team needs dozens of companies, then ownership, executives, and financing. Talent sourcing needs many candidates, each with supporting profile pages. WANDR tests exactly these wide-and-deep collection patterns at professional scale.

Because grading is per-record, teams can localize failures precisely. The score tree isolates loss to discovery, enrichment, or evidence extraction. This diagnosis helps engineers improve one weak stage at a time.

Key Takeaways

• WANDR is an open benchmark with 500 evidence-heavy, wide-and-deep tasks.
• Tasks use a qualification key hierarchy validated path by path.
• Grading is reference-free; the grader re-fetches and checks cited evidence.
• Perplexity Search as Code leads at 0.363 soft F1 and 0.133 hard F1.
• Discovery and complete evidence remain the biggest failure points.

Check out the Technical details and Repo here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Perplexity AI Releases WANDR: An Open Benchmark Evaluating Research Agents That Must Search Wide And Deep appeared first on MarkTechPost.