Google Research has released Agile Deliberation, a human-in-the-loop framework for building image classifiers around subjective concepts. Think ‘healthy food’, ‘inspiring artwork’ or clickbait in content moderation. Most tooling assumes users already know the concept boundary. Agile Deliberation assumes they do not, and helps them find it.
TL;DR
- Framework: It prompts Gemini 2.5 Pro and Gemini 2.5 Flash through the API, without fine-tuning.
- Runs on: A Jupyter or Colab notebook. Retrieval uses CLIP embeddings and a FAISS index on CPU or GPU. The demo index starts at about 260 MB.
- Performance: 10.5 F1 points above zero-shot Gemini 2.5 Flash on average, across 2 concepts and 18 live sessions.
- Best: Paid to play F1 of 0.59 vs 0.48 zero-shot, with precision up 22 points (0.35 to 0.57).
- Worst: Recall drops 17 points on paid to play (0.80 to 0.63) and 15 points on healthy food.
What is Agile Deliberation?
Agile Deliberation is a Google Research’s framework that turns a vague concept into a structured written definition. That definition does two jobs. It is a human-readable policy, and it is the text prompt for a VLM classifier. Each round keeps the candidate definition with the best F1 on the user’s labels.
The design came from interviews with a group of content moderation experts. Experts scoped a concept first, then refined it on borderline images. Finding those images was the slow part.
How does concept scoping work?
Stage 1 breaks the concept into at most 3 unit concepts, using prompt chaining with Gemini 2.5 Pro. Each unit gets candidate positive and negative subconcepts, shown with retrieved images. For healthy food, positives might be salads or grain bowls. Negatives might be processed food or raw, unprepared ingredients. The user keeps, flips or drops each one.
How does concept iteration find borderline images?
Stage 2 skips classic uncertainty sampling. The classifier is a prompted generative VLM, so its confidence scores are often uncalibrated. Instead, the system hunts for images that are borderline in meaning.
The pipeline runs 4 steps:
- Gemini 2.5 Flash writes borderline queries, such as ‘salads with heavy mayo dressings’. Each query retrieves 50 to 100 images.
- Images are de-duplicated, encoded, and grouped into clusters with dictionary learning.
- A UCB multi-armed bandit picks the next cluster. It favors clusters where users corrected the model or left rich feedback.
- A VLM summarizes each image in 1 sentence. The user sees at least 5 images that share 1 ambiguity.
Users label each image and add short notes on disagreements. An LLM turns those notes into candidate definitions. A greedy update keeps the best-scoring one.
How was it evaluated?
Subjective concepts have no fixed ground truth, so Google team skipped static benchmarks. It ran 18 live sessions of about 90 minutes each. 9 participants without professional moderation experience did 2 sessions each. One session used Agile Deliberation, the other used manual deliberation. Each participant spent about 45 minutes defining a concept, then labeled 200 held-out test images.
The two concepts were ‘paid to play’ clickbait and ‘healthy food’. Retrieval ran over a proprietary 100M-image dataset and WebLI, respectively.
What do the benchmarks show?
| F1 (Agile Deliberation group) | Zero-shot | Modeling Collaborator | Agile Deliberation | Gain vs zero-shot |
|---|---|---|---|---|
| Paid to play | 0.48 | 0.53 | 0.59 | +11 points |
| Healthy food | 0.48 | 0.50 | 0.58 | +10 points |
Source: Table 1 of the CVPR 2026 paper. F1 averaged over participants.
How does it compare with other subjective classifier methods?
| Feature | Agile Deliberation | Modeling Collaborator | Agile Modeling | Zero-shot VLM |
|---|---|---|---|---|
| Venue | CVPR 2026 | CVPR 2024 | ICCV 2023 | Baseline in paper |
| Core idea | Deliberation on semantic borderline images, then automatic prompt optimization | LLM agent decomposes the concept and auto-labels data | User labels images in a real-time loop | Classify from the initial description |
| Human effort | About 45 min of labeling and feedback | About 100 labels plus chat (vs about 2,000) | Under 30 min in a 14-person study | 1 concept description |
| Handles evolving definitions | Yes | No (static) | No (static) | No |
| Classifier | Prompted Gemini 2.5 Flash | Lightweight distilled classifier | Not disclosed | Prompted Gemini 2.5 Flash |
| F1, paid to play | 0.59 | 0.53 | Not tested | 0.48 |
| F1, healthy food | 0.58 | 0.50 | Not tested | 0.48 |
| Code | Apache 2.0, GitHub | Not disclosed | Not disclosed | Gemini API |
How can developers run it?
The repo ships a Demo.ipynb that walks through the healthy food example. It needs a Gemini API key and defaults to gemini-2.5-pro and gemini-2.5-flash. A setup script builds a FAISS index over LAION-400M CLIP embeddings. The default 2 shards take about 260 MB. All 100 shards take about 13 GB.
Key Takeaways
- Agile Deliberation helps users discover a subjective concept’s boundary, not just label it.
- It finds borderline images by meaning, not by classifier confidence.
- F1 rose 10.5 points over zero-shot Gemini 2.5 Flash on average.
- Users reported significantly lower effort and stress than with manual prompt editing.
- The code is open on GitHub, with a Gemini-based demo notebook.
Check out the paper, arXiv version, GitHub repo and Google Research blog. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers appeared first on MarkTechPost.
