Socrates Review 2026: A Question-Only AI Advisor That Boosts ML Performance by 55.9%

What if the secret to better AI performance isn’t a bigger model or more compute, but a second AI whose only job is to ask questions? That’s the counterintuitive premise behind Socrates, a multi-agent protocol from Hexo Labs that was published at COLM 2026. Instead of giving the AI detailed instructions or letting a supervisor agent issue directives, Socrates pairs a tool-using “Scientist” agent with an advisor that is programmatically forbidden from giving answers, using tools, or making suggestions. All it can do is ask questions. The result? A staggering 55.9% average improvement across five MLE-bench Kaggle competition tasks compared to the same agent running solo.

This isn’t just another prompt engineering trick. The Socrates protocol enforces its constraint at the code level, and every experiment plan the Scientist proposes must receive explicit [APPROVED] from the Socrates advisor before execution begins. This hard gate creates a quality checkpoint that forces the Scientist to introspect, reconsider assumptions, and surface blind spots that a directive-based supervisor would simply steamroll past. It’s a rare example of how a well-designed constraint can produce better outcomes than unbounded freedom.

What Socrates Does

Socrates is an open-source protocol rather than a standalone application. It operates in two complementary modes. The Sequential scaffold handles single-agent, one-experiment-at-a-time workflows where per-step reasoning quality matters most — the Scientist proposes, Socrates questions, and after a configurable number of discussion rounds (defaulting to three), the gate unlocks. The Evolutionary scaffold, powered by MLevolve and Monte Carlo Graph Search (MCGS), runs high-volume parallel exploration across multiple solution branches, introducing paradigm-shift mutations and cross-branch fusion while Socrates gates each major direction change.

The architecture itself is thoughtfully asymmetric: Socrates maintains state across sessions, building a mental model of the Scientist’s progress over time, while the Scientist is deliberately stateless per episode. It reads from and writes to a shared environment — filesystem, experiment logs — but carries no persistent memory between runs. This design prevents the Scientist from reinforcing its own biases across experiments, a subtle but important safeguard.

Use Cases

An ML researcher competing on Kaggle can use the sequential scaffold to pressure-test feature engineering assumptions before submitting. A PhD student exploring novel neural architectures can let the evolutionary scaffold run 50 parallel branches with Socrates pruning dead-end mutations, preventing wasted GPU hours on unpromising directions. Agent framework developers can study the question-only constraint as a reusable design pattern, adapting the [APPROVED] gate for their own multi-agent code review or planning pipelines. MLOps teams can integrate Socratic review checkpoints into automated model retraining workflows, catching feature drift and data leakage before models reach staging. Educators can use Socrates’ question logs as teaching material, demonstrating how structured questioning uncovers hidden assumptions in experimental design.

Key Features

Question-Only Advisor Protocol

Socrates cannot give answers, issue directives, or use tools — it can only ask clarifying questions. This constraint is enforced programmatically, not via prompt engineering. The result is a genuinely different interaction pattern: instead of being told what to do, the Scientist must arrive at its own conclusions through structured introspection.

Mandatory [APPROVED] Gate

Every experiment plan passes through a hard checkpoint. Socrates must explicitly approve before execution, and the gate is only bypassable after a configurable number of discussion rounds. This prevents the Scientist from rushing into poorly-reasoned experiments and creates a natural quality control layer.

Dual Scaffold Architecture

Two execution modes serve different research workflows. The sequential scaffold prioritizes per-step reasoning quality for single-experiment pipelines. The evolutionary scaffold, built on MCGS tree search, enables parallel exploration with mutation operators and cross-branch fusion for high-volume experimentation.

MLE-bench Benchmark Integration

Evaluated across five real Kaggle tasks from OpenAI’s MLE-bench: Statoil Iceberg (radar imagery), Stanford COVID Vaccine (RNA degradation), Ventilator Pressure (time-series), NFL Contact Detection (player tracking), and Smartphone Decimeter (GPS positioning). Each ships with dataset loaders and submission pipelines.

Academic Research Provenance

The COLM 2026 publication includes full reproducibility artifacts: configuration files, statistical analysis tools, and a Baseline PI control condition (a generic encouragement agent) that isolates the effect of structured questioning from mere multi-agent presence. This isn’t a vibes-based claim — it’s peer-reviewed science.

Pricing

Socrates itself is free and open-source under the MIT license. The real cost is LLM API usage. The default model is Claude Opus 4 via Anthropic’s API, and a full 50-step evolutionary run can cost $10–$50 depending on the task. The sequential scaffold is lighter, typically requiring only about 30 steps. Users can swap in cheaper models via configuration flags, though performance may degrade. GPU is optional for three of the five benchmark tasks (COVID, Ventilator, and Smartphone run on CPU).

Common Questions

Can Socrates work with models other than Claude Opus 4? Yes, the model is configurable. However, the published results are all based on Claude Opus 4, and the paper hasn’t yet quantified how much the 55.9% improvement depends on model quality. Cheaper models may produce weaker Socratic questioning, reducing the protocol’s effectiveness.

Is this useful outside of Kaggle competitions? The protocol is theoretically agent-agnostic and domain-agnostic — any LLM agent facing a decision could benefit from structured questioning. But at seven GitHub stars and zero community contributions as of mid-2026, nobody has yet demonstrated the protocol working on non-MLE-bench tasks. The generalization question remains open.

How does this compare to simply asking the AI to review its own work? The paper includes a control condition — a generic encouragement agent — that shows multi-agent presence alone doesn’t drive the improvement. The constraint of question-only interaction matters. A self-review prompt can produce useful reflection, but it lacks the adversarial dynamic and the hard gate that make Socrates effective.

Verdict

Socrates is a genuinely novel contribution to AI agent design with strong empirical backing. The +55.9% average improvement is attention-grabbing, the question-only constraint is clever and well-motivated, and the COLM 2026 publication lends academic credibility. However, this is an early-stage research project — seven stars, single-digit community engagement, and tight scoping to MLE-bench tasks — so practical utility today is limited to ML researchers willing to invest in setup. If the protocol generalizes beyond Kaggle and the community grows, Socrates could become a standard pattern for multi-agent quality control. For now, it’s an intriguing idea worth watching and experimenting with if you’re deep in ML research.