I'm Nitin Kumar, Lead Project Coordinator at Debugged Pro. I help software agencies in India and the USA ship AI delivery work clients can defend in review. On October 2, 2026, coverage of the SIFT framework from MIT and Sakana AI caught my eye for one reason: cheaper eval loops.

VentureBeat reporting by Ben Dickson explains how Recursive Self-Improvement via Fast Tree-search inserts a cheap LLM pairwise judge before expensive benchmark runs.

The research paper on arXiv (Self Improvement via Fast Tree-search) lists authors Xinghong Fu (MIT) with Aravinth Kulanthaivelu and Yutaro Yamada (Sakana AI).

This guide sticks to verified facts from that coverage and paper. It turns them into delivery checks for India and USA agency teams tuning coding-agent harnesses.

SIFT framework pipeline diagram with Bradley-Terry ranking and priority queue — Feature image: SIFT pipeline figure from the arXiv paper. Credit: Fu, Kulanthaivelu, Yamada / arXiv:2609.19526.

What the SIFT framework actually does

The SIFT framework targets recursive self-improvement for coding agents. An agent can patch its own harness: prompts, tools, and control code.

Each patch needs a signal. Full benchmarks give a strong signal but burn CPU hours and API credits. Tiny task subsets are cheap but noisy.

SIFT's bet is simple. A separate language model can compare two agent implementations pairwise, even before a large benchmark finishes.

Those preferences feed a regularized Bradley-Terry model. The resulting ranks guide which parents to expand and which candidates get costly evals first.

Builds on DGM, compared with HGM

SIFT builds on the Darwin Godel Machine (DGM) harness. DGM keeps an archive of agents and grows evaluation sets during search.

The Huxley-Godel Machine (HGM) weighs a parent and its descendants when choosing where to search next. It also varies how many randomly sampled tasks score a candidate.

SIFT keeps the archive idea from DGM, then adds the LLM judge, Bradley-Terry ranking, and a fully disaggregated async search loop. Patch generation, judging, and benchmark evals can run in parallel.

How SIFT coding agents move through one search step

Think of one expansion as a short CI-style pipeline. Work keeps moving while longer tests run in the background.

Sample a parent using judge rank, accuracy rank, and a visit-count penalty so one lineage does not monopolize search.
Propose a patch to the agent harness. Paper cost notes put propose-or-expand near about $0.12 with their default settings.
Run a quick gate on four coding tasks to catch broken or useless changes early.
Pairwise judge the new agent against up to 10 high-ranking archive agents. The judge sees agent code, not benchmark tasks or scores.
Fit Bradley-Terry strengths from the win-loss matrix, then enqueue the candidate for larger evaluation by combined priority.
Keep expanding promising branches even while earlier candidates are still waiting on full evals.

A single pairwise judge call costs about $0.044 in the paper's breakdown. Up to 10 comparisons means about $0.44 of judging per new node. Evaluating on 50 Polyglot tasks costs about $6 and 2.6 CPU hours.

That gap is the whole point for agency delivery. You spend cents to decide whether a candidate deserves dollars of benchmark time.

SIFT framework asynchronous expand, judge, and eval timeline diagram — Body image: disaggregated SIFT search pipeline from the arXiv paper. Credit: Fu, Kulanthaivelu, Yamada / arXiv:2609.19526.

Verified Polyglot and efficiency numbers

I only quote numbers that appear in VentureBeat or the paper tables. Do not pad client decks with unverified scores.

o3-mini Polyglot run highlighted in coverage

SIFT with an LLM judge reached 35.1% accuracy on full Polyglot.
Wall-clock time was under five hours in the highlighted run VentureBeat cites.
Compute was about 42 CPU hours and about $150 in API credits for that run.
DGM under the matched o3-mini setup scored 30.7%.
A SIFT ablation without the judge scored 29.8%, which supports the judge's role in the gain.

The paper also lists other o3-mini SIFT repeats in a 32.0% to 35.6% range. That still beat the DGM baseline they report while using less search compute.

Qwen3-Coder-30B comparison

With Qwen3-Coder-30B as the coding model, SIFT slightly beat HGM while using about one third fewer CPU hours in the reported comparison. Paper table rows show SIFT at 31.1% with a Qwen3-480B judge and 32.0% with a gpt-5.4 judge, versus HGM at 30.5%.

Base agents without self-evolution sat much lower in those tables. For Qwen3-Coder-30B the base agent is listed at 20.0%. For o3-mini the base agent is listed at 14.2%.

TerminalBench and SWE-bench Verified subset

On TerminalBench 2.1, search-time accuracy alone misled selection. The judge's top pick scored 18 of 50 on the search subset versus 19 of 50 for the accuracy leader.

Across repeated full-benchmark runs, the judge pick averaged 36.7%. The accuracy leader averaged 28.1%. The judge spotted code issues a subset score missed, including a verifier gated off by default and a risky shell rewrite.

On a 60-task SWE-bench Verified subset, judge-selected agents averaged 50.4% and 53.8% across repeated full evaluations. No-judge top selections averaged 44.6% and 50.4%. The starting agent scored 40.0%.

Average task duration chart comparing LLM judge calls to full Polyglot task times — Body image: average task duration figure from the arXiv paper showing judge calls versus full task times. Credit: Fu, Kulanthaivelu, Yamada / arXiv:2609.19526.

Why LLM-as-a-judge plus Bradley-Terry helps

Pairwise preference calls are easier to calibrate than absolute scores out of 100. Hiring managers know this instinct when they compare two finalists instead of scoring one candidate in isolation.

Bradley-Terry turns those noisy wins and losses into one strength score per archive node. SIFT then uses ranks, not raw score magnitudes, so the search stays less sensitive to scale quirks.

If a candidate is still waiting for evaluation, it temporarily inherits its parent's accuracy for sampling. Visit counts discourage the search from hammering one favorite branch forever.

A concrete lineage from the paper

On Polyglot search, Node 9 added a short instruction to run tests after edits plus a test_runner tool. A later descendant added more machinery around test analysis.

Both scored 44% on the smaller search set, so that subset could not separate them. On full Polyglot, the simpler Node 9 scored 35.6% versus 33.8% for the more elaborate child. The judge had already ranked the simpler version higher.

That story matters for agencies. More tools are not always better. Cheap ranking can protect you from shipping ornate harnesses that look busy and score worse.

Agency delivery angle for India and USA teams

Clients will not ask for SIFT by name in every RFP. They will ask for cheaper harness tuning, self-improving coding agents, or internal Polyglot-style suites.

If your team already tracks agent runtime packaging, our DigitalOcean Agent Droplets agency guide covers a related delivery surface. SIFT sits one layer above: how you search harness changes without burning the eval budget.

When SIFT-style loops fit a statement of work

You already maintain a coding-agent harness and an archive of candidate versions.
You have a small rejection gate (SIFT uses four tasks) and a medium search set (SIFT uses Polyglot-50 style subsets).
API and CPU budgets are real constraints, not slideware.
Stakeholders accept that full benchmarks still confirm final winners.
Security review can sandbox agent code and block patches that loosen the eval harness.

Delivery checklist I would give a Monday standup

Name the coding model, expansion model, and judge model in the SOW. Do not hide three roles behind one line item.
Budget propose cost, pairwise judge cost, and medium-set eval cost separately using the paper's order-of-magnitude figures as planning anchors.
Require a four-task (or similar) smoke gate before any pairwise judging spend.
Log judge inputs as full files when possible. The paper found full-file pairwise views correlated better with accuracy than diff-only views.
Keep async queues visible to the delivery lead: expansion slots, judge backlog, eval priority queue.
Reject patches that edit harness or benchmark code. The authors blocked timeout and retry relaxations for a reason.
Split India execution (sandbox runs, log triage, harness diffs) from USA client communication (budget burn, go or no-go on full evals).

Teams that need broader build help can start from our software development consulting page or related notes on our AI services page.

Cost model you can put in an estimate

Use the paper's Table 1 style breakdown as a planning sheet, then replace numbers with your vendor rates.

Propose / expand: about $0.12 and a fraction of a CPU hour per node in their gpt-5-mini setting.
Pairwise judge: about $0.044 per call; up to 10 calls is about $0.44.
Polyglot-50 eval: about $6 and 2.6 CPU hours in their o3-mini setting.
Highlighted o3-mini SIFT run: under 5 hours wall-clock, 42 CPU hours, about $150 API.
Qwen3 SIFT runs: paper rows show roughly $34 API and about 7 hours wall-clock with a few hundred CPU hours depending on judge choice.

Quote a fixed discovery package when you can. Open-ended self-evolution without eval gates burns goodwill when API invoices arrive.

Also plan a tiered judge if cost pressure is high. On TerminalBench, a weaker judge kept most global ranking signal, while the stronger judge was more reliable among the top candidates.

What is missing from a production pitch

Be honest with clients about limits. There is no standalone public SIFT repository yet. SIFT sits on the DGM harness.

Downstream benchmarks still matter. The judge ranks candidates; it does not replace held-out confirmation.

Safety work is not optional. Recursive self-improvement can propose patches that loosen timeouts, retries, or harness constraints. Sandbox, allow-list writable files, and reject eval-gaming diffs.

SIFT was tested on coding agents. The same pattern may transfer to other agents with clear success metrics, but that transfer is an engineering claim you must prove on client tasks.

Two-week agency pilot shape

Agencies need a calendar, not a white paper. Use this shape for a low-risk SIFT-style spike on an internal coding-agent harness.

Days 1-2: Read the VentureBeat summary and the arXiv abstract plus methods cost table. Confirm DGM or equivalent harness availability.
Days 3-4: Stand up sandbox Docker runs, a 4-task gate, and a medium task set that mirrors client work.
Days 5-7: Wire pairwise LLM-as-a-judge calls and a Bradley-Terry fit on a tiny archive of 5 to 10 nodes.
Days 8-10: Turn on async expand versus eval queues. Cap total expansion steps and max judge comparisons per node.
Days 11-14: Hold out a final benchmark. Write a one-page client brief with cost, accuracy, and a go or no-go on larger search.

India pods can own sandbox ops and log review. USA leads should own budget caps and client expectation setting around recursive self-improvement.

Key Takeaways

The SIFT framework from MIT and Sakana AI inserts a cheap LLM pairwise judge before expensive coding-agent benchmark runs.
Bradley-Terry ranking plus async tree search lets expansion continue while full evals are still running.
On Polyglot with o3-mini, a highlighted SIFT run hit 35.1% versus DGM 30.7% and a no-judge SIFT run at 29.8%, with under 5 hours wall-clock and about $150 API in that report.
With Qwen3-Coder-30B, SIFT slightly beat HGM while using about one third fewer CPU hours in the reported comparison.
TerminalBench and SWE-bench Verified subset results also favored judge-guided selection in the paper's tables.
No standalone public SIFT repo yet; build on DGM-style harnesses and keep sandbox plus harness allow-lists in every agency SOW.

FAQ

These answers match VentureBeat coverage from October 2, 2026, and the arXiv paper Self Improvement via Fast Tree-search.

About the author: Nitin Kumar is Lead Project Coordinator at Debugged Pro. He works with software agencies across India and the USA on AI product delivery, coding-agent harness work, and client-ready documentation. He focuses on plain-language guides that help teams ship without overpromising.