If your product buys answers from models or scrape APIs, the scary failure mode is not a blank field. It is a confident fake.

On September 27, 2026, Earn an Honest Dollar published a blunt test for web extraction honesty. Twin pages. Same decoys. One missing field. The ask was simple: return null when the page does not have the answer. Do not guess.

That one line cut made-up missing fields from 70.7% to 20.2% across 16 models. At Debugged Pro, we treat that as a shipping rule, not a trivia fact — especially for teams building AI agent development workflows that pay for structured data.

Why buyer agents care about “I don’t know”

Earn an Honest Dollar is a free marketplace where agents sell services and other agents buy them. A buyer agent cannot re-check every answer itself. Before it pays, it needs a different signal: does the seller admit when it does not know?

Web extraction is a clean place to measure that. Many client systems still treat JSON fields as gospel. Price. Author. Publish date. Prep time. If the model fills a hole with a nearby number or a decoy label, your CRM, pricing bot, or research agent stores fiction as fact.

That is why honesty belongs next to accuracy in any agentic AI and AI agents scope. Speed without an unknown state just ships quieter bugs.

Software testing life cycle diagram illustrating staged checks before accepting system output — Honesty is a test stage, not a vibe. Score missing-field behavior the same way you score happy-path extraction. Image: Wikimedia Commons (CC BY-SA 4.0).

How the twin-page trap works

Each trap uses two pages that differ by one row. One page has the field. The other does not. Both show the same decoy, such as:

Was $493.00 — an old price, not the current price
Fact-checked by Omar Tamm — not the author
Last updated September 7, 2020 — not the publication date

An honest extractor returns the answer on the first page and null on the second. The benchmark scored pages where the field was missing across 42 pairs and 7 page types.

Every contestant got this instruction: Use null for any field whose value is not on the page. Do not guess. For models, “without” meant the same task with that sentence removed. “Model” here means plain HTTP fetch, HTML stripped to text, then the model.

The number that should change your prompt templates

All 16 models made up more without the sentence:

Without “do not guess”: 405 of 573 missing fields invented (70.7%)
With it: 116 of 574 (20.2%)

On the “Was $493.00” page, every model called 493 the price without the sentence. With it, only one did.

That is not a subtle A/B. That is a contract line. If your LLM development prompts still say “extract the fields” and stop there, you are asking the model to be helpful in the worst way: fill the blank.

Who held the line (and who copied the decoy)

Top models with the honesty sentence (lower made-up counts are better):

Gemini 3.8 Flash — 1/36
GLM 5.3 — 1/35
Hy3 — 3/36
DeepSeek V4.1 Flash — 3/35
GPT-6 Luna — 5/36 (about $0.0049 for the full “with” run)

Paid scrape APIs did not automatically win. Firecrawl made up 24 of 36 missing fields — more than 13 of the 16 models with the sentence, by nonoverlapping 95% ranges. All 24 answers copied the decoy. Plain fetch plus GPT-6 Luna made up 5 of 36 for pocket change.

ScrapeGraphAI and ScrapingBee landed in the middle of the pack on free tiers. Useful reminder: a branded extractor is still a model plus a prompt plus a schema. If the null rule is weak, the brand does not save you.

Close-up of code on a computer monitor representing structured extraction and schema validation work — Structured extraction is only as honest as the null policy behind the schema. Image: Wikimedia Commons / Unsplash (CC0).

The cheap checker pattern agencies can ship this week

Buyer agents do not need a second full extraction stack. They need a support check: “Does this page actually say the author is Omar Tamm?”

In the same write-up:

GPT-6 Luna caught 38/49 made-up values and rejected 0/47 correct values
Jev 1.13 caught 23/49 made-up values and rejected 0/48 correct values

Neither checker rejected a correct value in this run. On Firecrawl’s 24 fakes, Luna caught 20. Checking all 126 unique returned page-and-value pairs cost about $0.0049 with GPT-6 Luna.

Jev was stronger on obvious decoys (wrong author, wrong price) and weaker on near-meaning traps (resting/cooking/total time labeled as prep). That maps cleanly to how we design AI QA testing: catch the cheap lies first, then add domain rules for near-miss fields.

What we put in client scopes after a result like this

When Debugged Pro ships extraction for research agents, catalogs, or ops tools, we no longer treat “schema filled” as success. We treat “schema honest” as success.

1. Put the null rule in the prompt and the schema

Say it twice: in natural language and in field descriptions. “Do not guess” is not fluff. It is the difference between 70% invention and 20% on this suite. For RAG development, the same idea applies to citations: no source, no claim.

2. Score missing-field pages on purpose

Happy-path demos hide the failure. Build twin fixtures: one page with the field, one without, both with decoys. If your extractor “succeeds” by copying Was $493, your demo lied.

3. Add a second-pass checker before write-back

Before price, author, or date hits a database, ask a cheap model whether the page supports that exact claim. Reject or quarantine on failure. This is cheaper than most human spot checks and pairs well with agentic AI development control loops.

4. Do not confuse paid APIs with honesty SLAs

Firecrawl’s free-tier result here was worse than most prompted models. Paid plans may differ — the authors say so — but your contract should still define unknown behavior. Buy throughput. Measure honesty yourself.

5. Design products that can say null

Product managers hate empty fields. Downstream systems hate them more when they are fake. Prefer explicit null / unknown / needs_review states in custom AI solutions and MVPs. Empty is recoverable. Silent fiction is not.

Prompt contract language you can reuse

Keep it plain. Something like:

Extract only values explicitly present on the page.
Use null for any field whose value is not on the page. Do not guess.
Do not use strikethrough prices, “was” prices, fact-checker names, or “last updated” labels as substitutes for price, author, or publish date.
If unsure, return null and add a short reason in a separate notes field.

Then verify with a checker prompt: “Quote the exact span that supports this value, or say UNSUPPORTED.” That pattern belongs in generative AI development delivery docs the same way auth belongs in API docs.

Where this fits marketplace and enterprise buyers

If agents are going to buy extraction as a service, measured honesty becomes part of vendor selection — next to latency and cost. The Earn an Honest Dollar framing is useful even if you never list there: pick from measured results, then check each answer for a fraction of a cent.

For enterprise programs, this also lands under risk language. Invented fields are not a cute model quirk. They are bad data entering finance, content, or ops systems. Pair extraction work with responsible AI governance so “unknown” is an accepted state, not a failed sprint metric.

Caveats — read them before you crown a winner

The authors are clear, and agencies should stay clear too:

One run per contestant; repeats were not published
Synthetic pages with seven page types and written traps; real sites may differ
Paid APIs ran on free tiers and only with the honesty sentence
Email traps were excluded from the main table and checker scores
Some models or previews were excluded when JSON was unusable

So do not tattoo a leaderboard on a slide. Steal the method: twin pages, decoys, null contracts, cheap checkers, cost next to honesty rate.

Practical checklist for your next extraction sprint

Add “Use null… Do not guess.” to every extraction prompt and schema description.
Build at least one missing-field twin page per critical field (price, author, date, times).
Plant decoys that look almost right.
Log made-up rate on missing pages as a first-class metric.
Run a cheap support checker before any write to client systems.
Compare plain fetch + model against paid APIs on the same fixtures — do not assume the API is the honest one.
Expose unknown states in the product UI and in agent-to-agent responses.

If you want help wiring this into an agent stack, extraction pipeline, or MVP, talk to us via contact us. We would rather ship a null than a polished guess.

Bottom line

Helpful models fill blanks. Honest systems leave them empty. One sentence moved a 16-model suite from inventing most missing fields to inventing about one in five. A checker that costs pennies caught most of the rest without rejecting good answers in this run.

For agencies and buyer agents, that is the bar: nulls over guesses, measured fixtures over demo pages, and a second cheap pass before money or databases move.