Benchmarking KnowledgeForge on GAIA: the Good, the Bad, and the Ugly

I'm building KnowledgeForge (KF) — a reasoning framework that sits on top of a base model and routes work to specialized modes: a Critic for finding gaps, a Debugger for diagnosing failures, a Strategist for trade-off decisions, a Builder for producing artifacts. The promise is easy to state and hard to prove. KF should make a capable model measurably better at the work where it skips steps, hides trade-offs, or over-engineers.

"Should" is doing a lot of work in that sentence. So I did the unglamorous thing: I built a benchmark designed to disprove my own framework, and I ran it.

This is the first post in what I expect to be a long series. It is not a victory lap. The headline result is a null — and I think publishing it is more useful than burying it.

How I set it up

The entire design rests on one principle: the only thing that may differ between conditions is KnowledgeForge. The model, tools, time and retry budgets, and scoring are identical across arms. I ran three arms:

• BARE — the raw model, no scaffolding.
• PLACEBO — the same model with a good, generic "think carefully, step by step, check your work" prompt. No KF-specific content.
• KF — the same model running under KnowledgeForge.

The placebo is the important one. Plenty of "our framework wins" demos are really just measuring "structured prompting beats no prompting." That's a real effect, but it isn't my effect. The honest question is KF minus PLACEBO, not KF minus bare. If KF only beats a naked model, I've proven nothing about KnowledgeForge.

I ran it on GAIA — a respected benchmark of real-world assistant tasks across three difficulty levels — using a held-constant model and a seed-random sample of 20 tasks (8 Level-1, 8 Level-2, 4 Level-3), three runs each, for 180 total executions. I pre-registered the placebo, the rubric, and the go/no-go criteria before writing the runner, so I couldn't move the goalposts after seeing results.

The Good

The harness works, and KF doesn't hurt. On the clean run, scoring only well-formed answers and excluding tasks the bare model had likely memorized [insert your memorization-exclusion criterion]:

KF was the top overall scorer, it beat the bare model by ~20 points, and it produced fewer malformed/incomplete answers than the other arms. Nothing about adding the framework degraded quality or got penalized on formatting. For a first real run of a brand-new evaluation pipeline, "it runs cleanly and the framework does no harm" is a good outcome.

KF's self-routing also behaved sensibly. It read simple factual-lookup questions and declined to spin up heavyweight reasoning machinery for them. The run only confirms half of that design, though. I learned that KF doesn't over-trigger on factual work. Whether it triggers correctly when a task actually needs a mode stayed untested, because nothing in the sample called for one.

The Bad

KF minus PLACEBO was a tie. The gap was +0.6 percentage points. At twenty tasks that's indistinguishable from zero, and the sample is too small to have caught a moderate real effect either, so treat it as no signal rather than a proven absence. The entire ~20-point lift over the bare model came from generic good prompting, not from KnowledgeForge. On a task-by-task basis, KF and the placebo won and lost the same tasks.

That's the result the pre-registration was built to catch, and it did. On this benchmark, KF bought me nothing a well-written generic prompt didn't already buy. I'm not going to pretend otherwise.

The Ugly

Here's the part that actually matters, and it only became visible because I instrumented for it.

I had KF emit a small, machine-readable telemetry marker on every turn, recording which mode it engaged. When I mined those markers across all 180 runs, I found this:

Every single KF run classified its task as a "reckoning" — a direct-answer question needing no specialized mode. Zero Critic. Zero Debugger. Zero Strategist. Zero adversarial verification passes.

This also explains the tie. A reckoning, in KF, collapses to a careful direct answer, which is close to what a good generic "think it through" prompt already produces. Once KF classified every task as a reckoning, matching the placebo was the only outcome the mechanism could produce, and there was nothing weak to explain. That makes the Bad and the Ugly one finding reported twice, not two results that happen to agree.

In other words: GAIA never exercised the part of KnowledgeForge that makes it KnowledgeForge. GAIA is, by construction, factual question-answering with tool use. KF correctly reads those tasks as reckonings and answers them plainly — so the modes, the whole reason the framework exists, simply never fired. I wasn't measuring a weak effect. I was measuring the wrong thing entirely.

That reframes the null completely. "KF ≈ placebo on GAIA" is not evidence that KF doesn't work. It's evidence that GAIA can't test KF.

There's a less comfortable way to read the same telemetry, and the honesty this whole exercise runs on means saying it out loud. GAIA bills itself as real-world assistant tasks. If every one of them is a reckoning, then KF's modes never reach the most common thing people ask an assistant to do, which is look something up and chain a few steps to a definite answer. The wrong-benchmark framing skips a second meaning. KF buys nothing on deterministic-answer work, and that work is a large share of real usage. That's a real boundary on where the framework helps, and I'd rather mark it than let "GAIA was a bad fit" stand in for it.

And there's a trap right next to it: if I'd published "KF = placebo" as a benchmark headline without this context, I'd have handed everyone a misleading soundbite about my own product. The telemetry is what let me tell a null result apart from a blind spot.

What I actually learned

Three things, in order of importance:

1. Benchmark fit is the whole game. A reasoning framework can only be measured on tasks that demand reasoning of the kind it scaffolds. GAIA's difficulty axis is multi-hop retrieval and tool use — not open-ended design, diagnosis, or judgment. Those are different axes. Picking the right benchmark matters more than the score on any given one.
2. Instrument the mechanism, not just the outcome. The pass-rate alone would have left me guessing. The mode telemetry turned "I don't know why it's flat" into "I know exactly why it's flat." From now on, mode activation is a first-class gate: a benchmark only counts as a valid test of KF if I can show the modes actually engage on it.
3. Don't build your own benchmark to flatter yourself. The tempting next move is to hand-pick tasks KF is good at. That's the exact reward-hack that has discredited other frameworks, and I won't do it. The answer is to use independent, externally-authored benchmarks that happen to exercise the right kind of work — never a curated task list of my own greatest hits.

Next steps

One objection writes itself here. I got a null on the one test that scores itself objectively, and the plan is to move to design and judgment tasks that resist objective scoring. From the outside that looks like relocating to ground where I get to grade my own homework. I take the objection seriously, so the safeguards travel with me. Rubrics get pre-registered before any run. The deciding gate is blinded human review, not my own read of the transcripts. And the nulls get published next to the wins, the same way this one did. A positive result only counts if it survives a setup designed to deny it.

This is a journey, and I'm treating it like one. The roadmap from here:

• A benchmark that fires the modes. I'm moving to multi-step, agentic evaluations where Critic/Debugger/Strategist have real surface to act — independently authored, contamination-resistant, and openly licensed. The first gate isn't the score; it's "do the modes activate at all?" If they don't, I stop and pick a better instrument before spending another dollar.
• Mode telemetry as the steering signal. The same markers that exposed this null become my map: which modes correlate with wins, on which task types, over time. I want longitudinal data on KF's behavior, not a one-off number.
• The real test: design and judgment work. KF's actual home turf — producing specs, reviewing systems, debugging, weighing trade-offs — is hard to score automatically, which is precisely why it's under-benchmarked. Rubric-graded, open-ended evaluations (with blinded human review as the final gate) are where a positive result would be both real and credible. That's the destination.

A framework you can only evaluate on tasks it's guaranteed to win isn't a framework — it's marketing. The whole point of building a benchmark that can embarrass me is that, when KF does move the needle, you'll have good reason to believe it.

More soon.