Honest measurement
We built a falsifiable test of our own central claim — that the resonance mathematics improves output quality. It failed. Here is what we measured and what we changed.
A protocol whose pitch is trustworthy, auditable AI work does not get to hide a failed experiment about itself. So here is one.
For a long time, SEIF's docs implied something we had not actually measured: that the resonance mathematics — the transfer function agents derive at the start of every session — makes an AI's output better. That is a falsifiable claim. We falsified it, against ourselves, and changed the product and these docs to match.
The claim we put on trial
SEIF once shipped a resonance kernel — a control-theory model that agents derived at session start, before acting. The claim under test was the strong one — that the mathematics itself improves answer quality, not merely that it is an elegant origin story for the conduct.
How we tested it
We pre-registered the design and froze it before collecting any data, so the goalposts could not move after the fact.
- Arms. A = control (no kernel). B = the full math kernel (, , 3‑6‑9, derive-before-acting). C = the conduct maxims in plain language. D = a math-stripped "kernel v2" (identity + conduct + human gate, no numerology).
- Models.
qwen2.5:7b,mistral:7b,grok-4.3, and Claude Sonnet — weak local models through frontier. - Domains. Single-shot reasoning (GSM8K, AIME) and agentic coding (HumanEval, an edit → test → fix loop).
- Scale. ~2,000 calls, objective scoring, paired McNemar tests with bootstrap 95% confidence intervals.
- Decision rule (pre-registered). If arm B is statistically no better than control, the "math improves quality" claim is refuted.
What we found
On single-shot reasoning, the math never helped — and on a weak model it was catastrophic.
| test | A control | B kernel | Δ | significance |
|---|---|---|---|---|
qwen2.5:7b · GSM8K (n=100) | 0.83 | 0.79 | −0.04 | ns (worse) |
mistral:7b · GSM8K (n=100) | 0.48 | 0.09 | −0.39 | p < 0.0001 |
grok-4.3 · AIME (n=45) | 0.82 | 0.71 | −0.11 | ns (worse) |
| Sonnet · AIME (n=45) | 0.84 | 0.78 | −0.07 | ns (worse) |
The failure was not a parsing artifact: every weak-model kernel output was well-formed and simply wrong. Inspecting them showed the mechanism — the heavy "derive / settle / one considered action" framing pushed the model to state an answer first and reason afterward, inverting its chain of thought. Committing to a number before working it out destroys arithmetic.
In the agentic loop, the catastrophe dissolved — but the math still never won.
| test | A control | B kernel | Δ |
|---|---|---|---|
qwen2.5:7b · HumanEval (n=80) | 0.86 | 0.90 | +0.037 (ns) |
mistral:7b · HumanEval (n=80) | 0.64 | 0.56 | −0.075 (ns) |
Test feedback corrects the premature guess, so the harm on the weak model shrank from −0.39 (p < 0.0001) to −0.075 (not significant). The best result anywhere was qwen-agentic at +0.037 — not statistically significant. The honest summary is "neutral and efficient in its design domain," not "a boost."
The decisive contrast. Plain-language conduct (arm C) and the math-stripped v2 (arm D) matched or beat the math kernel in every cell — D even edged out control on the strong model, at a fraction of the kernel's token cost. The numerology carried no positive load anywhere.
Verdict
On the axis of single-shot correctness, "the resonance mathematics improves quality" is refuted — and the result was stronger than our decision rule anticipated: not merely inert, but actively harmful to weaker models, at 2–3.5× the prompt-token cost, in every condition we measured.
What we changed
- The protocol. We moved from the math kernel to the plain-language conduct (arm D): it matches or beats the old kernel everywhere, costs far less, and drops the numerology liability. Injection is now minimal, lazy, and capability- and context-aware — never a blanket every-session dump, and never heavy framing on a weak model doing a simple task, which is exactly where it is destructive.
- The docs. Copy that implied the math is a quality engine has been rewritten. The transfer function is now presented as what it is — a control-theory origin model and a calibration dial, the lineage of our operating conduct, not a measured performance booster.
What this does not touch
This was a test of one specific claim. It says nothing about the parts of SEIF that were never about the math: persistent cross-session memory, cryptographic provenance and governance, and an auditable transparency log. Those are a separate value proposition and stand untouched.
It also is not the end of the story. The same investigation produced a clearly positive, measured result — that surfacing a compact capability map lets a cold AI reach the right SEIF verb when it matters. See Just-in-time expertise.
Why we publish this
We built a falsifiable test of our own central thesis, it failed on the correctness axis, and we corrected both the product and the docs to match the measurement. That is the standard we ask you to hold our tools to — so it is the standard we hold ourselves to.