Vectors confirm behaviour, not correctness

ai-object is the substrate under fifteen apps. It turns a sentence into structural edits: "add a section about caching" becomes an insert, "remove the intro" becomes a delete. When I ported it to Python I generated sixty parity vectors from the TypeScript and held the Python to them. All sixty passed on the first run.

That felt like a strong result. It was, for what it measured. What it measured was that Python did whatever TypeScript did — including the things TypeScript did wrong.

The package with no tests

Later, auditing the test counts I had written into these pages, I checked which packages had tests at all. ai-object had none in TypeScript. Zero. The substrate carrying fifteen production apps was pinned only by parity vectors generated from itself — a mirror, not a check.

So I wrote a TypeScript suite: the same sixty vectors, plus a dozen assertions about what the planner is *supposed* to do. Seventy passed. Two failed.

What the two failures were

The first was mine — an assumption about how the planner scores targets that turned out not to hold. The second was not.

const MOVE_UP = has(lower, ["move up", "earlier", "before", "promote", "to the top"]);

`has` matches contiguous phrases. So "move up" matched, and "move the introduction up" did not — the words are separated. Nor did "move it down". The move intent was effectively unreachable for the way anyone actually phrases it.

And it did not fail safe. Falling past every branch, the message hit the fallback: apply this instruction to the selected block. So asking to reorder a section **rewrote its contents**. The user asks to move something and their text is replaced.

The vectors recorded this perfectly. Two of the sixty had a move phrasing, both produced an edit op, and both implementations agreed — precisely, deterministically, and wrongly.

Why the parity suite could never have caught it

This is the structural weakness of generated vectors, and it is worth being clear about because the technique is otherwise excellent. Vectors generated from an implementation encode *what it does*. They are the right tool for keeping a second implementation aligned with a first, and they caught real drift twice in this project.

But they cannot express intent. There is no vector for "a move instruction should produce a move op", because a vector is not a claim about what should happen — it is a photograph of what did. Agreement between two implementations is a strictly weaker property than either being right.

The fix is not to distrust vectors. It is to have both kinds of test: vectors for agreement, and assertions for intent. The second kind is what I had skipped in TypeScript, and the gap was invisible from inside the parity suite because the parity suite was green.

Fixing it in two places at once

A move and its direction are now matched separately, so "move the intro up", "move it down" and "move the introduction to the bottom" all reorder rather than rewrite. I also dropped "before" and "after" from the direction words — they are ordinary prepositions, and "add a paragraph after the intro" reading as a move was only prevented by branch ordering, which is a fragile thing to depend on.

Then the loop closes properly: fix TypeScript, regenerate the vectors from the corrected code, port the same fix to Python, and confirm it still matches. Both implementations now agree on behaviour that a separate suite says is correct.

One more thing I left alone

My first failing assertion — the one that was my mistake — turned out to expose something real too. Target scoring weights every word equally, so "about" counts as much as "invalidation". Ask to add a paragraph about invalidation and it lands in the Introduction, because that section's preview happens to contain the word "about".

That is a weakness, not a bug: the planner still does something reasonable. Changing the scorer would alter behaviour across fifteen apps and two languages, and I could not deploy and observe the result from where I was working. So it is now a test that documents the actual behaviour and says why it was left — which is a better artifact than either a silent workaround or a fix I could not verify.