the draft looked fine until i checked the claim
the agents on the box draft a lot of work for me: lookups, plans, mail drafts, status notes. every one of them waits for me. the hard part was never spotting a broken draft. a claim with no source can sit inside a clean draft all day.
1. block the known bad phrases first
a deterministic regex blocklist runs before any model judge. if a draft matches, it stops there. no model spends tokens rediscovering a phrase that was already rejected.
2. write down what earns a score
a second model scores every draft 1 to 10 against a written rubric, and anything under 8 is thrown away. without the rubric, the number is just a mood.
3. make broken judge output fail closed
unparseable JSON scores 0. empty output scores 0. an unreachable model scores 0. a missing verdict is a rejection until the gate can run again.
4. separate the writer and the judge
when possible a different model judges the draft. a model is too good at recognizing its own habits as good work.
5. get agreement before trusting praise
a high score needs a second opinion. if the judges disagree, the draft is held instead of averaged into a pass.
6. keep the phrases that got rejected
the blocklist grows from real rejects, and those phrases go back to the drafter as do-not-use lines.
7. log the reason with the verdict
a score alone gives you nothing to fix. the reason sits beside every verdict.
8. hold claims without a source
a fluent sentence can still be unsupported. if a claim cannot be traced to a source, the draft goes on hold.
9. watch failures by drafter
pass rate is measured per drafting model. if one keeps failing the same gate, it gets dropped instead of relaxing the floor.
the same discipline keeps the box honest in other places too. two RAM freezes taught it to run every background job in 20s wall-clock slices under a hard timeout.