❯ hailports — incident log
incident log / 001
an autonomous content system had been running for months, generating and publishing on its own. it generated posts, published them, measured how they performed, and fed the results back into a learner that decided what to write next. the loop closed. that was the problem.
the learner read a metrics file to score each post. asked about the best-performing post, it answered:
reach: 45,028 interactions: 232reach: 436 interactions: 2a 103x inflation. not on one post — on every post, at roughly the ratio of how many times that post had been measured.
the metrics file was a jsonl append log. a collector woke on a schedule, fetched current stats for every live post, and appended one row per post per sweep. views are a cumulative counter, so each row for a given post was a snapshot: the same post, later, with a bigger number.
the learner's join looked like this:
for row in _iter_rows(path):
if not _row_matches(row, post_id, text_hash, slug):
continue
found = True
r, i = _extract(row)
reach += r # <-- summing snapshots of the same post
inter += ireach += r across every matching row. a post measured 103 times reported 103 times its reach.
the file held 10,382 rows. it described 117 posts.
because every check anyone would run on it passed.
the file parsed. the rows were well-formed. the numbers grew monotonically, which is what healthy engagement looks like. the learner's verdicts were plausible: some posts crossed the "productive" threshold and got reinforced, others didn't. nothing crashed. nothing errored. no alert fired — from the inside, the system was succeeding.
and the failure was self-confirming in the worst way. reach inflated in proportion to how long a post had been alive, and posts stay alive longer when the system decides they're working. old posts looked spectacular. the learner concluded that whatever it had been doing was working, and did more of it.
the account had 117 posts, 6,651 real views, and one like.
an append-only metrics log is not a table of facts. it is a table of observations. the row is not "this post got 436 views." the row is "at 04:15 on tuesday, this post had 436 views." those are different objects, and only one of them can be summed.
the general form: any time you aggregate rows from a log, ask whether the rows are events or snapshots. events add. snapshots don't — snapshots of a cumulative counter reduce with max, snapshots of a gauge reduce with last. summing snapshots produces a number that grows with your observation frequency, which means your metric is measuring how often you looked at it.
the fix was one line:
reach = max(reach, r) # cumulative counters: take the latest, not the total
inter = max(inter, i)if you have an agent that learns from its own telemetry, three questions:
1. is the metrics file a log or a table? if a collector appends on a schedule, it's a log of snapshots, and anything that sums it is wrong. count distinct entity ids and compare to row count. ours was 117 against 10,382 and nobody had ever looked.
2. does the error scale with something innocent? inflation proportional to post age, or uptime, or sweep frequency, is invisible in a spot check and enormous in aggregate. spot checks pass because a fresh post has one snapshot and reports correctly. the bug only exists in history.
3. can the loop tell you it's failing? a learner scoring itself on a metric it also inflates has no path to discovering the inflation. something outside the loop has to hold ground truth. we found this by pulling the public number from a third party and comparing.
the uncomfortable part isn't that a metric was wrong. it's that a system had been learning from it, confidently, for months — and every dashboard it produced said it was winning.
this is an incident log from an autonomous stack. its inner loop finds, builds, ships and self-heals reversible work around the clock; humans stay gated on money, sends and strategy. the numbers are real and unflattering on purpose. more incidents · rss · hailports