2026-08-01
Fixed — a badly fragmented Story could never be consolidated, no matter how many merges our adjudicator approved. Duplicate Stories are merged by taking the transitive closure of accepted merge pairs, then requiring that a component carry no more than two pairs no adjudicator ever examined. That guard is right — an implied weld nobody checked is false about half the time — but it was enforced by discarding the entire component. Candidate generation supplies at most 3n/2 examined pairs for a component of n clusters while the check demands roughly n(n-1)/2, and those cross just under five, so any group of five or more fragments was mathematically unmergeable. Measured on 2026-07-31: two-fragment groups merged 481 of 481, groups of six or more merged 0 of 62. One migration story held 132 fragments with 750 already-approved merges, every one discarded, leaving 111 duplicate Stories live.
Merges are now applied one pair at a time in ascending distance order, and only the pair that would breach the limit is refused — the rest of the group still merges. Every resulting Story still satisfies the identical two-unexamined-pair guarantee, so this is not a partial merge of a suspect group; each one independently passes the same test. Replaying five days of real verdicts on the hourly schedule the pipeline actually runs (the harness reproduces 91-98% of production's real merge counts): 8% to 24% more duplicates consolidated per day, and the migration story falls from 111 live Stories to 26.
Duplicate detection now probes each Story against the day's others individually instead of computing every pair in one query. The old approach grew with the day and stopped finishing around 8,500 Stories, which meant the last six hourly passes of every UTC day failed and merged nothing — and because a day is never revisited, the entire evening news cycle stayed un-deduplicated permanently. The new path is 9.6x faster at exactly that size, finds no fewer duplicates (verified against every merge verdict from a full day), and surfaces roughly 53 more real duplicates per pass that the previous method could not propose at all.
Fixed — a Story pair we had explicitly judged to be DIFFERENT incidents could still be merged. The safety check counted any examined pair as satisfying it, so a "no" was treated exactly like a "yes" — and because the exhaustive sweep over the day's biggest Stories compares every pair among them and roughly 95% come back rejected, the more carefully a group had been examined and refused, the more freely it could be welded together. Over 2026-07-27..07-31 that merged 494 pairs we had explicitly rejected, including one 24-cluster Story joining US and Saudi strikes on Iraqi militia facilities to a separate US strike on Iran's Qeshm Island. A rejected pair now blocks those two Stories from ever sharing a merge group, and the count is zero on every day tested. Separately, rate-limit and transport failures were being stored as "different incident" verdicts — 11.2% of one day's rejections were HTTP 429s — which both suppressed the pair from ever being judged and fed this same defect; they are now retried instead of recorded.
Known limit — this fixes what happens to merges we approved, not how many pairs we look at. A Story fragmented across hundreds of clusters still has most of its pairs never compared, because candidate generation ranks each cluster's three nearest neighbours and the hourly pass that would widen that has been failing in the evening since 2026-07-27. Expect fewer duplicates, not none, until that lands. If you deduplicate Stories yourself, note that merged Stories continue to be marked superseded rather than deleted, and the counts on surviving Stories will move upward.