One issue a month about the things we broke: what happened, what actually caused it, and the rule we now hold ourselves to. We do not pick the flattering ones — an incident list with nothing on it is the kind you should not trust.
See the daily operating numbers →
This issue covers 18 July to 16 August 2026 — 30 days.
This is the list of walls we walked into. We run this publication on our own system every day;
what it looks like when it breaks, how long we took to notice, and whether it is fixed — it is all here.
The big numbers are deliberate. This is not a "we're rock solid" brochure. It is proof that
we hide nothing. A system that runs itself every day will throw a couple of thousand failure
signals in 30 days; what would be abnormal is an incident list with nothing on it.
Over these 30 days we itemised 43 incidents: 21 fixed, 17 still open, plus 5 classes of failure
the system absorbs by itself. Five of them are told in full below, each in the same shape:
what happened → what actually caused it → the rule we now hold ourselves to.
What happened: On 16 August, a Chinese-language deep dive that had already passed quality
review existed only in a working directory, never committed. Someone tidying that directory
silently reverted it to an older version. For 6.7 hours nothing flagged it. When the site
rebuilt that afternoon, it shipped the old version.
What actually caused it: Several jobs work in the same directory at once, and every command
the version-control tool offers — stash, restore, switch — acts on the entire tree, not on
"the lines I just changed." One "put my changes aside for a moment" command that afternoon
swept up 230 files. Exactly one was recovered.
The rule we now hold ourselves to: (1) anything that passes quality review is committed on
the spot — no finished piece may live only in a working directory; (2) before shipping, a machine
reconciles "the file about to go out" against "the version that passed review," and stops the
send if they differ; (3) sending only your own hunks now goes through one official pinned-revision
procedure, which structurally cannot pick up someone else's work.
What happened: The same day, two features were simply absent from the live service for
3.5 hours — a reader survey could not write back, and one public endpoint did not exist at all.
Drift monitoring reported "in sync" the whole time. Zero alerts.
What actually caused it: The monitor asked "did the source change?" The source genuinely had
not. What had been swapped out was the artefact built from that source.
Measuring only the input and never the output is a structural blind spot.
The rule we now hold ourselves to: the monitor's question changed from "did the input change?"
to "is what I last produced still where I put it?" If the artefact is gone, it goes red — whether
or not the input moved.
What happened: That same evening, two builds made purely for testing went live for real.
No one pressed publish, nothing was committed, and none of the 19 quality checks ran.
What actually caused it: The build artefact was the deployment source. There was no shipping
gate in between, so "making it" and "deciding to ship it" were the same act — which means any
slip of the hand is a release.
The rule we now hold ourselves to: a shipping-provenance check now stands in the way. If the
version stamp does not match the actual contents, or the working directory is dirty, nothing is
swapped in. Making and shipping are two separate acts from now on.
What happened: The internal management brief failed to arrive on 16 of these 30 days.
A machine reconciled delivery at 12:00 every day and logged the miss every day — and nobody
was on the other end of that log.
What actually caused it: The reconciliation ran, the record was written, but **no role was
ever named as the recipient of that record.** The alarm was ringing into an empty room. Which is
why, on the day we compiled this list, we still could not say whether those 16 days were a
breakage or a deliberate pause — nobody had ever attributed them day by day.
The rule we now hold ourselves to: everything that gets blocked, filtered, or logged must have
a named destination and a named recipient. A record with no recipient does not count as handled.
Compiling this list was itself the first product of that rule — **listing your incidents is how you
find the bells that have been ringing with nobody there to answer.**
What happened: After an earlier "this chart looks tampered with" incident, we built a
safeguard: a legitimate revision must declare in the data who changed it. The safeguard's own
checks were green, and the field really was there in production — but the interface never
displayed it. So the same problem happened a second time, and a second time the only reading
available to the user was "the data was tampered with."
What actually caused it: acceptance stopped at the data layer instead of at the user's eyes.
A green light measures what we did, not what the reader saw.
The rule we now hold ourselves to: a safeguard's acceptance criterion now always ends at
"the user can see it." Present in the data but absent from the screen counts as unfinished.
This month had four main threads — whole-tree rollbacks, shipping without checks, two jobs
colliding on one file, and commits being blocked by someone else's work in progress. All four had
a defence land the same day. But three of them sealed the symptom, not the cause:
itself still measures the wrong object;
running — the third occurrence happened on the same day.
Every one of the 17 open incidents has been seen, filed, and assigned a work item. The status is
"not yet converged," not "unknown." We write this out rather than reporting only the 21 fixed ones
because "symptom fixed" and "cannot recur" are two different things, and anyone considering this
system needs to know that we can tell them apart.
Worth saying last: the 43 incidents above are the residue left after several layers of
automatic defence.
Over the same period, lock contention on commit was absorbed by automatic retry more than a
thousand times; over seven hundred same-type signals were chain-muted so that one root cause could
not flood the diagnostic queue; more than a hundred genuine overlaps between two pieces of work
were parked intact in a quarantine area for a human to review rather than force-merged; six times
the system hit a service quota ceiling and stopped dispatching or switched accounts instead of
burning out mid-task; and 13 drafts that did not meet the bar were stopped before they were sent.
**A system's reliability is not that it does not fail. It is whether something catches it when it
does, and whether someone writes it down when nothing does.** This digest is the evidence for the
second half.
1. "A couple of thousand failure signals" is a signal count, not an incident count; one root
cause can emit hundreds. The 43 items here are incidents after manual consolidation. The two
numbers cannot be derived from each other.
2. The self-healing classification log is only complete after 6 August; earlier figures are
reconstructed from other records and may be understated.
3. Dates for lesson files are taken from the text inside them, not from when they were committed —
on 16 August a batch of older lessons was committed late, and counting by commit date would
record all of them as that day.
4. **"Fixed" means there is a work item, a fix, and a regression test. It does not mean
"cannot recur."** The section above lists three where the symptom is sealed and the cause is not.
The above is this site's own operating record. It is not a guarantee of any outcome or quality, and it does not predict future results.