92 repositories · two model families · blinded annotation · threshold set in advance
The completion-gate study
I wrote down what would count as success. Then I missed it.
Coding agents decide for themselves when they're finished. I wanted to know whether repositories already contain enough written instruction to check that decision automatically — whether docs, contribution guides and agent-instruction files actually expose what a project's CI enforces. If they did, you could build a gate out of what teams have already written. I fixed the bar for calling that true at 70%, before I collected any data.
- Pre-registration with frozen decision rules
- Third-party labelled corpus
- Blinded annotation · two model families · position check
- Held-out split, scored once
- Python · reproducible harness
Origin
How it started
The problem with measuring your own idea is that you get to decide what counts as success after you've seen the answer. Unless you refuse to. So I wrote the success criterion down first, somewhere I couldn't quietly move it, along with what I'd do if I missed it: publish the comparison and stop. Which is what happened.
Features
What it does
The bar came first
70% coverage of what CI enforces, written down and frozen before I collected anything, along with what I'd do if I missed it. Nothing came close — not one channel, not any combination, not even the upper bound. I picked the number while it was still possible for it to embarrass me.
The answer is a range, and stays a range
22.6–31.1% across 92 repositories. I report it as an interval because the classification involves judgement, and a single number would imply precision the method doesn't have. The fair objection — that 70% was arbitrary and a tool catching 30% is still useful — I concede rather than argue. What the distribution actually shows is that the coverage isn't spread thin, it's concentrated in a handful of repositories while most expose nothing.
Blinded, and I checked the blinding
Judgements were made without knowing which arm produced what, by two independent model families rather than one, with a position check to confirm the blinding held instead of assuming it. Isolating annotators isn't the same as making them independent, and the difference was worth the extra run.
A ceiling I didn't want to find
Both model families agreed that a large share of code-bearing repositories with working CI document no runnable obligation at all. A perfect extractor gets nothing out of those. That ceiling limits every tool in this space including the one I built afterwards, and it's why that tool reports an empty result rather than finding something to run.
Thirteen bugs and not one of them crashed
The harness kept producing clean, plausible numbers that were wrong. Every defect returned something I'd have been happy to publish — some flattered the hypothesis, one pointed away from it, none of them looked like bugs. I wrote that up separately, because an experiment that fails silently is more dangerous than one that falls over.
I withdrew my own best numbers
A later part of the same programme produced results I liked a lot. I pulled them after finding four defects in the harness, including one that counted a command which couldn't run from the repository root as a success in both arms. I retracted rather than caveated, and I couldn't rescore the original run because I hadn't kept the exit codes. That's the part of the method that actually costs something.
Under the hood
Engineering
Why the corpus had to be someone else's
A benchmark you write yourself measures how well your tool handles sentences you chose. I used a labelled corpus published independently — provisions labelled by people with no stake in how it came out — plus a held-out split I scored exactly once. The gap between those two is the number that matters: on a set I built by hand the same extractor scores far higher than on strangers' data. That difference is what grading your own homework is worth, and I'd rather have it measured than assumed.
The failure is what produced the product
Documentation on its own doesn't carry the load. That killed what I'd planned and produced something else — a tool that reads machine-readable files and prose together, treats them as different kinds of evidence, and keeps what a repository declares separate from what it merely mentions. Every refusal verdict in that tool exists because this study established there's often nothing there to find.
Rule text and commands aren't the same payload
A related result from independent work: quoting a rule's text substantially moves whether a model discloses something, and does nothing measurable for whether it verifies anything. Commands are what move verification. That's why the tool I built keeps the commands a repository declares separate from the obligations its prose states instead of blending both into one context blob — they buy different things, and mixing them hides which one is working.
What this doesn't establish
One researcher, public repositories, and models doing the annotation rather than people. I varied the model families and checked the blinding, but a model is a subject here as much as it's an instrument, and that's a real limit rather than a footnote. I also chose the threshold rather than deriving it. What the study does establish is narrow: the documentation channel, on its own, doesn't carry what a completion gate would need.