Compiled CLI · macOS + Linux · no third-party dependencies · offline licence · plumbing.sh
plumb
Your coding agent says it's done. Nothing checked.
An agent saying it's finished and the work actually being finished are two different things, and nothing in between them checks. plumb reads what your repository says has to pass, runs one of those checks, and reports the result — including the cases where it couldn't run anything, which most tools skip past. One compiled binary, no dependencies, no network call.
- Python 3.11+ (stdlib only — no third-party dependencies)
- Compiled to a single binary, source stripped at build time
- Offline licence verification — signed file, key compiled into the binary
- macOS arm64 · Linux x86-64 + aarch64
- Polar (billing) · Vercel (site + artifact routing)
- Next.js App Router · TypeScript · Tailwind
- Proprietary — reading is free, running costs money
Origin
How it started
I built this because a study I ran failed. I'd wanted to know whether a repository's documentation says enough about what it enforces that you could build a gate on it. It doesn't. So the tool had to read the machine-readable files too, and be honest about the gap between what a project writes down and what it actually enforces. Most of the design follows from that.
Features
What it does
Three of the five verdicts are refusals
PASS and FAIL are the easy ones. BLOCKED means the check exists but couldn't run here — missing binary, uninstalled dependency. INCONCLUSIVE means it ran and settled nothing. NOT-RUN means your repository declares nothing to run. That last one is common; five of my own seven projects come back that way. If a tool answers PASS or FAIL to all five situations, it's wrong in three of them.
It reads your prose. It never runs your prose.
plumb pulls out what your docs require and quotes them with the file and line. It won't turn a sentence into a command just because the sentence contains something command-shaped. I checked this on a 1,855-file repo: every executable candidate came from the Makefile, the CI workflows, or scripts those reference. None from any Markdown file. I drew that line early, before the research on agents installing packages named in vendor docs made it a topic.
It refuses what announces itself — and I say so plainly
If a command's name says it rewrites your files rather than checking them — make fmt is the obvious one — plumb skips it, and the skip never counts as a pass. What it can't do is know that a target called lint quietly rewrites things, because that's indistinguishable from a real check until it runs. The site says that in those words. I'd rather lose the stronger sentence than have someone find out I couldn't back it.
The licence works offline because a licence server is a dependency
Your licence is a signed file, checked on your machine every run against a key compiled into the binary. No server to be down, no activation call, nothing about your repositories leaving your machine. It works air-gapped and behind a corporate proxy for the same reason. If a renewal is late it keeps working for a week and tells you, because cards fail and invoices sit in approval queues.
A gate that fails the build when a number drifts
Every number the product publishes is registered in a file, recomputed from the shipped binary, and checked against every document. If one drifts, the build fails instead of the number reaching a customer. I built it after three wrong numbers made it onto the website in one afternoon — a test count from an old draft, an acceptance figure that belonged to the previous binary, and a comparison count that had been right that morning.
You don't have to take my word for the network claim
Run it with your network interface down. Or watch it under tcpdump while it works. It's the one claim that's easier to test than to read, which is the single thing a closed binary has going for it over open source — you can settle it yourself in ten seconds instead of reading my code and hoping.
Under the hood
Engineering
Held out once, spent once
The part that reads prose for obligations was scored against a corpus someone else labelled — their questions, their answers. I split it by repository and froze the split in a commit before I changed any code, and wrote the pass mark down first. It went from 41.2% to 70.6% on 48 repositories I'd never touched while building it, against a 65% bar set beforehand. The development split scored 87.1%. I don't quote that number anywhere, because it only measures how well the fix fits the sentences I used to design it. The held-out set got scored once. That's the whole reason 70.6% means anything.
The number that keeps the rest honest
Across 92 repositories, only 22.6–31.1% had a runnable declared check at all. Of my own seven projects, five had nothing. I made that counterweight travel with every corpus figure, because a discovery tool that only quotes its hit rate is describing a world it picked. plumb reports the empty case in full instead of finding something to run, and on real repositories the empty case is the most common answer.
One feature I stopped instead of fixing
There's a mode that compares what your docs claim against what your build does. I reviewed it by hand, finding by finding, on repositories I'd kept out of development, against a rubric I froze before starting, with a stopping rule I wrote down in advance. It didn't clear the bar. So it ships marked experimental, no licence pays for it, and I never sent a single finding to a maintainer — they weren't good enough to send. Saying that here is cheaper than someone else discovering it.
Everything that mattered, I found by running it
Not one of these turned up by reading code. A licence file truncated by a full disk produced byte-identical output to having no licence, so a paying customer got told to go buy one. An unknown flag exited with the same code the docs reserved for an unlicensed run, so a CI script following my own documentation would report the wrong cause for a typo. A test fixture I'd appended after an exit call was three dead tests that would have crashed if they'd ever run. All obvious the first time someone executed the path, all invisible before that.
The half I could actually solve
It compiles to a single binary with the source stripped at build time, signed and notarised on macOS, built against an old glibc floor so it runs on Linux hosts far older than my build machine. Every artifact carries a digest of the tree it came from and the release gate refuses a set that doesn't all share one source — I added that after a build segfaulted and left a stale archive that passed every other check. Install is one shell line that verifies a checksum before it puts anything on disk. None of which solves the part I'm actually stuck on, which is getting anyone to want it.