Background
Nobody reads the documents they agree to. Terms of service, privacy policies, consent dialogs — the people who draft them know exactly what is in them, and the people who sign them almost never do. Compass exists to close that gap: a local-first companion that detects notable clauses (arbitration, class-action waivers, third-party sharing, data sale, biometrics, AI training, and more), scores them by section, and renders nutrition-label summaries a normal person can act on.
Everything runs on-device. The SQLite store is created with owner-only permissions and the tool refuses to start if they are broader. There is no server, no telemetry, and no account.
What I built
A rule-based analysis engine with a CLI and a local web view, organized around detectors — one per clause family, each with its own documentation explaining what it looks for and why the clause matters. Detected clauses roll up into per-section scores and an overall severity band, so a dining-reservation flow that quietly shares contact data with a third-party provider surfaces as exactly that.
The part I care most about is how it stays correct. Compass is built against a calibration corpus: real excerpted policy text paired with gold-standard expected outputs, snapshot-tested so any change in detector behavior surfaces as drift that has to be explained or deliberately re-blessed. For a problem this fuzzy, that discipline is the difference between a tool and a guess.
What the corpus caught
The corpus has earned its keep repeatedly, and the catches make a better story than the features.
One suspicious fixture — 21 clauses, 16 of them the same type — exposed a systemic bug: keyword matchers emitted one clause per keyword occurrence, and the range-finder always resolved to the first occurrence, producing byte-identical, mislocated duplicates. Before touching code, I measured the blast radius: the bug was latent in 5 of 20 already-blessed fixtures. The fix was a deduplication pass at the pipeline’s single chokepoint, and the re-blessed fixtures were pure deletions with unchanged section scores — proof of zero scoring regression.
Later, a false positive (reporting that a policy collects biometrics when its own disclosure table says it does not) was fixed with a line-local guard — and building the regression fixture for that fix exposed the inverse bug, a false negative that silently suppressed a genuinely disclosed practice. Both errors invert the information asymmetry the tool exists to fix, so both got regression tests before the fix landed.
The most uncomfortable catch: the snapshot path was analyzing each fixture’s own metadata header while the audit path stripped it, which meant the gold standard had been contaminated by my own annotation hints. The test was grading against a poisoned answer key. One shared strip helper, a clause-by-clause re-bless, and one fixture’s severity band flipped — proving the contamination had been materially skewing results. The corpus statistics now come from a generator instead of being hand-maintained, so they cannot drift again.
Why it matters
The interesting engineering here is not the detectors — it is treating a solo project’s correctness the way a production team would: gold-standard fixtures, drift that must be explained, blast-radius measurement before fixes, and symmetric treatment of false positives and false negatives. The result is a privacy tool whose answers I can actually stand behind.