Skip to content

ADR 067: Governance tests are verified by breaking them

This page is generated from docs/decisions/*.yaml by task docs:export-adr-markdown. Do not edit manually.

  • Number: 067
  • Title: Governance tests are verified by breaking them
  • Category: development
  • Status: accepted
  • Provenance: guided-ai
  • Source: docs/decisions/067-governance-tests-are-verified-by-breaking-them.yaml

Decision

A governance test asserts an invariant about the codebase rather than a behaviour: every command is classified, every MCP tool has a scope rule, no record names something that was removed. They are what stops documentation and registries drifting away from the code they describe. Three rules govern them. Break a guard before trusting it, and write the sabotage as a test where it can be expressed as one. Prefer making a contradiction unrepresentable over testing for it: when two declarations answer the same question, derive one from the other; when they answer different questions, assert both directions. And keep the set listed here, because it follows no naming convention and cannot be found by grep. The set: - TestAllRunnableCommandsDeclareArgsPolicy: a command with a positional placeholder declares an Args policy. - TestAllCommandsExhaustivelyClassifiedForDryRun: every command is in exactly one dry-run category. - TestCommandVerbsAgreeWithTheirDryRunClassification: a command's name and its category do not contradict. - TestClassifyUsageErrorMatchesCobrasRealMessages: the usage-error markers match what Cobra emits. - TestEveryMCPToolIsAccountedFor: every MCP tool maps to a command or is recorded as MCP-only. - TestEveryMappedCLICommandExists: no mapping names a command that was removed. - TestEveryToolHasAScopeRule: no MCP tool escapes workspace scoping. - TestADRDoesNotNameToolsThatDoNotExist: a record does not name a removed tool. - TestGatedToolsAreTheOnesThatMergeOrGate: the --yolo set is exactly the tools that merge or gate. - TestReadOnlyToolsAreNotGated: a tool that writes nothing is not withheld. - TestLiveMCPEveryToolReturnsAClientCompatibleResult: every MCP tool is called, and its result is a JSON object with a text fallback. - TestEveryHookRunnableGateRunsOnBothSides: every gate a git hook can run runs locally and in CI. - TestNoGateIsDefinedAndNeverRun: a task named like a check is reachable from something that runs it. - TestEveryADRCrossReferenceResolves: a record does not cite a record that does not exist. - TestAcceptedRecordsDoNotNameABitbucketVersion: a record in force does not restate a Bitbucket release. - TestBitbucketImageIsProposedButNotAutoMerged: the product image is proposed but held from auto-merge. - TestDependabotProposesUpdatesAgainstNext: every Dependabot update targets next, and auto-merge leaves one opened elsewhere to a person. - TestEveryActionIsPinnedToACommit: every workflow action is pinned to a release's commit, with the release named beside it. - TestEveryReleasedPlatformHasItsSBOMsAttested: every platform the release builds has its archives attested with their SBOMs. - TestPolicyLoadingNeverCreatesTheSystemConfigDirectory: reading policy does not create its directory. - TestVerbClassificationExemptionsNameRealCommands: an exemption names a real command and gives a reason. - TestAmbientGitConfigGuardIsInstalledWhereTestsShellOutToGit: a package running git installs the guard. - TestOnlyTheSharedHelperDecidesInteractivity: only the shared helper asks whether a stream is a terminal. - TestEveryUseOfStandardInputIsAccountedFor: every place standard input is used is recorded. - TestNoServiceOptionIsCalledLimit: a service list option says whether it caps or pages. - TestACappedListingSaysSoIsEnforced: a command that takes --limit reports reaching it, in JSON and in text. - TestEveryLimitedToolSaysWhenItStopped: an MCP tool that takes a limit returns a required limit_reached. - TestNoFailureIsWrappedAsTransient: a transport failure keeps its classification through the service that wraps it. - TestAnEchoedFlagPublishesTheValuesItAccepts: a payload field that echoes an enum flag publishes the flag's values. - TestAMissingArgumentIsNamed: a missing positional argument is named in the error. - TestNoFixtureIsNamedFromTheClock: no test builds a fixture name from time.Now(). - TestGovernanceTestsNamedInThisRecordExist: this list names only tests that exist.

Agent Instructions

Break a governance test before adding it, and before trusting one you did not write. Record what the invariant is, what breaks it, and that you saw it fail. Add the test to the list above in the same change; a guard nobody can find is one nobody maintains. Do not write a test that compares a value to something derived from it. Check first whether the assertion is already true by construction; if it is, the code is where the guarantee lives. Do not describe a check as CI-safe. Every check can run anywhere since ADR-043 gave CI its own licensed Bitbucket; what separates the hook set from the CI-only set is the cost of booting the stack, not the ability to.

Rationale

Two guards here had stopped guarding anything and still passed. One asked whether every mutating command was registered while defining "mutating" as "registered", so it asserted that a map contains what it contains. Review is what let them in: they read correctly, and the only fault is that both sides of the comparison come from the same place, which is invisible without running it. A third looked identical and turned out to be real, and only running the sabotage told them apart. Deriving beats checking wherever it is available. The MCP safety flag and the destructive annotation answered the same question in two hand-written places, and the unguarded direction was the dangerous one: the flag is what the server enforces, the annotation is advice a client may ignore. They are now derived, and the test comparing them is gone.

Rejected Alternatives

  • Centralise the governance tests in one package: Each sits beside the thing it guards, where someone changing that thing will trip over it. Enumerating the set is what this record does instead.
  • Keep the list in AGENTS.md only: Nothing verifies AGENTS.md. A list is only worth having if something checks it.
  • Trust code review to catch a tautological guard: Review is what let both of them in.