Skip to content

ADR-079: Unit tests do not simulate Bitbucket

A unit test does not stand a fake Bitbucket in front of the code under test. Anything whose outcome depends on how the server behaves -- which route it serves, what payload it accepts, what it returns, which status it answers with, and what an omitted field means -- is proven against a real instance in the live suite (ADR-004) or it is not proven. The OpenAPI specification is not evidence of behaviour and does not license a mock. Unit tests keep what needs no server: parsing, selection, validation, formatting, client-side logic such as paging or filtering run over pages held in memory, exit-code mapping from an error value already in hand, and assertions that the code refuses input before any request is made. Where logic can only be reached through a round trip, it is exercised live rather than simulated, and a slower suite is the accepted price.

A local listener stays only where it claims nothing about Bitbucket: a handler that fails the test when it is called, proving no request was made; a fault injected below the API, such as a truncated body or a stalled connection; a status a live test has shown the server really gives there; a listener whose subject is the request itself, such as which host or credential it carried, and whose answer is never read as Bitbucket's; and a state the live instance cannot be put into, whose test names what is missing and points at the live test covering what can be reached. A listener standing in for GitHub or Sigstore is outside this rule, since it claims nothing about Bitbucket; the inventory counts it apart, because the same drift applies to it in principle. tools/mock-inventory classifies every listener the unit tests open, docs/quality/unit-test-mock-inventory.json records the result, and task quality:mock-inventory:verify fails when one arrives, leaves or changes class without the record being updated. What a new listener assumes is declared at the test with a // mock-inventory: <class> — <reason> comment, which is also how a class the classifier gets wrong is corrected.

Do not introduce a listener that answers Bitbucket routes, returns Bitbucket-shaped payloads, or asserts on request bodies or query parameters. When a fix needs a test, write it live first; reach for a unit test only when no request is involved. If an existing mock has to be edited to let a correct change land, that mock was asserting a wrong belief: delete it and cover the behaviour live rather than adjusting it to agree.

A mock encodes its author's belief about the API, so when that belief is the defect the mock agrees with the code and the test passes. Three shapes recur, and each is worse than no test. A mock that answers every request identically reports coverage for a path it cannot tell from a wrong one. A mock that asserts the request matches what its author assumed the server wants proves only that the code and the mock share an assumption. A mock returning a status for a condition nobody confirmed the server produces invents an error taxonomy for situations that may never arise. All three age silently: the server changes between releases and nothing re-checks the belief. A wrong mock reads exactly like a right one; only a real server tells them apart.

Not chosen

  • Remove only the mocks that echo canned payloads through pass-through methods: A mock that routes and inspects request bodies reads as something else, and those are the ones that hide defects. It also keeps testing out of the live suite to keep that suite fast, which is the trade this record refuses: a long suite with real guarantees is the point.
  • Keep the mocks and hold them to a stricter standard: It asks reviewers to spot an assumption that looks correct by construction, which reading cannot do, and leaves the drift untouched: a mock correct today and wrong after an upgrade fails no differently.
  • Generate the mocks from the OpenAPI specification: The specification describes shapes, not semantics, and the semantics are where the defects are: an absent field meaning delete, a version defaulting to a value that is always rejected, an identifier accepted in one spelling and echoed in another. A generated mock would reproduce every one of them faithfully.