Test triangulation: properties that say what must never happen

1One angle is not enough

Most test suites are a list of examples. An engineer picks an input, works out the expected output and asserts it. That is useful, and it has a ceiling: the suite checks the cases somebody thought of. Aniche starts from the same limit. Exhaustive testing is impossible, so testing is a matter of choosing inputs well and knowing when to stop. In his words, “No matter what testing you do, it will never be perfect or enough” [EST, ch. 1].

Coding agents make the ceiling lower. An agent can write a feature and its tests in the same minute. If both come from the same reading of the requirement, they share its blind spots: a misread rule produces code and tests that agree with each other and are both wrong. Reviewers cannot read every line any more, so the suite has to find the cases nobody looked for.

Triangulation is borrowed from surveying, where a point is fixed by taking bearings from more than one place. In testing it means checking one behaviour from several independent angles, so that a gap one angle misses is caught by another.

2Four angles on one behaviour

Effective Software Testing builds a developer's suite in layers. It starts from the specification, adds tests the source code reveals, designs contracts into the code, and then states properties for a framework to check [EST, chs. 2–5]. Each layer finds something the others miss.

No single angle is enough. The useful signal is disagreement: a branch that no specification test reaches, or a property that fails on an input the examples skipped, marks a gap in the code or in the understanding of the requirement. Of the four, properties change the most about how tests are written, so the rest of this note concentrates on them.

3Properties, not examples

An example-based test encodes one engineer's idea of a case: charge 120, refund 30, expect 90 left. The engineer chooses the input, so the test inherits the engineer's assumptions about which inputs matter. Property-based testing takes that choice away. Aniche describes the shift directly:

“We do not pick a concrete example; rather, we define a property (or a set of properties) that the program should adhere to, and the test framework tries to find a counterexample.” [EST, ch. 5]

The engineer's job changes from choosing inputs to stating constraints. A property is a rule that must hold for every valid input. The output of a sort is in order and contains the same elements as the input. A discount never takes a price below zero. Serialising a message and parsing it again returns the original. Together, the properties form the constraint set: the team's written claim about what the behaviour is, independent of any particular example.

The framework then plays the adversary. It generates inputs, mixes in edge cases such as zero, empty lists and the largest allowed values, and tries to break each property [JQWIK]. The test passes only when the framework fails to find a counterexample. When it succeeds, the failing input is evidence that the code, the property or the constraint set is wrong.

Examples do not disappear. They remain the clearest way to document a specific case, and every counterexample the framework finds should become one (R5). Aniche compares the two styles and uses both [EST, ch. 5]. What changes is that examples are no longer the only description of what the system must do.

4Write down what must never happen

The most valuable properties are often negative. A system that moves money, grants access or spends a budget is defined as much by what it must never do as by what it does. Writing those rules down is design work, not a testing chore, which is why this note sits at Design on the route. In an AI system, the never-rules usually live in the ordinary code around the model:

• A refund never exceeds the amount charged.
• An agent never calls a tool outside the permissions of the user it acts for.
• Retries for one request never exceed their budget, however the failures are arranged. Retries without a budget are on Hold on the radar as unbudgeted automatic retries.
• A parser never passes on model output that fails its schema.

Rules like these are what the radar calls controls enforced outside the model. That code is deterministic, so a generated input either breaks the rule or it does not, and properties are the most direct way to test it. Many never-rules are also invariants in Aniche's sense, so the same rule can be enforced as a contract in the code and checked as a property in the tests [EST, chs. 4–5].

5Thousands of cases, one counterexample

Exhibit 1 is a property for the first rule above, written for jqwik, the Java property-based testing library used in the book [EST, ch. 5] [JQWIK]. The guard under test checks each refund request against the amount charged. The property does not list cases. It says that, for any charge and any short sequence of requests, the total refunded never exceeds the charge.

@Property(tries = 10_000)
void refundsNeverExceedTheCharge(
    @ForAll @IntRange(min = 1, max = 100_000) int chargedCents,
    @ForAll @Size(max = 5)
    List<@IntRange(min = 1, max = 100_000) Integer> requestsCents) {

  RefundGuard guard = new RefundGuard(chargedCents);
  int refundedCents = 0;
  for (int request : requestsCents) {
    if (guard.allow(request)) {
      refundedCents += request;
    }
  }
  assertThat(refundedCents).isLessThanOrEqualTo(chargedCents);
}

tries = 4                     | # of calls to property
checks = 4                    | # of not rejected calls
seed = -1428803127462810357   | random seed to reproduce generated values

Shrunk Sample (9 steps)
-----------------------
  chargedCents: 1
  requestsCents: [1, 1]

Original Sample
---------------
  chargedCents: 61428
  requestsCents: [48310, 27795, 1]

Three things in that report matter to a leader, not only to an engineer:

• Nobody chose the inputs. jqwik tries each property 1,000 times by default, and the team can raise that, as here, to 10,000 [JQWIK]. Across a suite of properties, every push runs thousands of test cases that nobody had to write.
• The counterexample is minimal. After a failure, the framework shrinks the input to the smallest one that still fails: a one-cent charge and two one-cent requests. Nobody has to debug the original 61,428-cent case.
• The failure is reproducible. The report records the seed, and by default the next run tries the failing sample first, so the failure cannot quietly disappear on the next build [JQWIK].

6A counterexample is a question about the constraint set

The guard in Exhibit 1 does exactly what its rule says: it rejects any single refund larger than the charge. The property still fails, because two refunds of one cent add up to more than a one-cent charge. In an agent system that input is not exotic. An agent that times out and retries the same refund produces it. Asking why the property failed has three possible answers, and each has a different owner:

• The code is wrong. The rule meant a limit on the total, and the guard checks each refund on its own. Engineering fixes the guard.
• The property is wrong. The business allows refunds above the charge, for example as goodwill credits. Engineering corrects the property and records why.
• The constraint set has a gap. The requirement never said whether the limit applies to each refund or to the total, or whether a retried request counts as a new refund. Neither the engineer nor the agent should guess. The question goes to the owner of the requirement.

The third answer is the main reason to do this. A set of properties is a precise statement of what the team believes the rules are. When the framework breaks one, it often shows that the rules were incomplete or contradicted each other. Finding that in a test run costs a conversation; finding it in production costs a dispute with a customer. Tracing each property to the need it protects, in the way Problem-Based SRS traces requirements to customer problems, makes it clear who owns the answer.

7Map the value stream to find the waste

Section 6 claims that a gap found in a test run costs less than one found later. A value stream map shows how much less, for a particular team. The idea comes from lean thinking, which Mary and Tom Poppendieck applied to software in Lean Software Development. They define waste as “anything that does not add value to a product, value as perceived by the customer”, and give handoffs as an example: “In product development, handing off development from one group to another is waste” [LSD, intro]. Their first chapter pairs two tools: seeing waste, whose list includes waiting and defects, and value stream mapping [LSD, ch. 1]. A map follows one request from the moment it is asked for to the moment it is delivered, and records where the time went: doing the work, waiting for it, or doing it again.

Read as a value stream, the change was worked on for 4 of its 20 days and waited in a queue for the other 16. Three points of improvement stand out:

• Faster coding barely moves the total. The agent's half day is already one of the shortest steps on the map; halving it again saves a quarter of a day in twenty. The Poppendiecks observe that experts “have a tendency to maximize the performance of the part of the product representing their own specialty rather than focusing on overall system performance” [LSD, intro]. Buying faster code generation without looking at the queues makes the same mistake at the scale of a program. Their chapter 7, See the Whole, returns to the theme [LSD, ch. 7].
• The defect was found late, after two handoffs. The code that allowed a double refund was written on day 4 and the defect was found seven working days later, after the change had passed from the agent to a reviewer and from the reviewer to a tester. A property run on every push (R2) fails within minutes of the code being written, before the first queue.
• The real question was asked last. The defect raised the constraint-set question from section 6: does the limit apply to each refund or to the total? Answering it cost three days of waiting for the requirement owner and a day and a half of rework. Asked on day 0, while the never-list is written with that owner (R1), the same question costs one conversation.

In the terms of this note, the map is one more angle. The test suite says whether the code is right; the map says when the team finds out, and what the delay costs. None of the three points is visible from inside a single step: the agent, the reviewer and the tester can each be quick while the change still takes four weeks. The Poppendiecks put the remedy plainly: “Compressing the value stream as much as possible is a fundamental lean strategy for eliminating waste” [LSD, intro]. Properties compress one part of it by moving the discovery of defects and gaps to the start. Mapping a comparable change again after adopting them (R7) shows whether they did, and where the largest waste now sits.

8Requirements for leadership

These are written so that they can be copied into an engineering standard or a definition of done. Each one links to the section above that motivates it.

The key words MUST, MUST NOT and SHOULD are to be read as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.

1. R1

   Every component that enforces a business rule, a permission or a limit MUST have a written list of what must never happen, owned by the person accountable for the rule.

   §4 · [EST]

2. R2

   Each item on that list SHOULD be expressed as a property and run on every push, with the number of tries and the seed recorded in the test report.

   §3 · §5 · [JQWIK]

3. R3

   Test suites for such components MUST combine more than one angle from section 2. Code coverage alone MUST NOT be accepted as evidence that a component is tested.

   §2 · [EST]

4. R4

   Every counterexample MUST be triaged as a code defect, a wrong property or a gap in the constraint set. Gaps MUST go to the owner of the requirement and MUST NOT be settled by the engineer or agent that found them.

   §6

5. R5

   Each shrunk counterexample SHOULD be kept as a named example test, so that the case stays covered when generators or seeds change.

   §3 · §5

6. R6

   Where a coding agent writes the code, properties SHOULD be derived from the requirement separately from that code, by a person or by an agent that has not seen the implementation.

   §1 · §3

7. R7

   Before adopting these requirements, a team SHOULD map at least one change from request to release, recording working time, waiting time and where each defect was found. It SHOULD map a comparable change again afterwards and compare the two.

   §7 · [LSD]

9Where to start

1. Map one change, then pick one rule. Follow a recent change from request to release and note where it waited and where its defects were found (R7). Then pick one rule that costs money when it breaks, such as refunds, credits, permission checks or retry budgets, and write its never-list with the rule's business owner (R1).

2. Turn the list into properties. Add a property-based testing library for the team's language; the book's examples use jqwik for Java. Run the properties on every push and keep every shrunk counterexample as an example test (R2, R5).

3. Triage together, then map again. Review the first counterexamples with the business owner and sort each into the three answers from section 6. Map a comparable change and compare it with the first map. Then write R1 to R7 into the definition of done for components that enforce rules (R3, R4, R6).

10What is still open

Being specific about the limits is part of the guidance:

• Good properties take creativity and practice. Aniche gives a section of the chapter to exactly that, and another to common issues [EST, ch. 5].
• A property that repeats the implementation proves nothing. If the test computes the expected answer the same way the code does, both carry the same mistake.
• Generators can be biased. A generator that never produces an empty list, or only produces values in a narrow range, cannot find defects outside it.
• Passing is not proof. A property that survives 10,000 tries has not found a counterexample in those tries. It has not shown that none exists.
• Model outputs are not deterministic. Properties test the code around the model. Judging the model's own answers needs repeated trials and evaluation, such as repeated-trial reliability on the radar.
• One map is an anecdote. A single change can be unusual, and durations recalled from memory can be wrong. Map several changes, taking the times from the tracker and the delivery pipeline where possible, before deciding where the largest waste is.

None of this argues for fewer example tests. It argues against letting examples be the only statement of what the system must do.

11References

• [EST]Aniche, M. Effective Software Testing: A developer's guide. Manning, 2022. Chapters cited: 1, Effective and systematic software testing; 2, Specification-based testing; 3, Structural testing and code coverage; 4, Designing contracts; 5, Property-based testing. Listed in this site's field guide.
• [JQWIK]jqwik. jqwik user guide, current version.
• [LSD]Poppendieck, M. and Poppendieck, T. Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003. Parts cited: the Introduction; chapter 1, Eliminate Waste (Tool 1, Seeing Waste; Tool 2, Value Stream Mapping); chapter 7, See the Whole. Listed in this site's field guide.
• [RFC2119]Bradner, S. Key words for use in RFCs to Indicate Requirement Levels, and RFC 8174.