Property-Based Testing - #7052

I want PHPUnit to support property-based testing natively, without depending on Eris or any other library. A property should be an ordinary test method whose parameters PHPUnit generates from their types, and every generated example should run as a real test, with its own before-test and after-test methods, its own test doubles, and its own expectations. A library cannot provide this from the outside, because only PHPUnit controls the lifecycle of a test.

Part 1: Why PHPUnit should do this itself

Properties find the inputs we did not think of

When we write tests the usual way, we pick example inputs and specify the expected output for each of them, for instance with a data provider. We only test what we think of, and the bugs are where we did not think to look.

A property describes an invariant that must hold for all valid inputs instead: reversing a list twice yields the original list, adding two amounts of money is commutative. A generator produces many random inputs, and the property is checked for each of them. When an input is found for which the property does not hold, shrinking simplifies it step by step until a minimal counterexample remains: [-1] instead of [-12, 5, -7, 3, 19, -2].

I have written about this in more detail: Property-Based Testing introduces the idea, Smaller Input, Greater Insight explains shrinking, and Data Provider or Properties? discusses when to use which.

Eris works from outside PHPUnit, and that is where its limits come from

Today, property-based testing with PHPUnit means using Eris. You use its trait in your test case class, declare generators with forAll() inside a test method, and pass the property to then() as a closure:

<?php declare(strict_types=1);
use Eris\TestTrait;
use PHPUnit\Framework\TestCase;
use function Eris\Generator\int;
use function Eris\Generator\seq;

final class ReverseTest extends TestCase
{
    use TestTrait;

    public function testReversingTwiceIsIdentity(): void
    {
        $this->forAll(seq(int()))->then(
            function (array $input): void
            {
                $this->assertSame($input, array_reverse(array_reverse($input)));
            },
        );
    }
}

I am grateful for Eris. @giorgiosironi started it in 2014 and has made property-based testing available to PHP developers for more than a decade, most recently by adding support for PHPUnit 11, 12, and 13. The articles linked above use Eris. I am not proposing this to compete with Eris, and certainly not to spite its maintainer and his contributors.

I am proposing it because Eris has to integrate with PHPUnit through a trait and the few extension points PHPUnit offers, and some of the resulting limitations can only be removed inside PHPUnit.

The whole property runs inside a single execution of the test method. before-test and after-test methods run once for all examples, so state leaks from one example into the next, and into shrinking. A test that creates an empty stack in a before-test method, pushes the generated value onto it, and asserts that the stack now holds one element fails with a message such as Failed asserting that 80 is identical to 1. Expectations for test doubles such as once() are verified after all examples have run. And expectException() applies to the property as a whole: the first example that throws the expected exception ends the test, and the test passes, even when the property does not hold for other inputs.

Only assertion failures are shrunk. When the code under test throws an exception or raises an error, the property ends without a minimal counterexample, and the report does not show the input that caused it. The same is true for expectations for test doubles that are verified after the property has run. When an example triggers a deprecation, notice, or warning, PHPUnit reports it for the test but cannot say which input triggered it.

The seed that reproduces a run comes from an environment variable or from the current time and is echoed to the console, so it is missing from Open Test Reporting and other logfiles. The random values come from PHP's global Mersenne Twister, which the code under test shares.

Finally, generators are declared apart from the parameters of the closure that receives the values, so static analysis cannot check that the closure accepts what the generators produce.

A property should be a test method with generated parameters

A property is a test method with the #[Property] attribute. Its parameters are the inputs of the property. By default, PHPUnit derives a generator for each parameter from its type, using the same PHPDoc type syntax that PHPStan and Psalm understand:

<?php declare(strict_types=1);
use PHPUnit\Framework\Attributes\Property;
use PHPUnit\Framework\TestCase;

final class ReverseTest extends TestCase
{
    /**
     * @param list<int> $input
     */
    #[Property]
    public function testReversingTwiceIsIdentity(array $input): void
    {
        $this->assertSame($input, array_reverse(array_reverse($input)));
    }
}

Refined types such as integer ranges, non-empty-string, and array shapes work as well:

/**
 * @param array{name: non-empty-string, age: int<0, 150>} $person
 */
#[Property]
public function testEveryPersonCanBeGreeted(array $person): void
{
    $this->assertStringStartsWith('Hello', (new Greeter)->greet($person));
}

Enumerations are generated from their cases. Objects are generated through their constructor, recursively, using the types of the constructor's parameters:

enum Currency: string
{
    case EUR = 'EUR';
    case USD = 'USD';
}

final readonly class Money
{
    /**
     * @param non-negative-int $amount
     */
    public function __construct(public int $amount, public Currency $currency)
    {
    }

    // ...
}

#[Property]
public function testAdditionIsCommutative(Money $a, Money $b): void
{
    $this->assertEquals($a->add($b), $b->add($a));
}

/**
 * @param int<1, 10> $parts
 */
#[Property]
public function testAllocationPreservesTheAmount(Money $money, int $parts): void
{
    $this->assertSame($money->amount, array_sum(array_map(
        static fn (Money $share): int => $share->amount,
        $money->allocate($parts),
    )));
}

When a type does not say enough, the #[ForAll] attribute specifies the generator for a parameter explicitly. The generator is built with composable functions such as Generators::lists(), Generators::tuples(), map(), and filter(). It can come from a static method of the test case class:

use PHPUnit\Framework\Attributes\ForAll;
use PHPUnit\Framework\Attributes\Property;
use PHPUnit\Framework\PropertyBased\Generators;
use PHPUnit\Framework\PropertyBased\ValueGenerator;

/**
 * @return ValueGenerator<non-empty-string>
 */
public static function bitStringsWithoutLeadingZeros(): ValueGenerator
{
    return Generators::strings(alphabet: '01', maximumLength: 62)->map(
        static function (string $bits): string
        {
            return '1' . $bits;
        },
    );
}

#[Property]
public function testConversionToDecimalAndBackIsLossless(#[ForAll('bitStringsWithoutLeadingZeros')] string $bits): void
{
    $this->assertSame($bits, decbin(bindec($bits)));
}

A generator that several test case classes need can be a static method of another class, which the #[ForAllExternal] attribute names, just as #[DataProviderExternal] names a data provider of another class:

use PHPUnit\Framework\Attributes\ForAllExternal;

#[Property]
public function testConversionToDecimalAndBackIsLossless(#[ForAllExternal(BitStrings::class, 'withoutLeadingZeros')] string $bits): void
{
    $this->assertSame($bits, decbin(bindec($bits)));
}

With PHP 8.5, which allows static closures in constant expressions, the generator can be declared right in the attribute:

#[Property(examples: 1000)]
public function testEmailAddressCanBeParsed(
    #[ForAll(static function (): ValueGenerator
    {
        return Generators::emailAddresses(quotedLocalParts: true);
    })]
    string $address,
): void
{
    $this->assertSame($address, EmailAddress::fromString($address)->asString());
}

Generators::emailAddresses() is one of the generators for values that PHP code handles as formatted strings: UUIDs, IP addresses, domain names, email addresses, and URLs, as well as points in time. Their values are valid by construction, so no example is rejected, and they are biased towards the special cases where code that handles them tends to break: a local part of 64 characters, an email address of 254 characters, 127.0.0.1, ::ffff:192.0.2.1, the second before daylight saving time ends.

For domain objects that must not be created through their constructor, a test case class can provide a generator that is used wherever a value of that type is needed, including in constructor parameters of other objects:

use PHPUnit\Framework\Attributes\GeneratorFor;

/**
 * @return ValueGenerator<EmailAddress>
 */
#[GeneratorFor(EmailAddress::class)]
public static function emailAddresses(): ValueGenerator
{
    return Generators::emailAddresses()->map(
        static function (string $address): EmailAddress
        {
            return EmailAddress::fromString($address);
        },
    );
}

#[Property]
public function testCustomerCanChangeTheirEmailAddress(Customer $customer, EmailAddress $newAddress): void
{
    $customer->changeEmailAddress($newAddress);

    $this->assertSame($newAddress, $customer->emailAddress());
}

When a property does not hold, PHPUnit reports the minimal counterexample as PHP code that can be copied into a data provider for a regression test:

Random Seed: 1759741221

There was 1 failure:

1) ShopTest::testAllocationPreservesTheAmount
Failed asserting that 0 is identical to 1.

Property falsified after 2 examples, counterexample shrunk in 17 steps:

    $money = new Money(amount: 1, currency: Currency::EUR);
    $parts = 2;

/path/to/tests/ShopTest.php:31

When a value depends on another one, the property draws it in its body with draw(). Drawn values are replayed and shrunk together with the parameters, and the report lists them in the order in which they were drawn:

#[Property]
public function testElementIsFoundAtItsIndex(): void
{
    $list  = $this->draw(Generators::lists(Generators::integers(), 1), 'list');
    $index = $this->draw(Generators::integers(0, count($list) - 1), 'index');

    $this->assertSame($index, array_search($list[$index], $list, true));
}

1) ListTest::testElementIsFoundAtItsIndex
Failed asserting that 0 is identical to 1.

Property falsified after 2 examples, counterexample shrunk in 23 steps:

    Draw 1 (list): [0, 0]
    Draw 2 (index): 1

When the code under test has state, a property runs a state machine. Its rules are the commands that can be issued, in any order, and its invariants compare the system with a model that is as simple as possible. This is how I want to test event-sourced software: the aggregates and the projections are checked against the model after every command, and catching up a projection is a rule of its own, so that the order in which commands and catch-ups happen is explored as well:

final class BankMachine
{
    private array $balances          = [];
    private array $projectedBalances = [];

    public function __construct(private readonly Bank $bank, private readonly EventStore $store, private readonly BalanceProjection $projection)
    {
    }

    #[Rule(produces: 'accounts')]
    public function open(): AccountId
    {
        $account = $this->bank->openAccount();

        $this->balances[$account->value] = 0;

        return $account;
    }

    /**
     * @param int<1, 100> $amount
     */
    #[Rule]
    public function deposit(#[FromPool('accounts')] AccountId $account, int $amount): array
    {
        $this->balances[$account->value] += $amount;

        return $this->bank->deposit($account, $amount);
    }

    #[Rule]
    public function catchUpProjection(): void
    {
        $this->projection->catchUp($this->store);

        $this->projectedBalances = $this->balances;
    }

    #[Invariant]
    public function projectionMatchesTheModel(): void
    {
        Assert::assertSame($this->projectedBalances, $this->projection->balances());
    }

    #[Invariant(atEnd: true)]
    public function accountsRebuiltFromTheirEventsMatchTheModel(): void
    {
        foreach ($this->balances as $account => $balance) {
            Assert::assertSame($balance, $this->bank->balanceOf(new AccountId($account)));
        }
    }
}

#[Property]
public function testBankKeepsTheBalancesOfAccounts(): void
{
    $store = new EventStore;

    $this->runStateMachine(new BankMachine(new Bank($store), $store, new BalanceProjection));
}

The steps are drawn from the choice sequence of the example, like the values that are drawn with draw(). They are shrunk, replayed, and remembered without shrinking code that is specific to state machines, because deleting a step is deleting a span. The minimal counterexample is reported as the PHP code that takes its steps. A value that a rule produced is assigned to a variable, the value that another rule returned is shown as a comment, which for event-sourced software are the recorded events, and an invariant is shown when it failed:

1) BankTest::testBankKeepsTheBalancesOfAccounts
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 [
     'account-1' => 0,
-    'account-2' => 0,
 ]

Property falsified after 4 examples, counterexample shrunk in 17 steps:

    $accounts1 = $machine->open();
    $machine->catchUpProjection();
    $accounts2 = $machine->open();
    $machine->catchUpProjection();
    $machine->projectionMatchesTheModel(); // invariant failed

When rebuilding an account from its events applies a withdrawal like a deposit, the comments show the events that the steps recorded (the withdraw() rule is not shown above):

Property falsified after 5 examples, counterexample shrunk in 34 steps:

    $accounts1 = $machine->open();
    $machine->deposit(account: $accounts1, amount: 1); // [new MoneyDeposited(account: new AccountId(value: 'account-1'), amount: 1)]
    $machine->withdraw(account: $accounts1, amount: 1); // [new MoneyWithdrawn(account: new AccountId(value: 'account-1'), amount: 1)]
    $machine->accountsRebuiltFromTheirEventsMatchTheModel(); // invariant failed

After creating the state machine the way the property does, this can be pasted into a regression test.

Only PHPUnit can make every example a real test

PHPUnit runs every example through the complete lifecycle of a test: before-test and after-test methods, creation and verification of test doubles, expectException() and the other expectations, output buffering, error handling, and the backup of global state. Properties get the isolation that data providers have always had. Any failure, be it a failed assertion, an exception, an error, or an unmet expectation for a test double, falsifies the property and is shrunk to a minimal counterexample. When shrinking runs into a second bug, it does not switch to it, but the bug is not lost either: the report lists it with a counterexample of its own.

This isolation has a price, and I think it is worth paying. An example costs as much as a data set of a data provider, whereas Eris calls a closure inside a single test. Shrinking makes it more expensive still, because every attempt to find a smaller counterexample is a complete test, so shrinking has to get by with few attempts. That is why I want generators to record the random choices from which they build a value, and shrinking to work on that record instead of each generator implementing its own shrinking. This is the approach that Hypothesis introduced. Generators composed with map(), filter(), and flatMap() then shrink as well as the built-in ones, and small counterexamples are found with far fewer runs of the property.

Deriving generators from types means that the type is written once. The PHPDoc type that PHPStan and Psalm use to analyse the body of the property is the type from which its arguments are generated. For PHP code, the generators have to include the values for which PHP behaves in surprising ways: PHP_INT_MAX + 1 is a float, strlen() counts bytes and not characters, and -0.0 becomes the string "-0".

Reproducing a failure should not depend on anything outside PHPUnit. One seed per test run is printed with the other runtime information. Each property derives its own seed from it and from its test ID, so --filter and execution order do not change the examples a property sees. The counterexamples of a falsified property, the minimal one and one for each other way in which the property was falsified, are remembered in the test run history and tried first the next time. Because PHPUnit knows about properties, counterexamples and seeds appear in the default output, in Open Test Reporting logs etc., and a property is one test for --filter, --list-tests, --stop-on-failure, test sizes, code coverage, and the baseline.

The other price is that PHPUnit takes on the maintenance of generators and a shrinker, work that Eris has done for PHPUnit users so far. I think this is the right trade, because the most important of the limitations described above, the missing isolation between examples, cannot be removed any other way.

Eris will keep working with PHPUnit, and I will not deliberately break its integration. If you prefer Eris, or need a generator that Eris has and PHPUnit does not, keep using it. I would welcome feedback on this proposal from the maintainers of Eris and from its users.

Part 2: Specification

1. Scope

• Property-based testing is implemented in PHPUnit itself. It does not depend on Eris or any other property-based testing library.
• The public API consists of attributes, the assume(), classify(), observe(), target(), draw(), and runStateMachine() methods of TestCase, the Generators factory and the ValueGenerator interface, two new events, and configuration options. Everything else is internal.
• Stateful testing is sequential: the steps of a state machine are taken one after the other (section 3.9).
• Running properties in process isolation and running the steps of a state machine concurrently are out of scope for the first iteration; see section 11. The design must not prevent them.

2. Terminology

• Property: a test method with the #[Property] attribute.
• Example: one execution of a property with generated arguments.
• Generator: an object of type ValueGenerator<T> that generates values of type T.
• Choice sequence: the list of non-negative integers from which a generator constructs a value (see section 6).
• Drawn value: a value that the body of a property generates with draw() (section 3.7).
• Score: a number that an example targets with target(), so that examples with higher scores are searched for (section 3.8).
• State machine: an object whose methods with #[Rule] are the steps that a property takes with runStateMachine() (section 3.9).
• Step: one call of a rule of a state machine.
• Pool: the values that the steps of a state machine produced for later steps, for instance the IDs of the accounts that were opened.
• Counterexample: arguments, and the values drawn and the steps taken with them, for which the property does not hold. The minimal counterexample is the result of shrinking.
• Rejected example: an example that is discarded because a value could not be generated, for instance because a generator filter could not be satisfied, or because assume() was called with false.

3. Declaring properties

3.1 #[Property]

• PHPUnit\Framework\Attributes\Property targets methods. A method with this attribute is a test method; neither the test prefix nor #[Test] is required.
• Optional argument examples (positive integer): the number of examples to check. Without it, the configured default is used (section 9). Either number is multiplied by the configured multiplier (section 9).
• The method should have a void return type. A return value is ignored and is not passed to dependent tests.

3.2 Parameters

Each parameter of a property receives a generated value. The generator for a parameter is determined in this order:

1. the #[ForAll] or #[ForAllExternal] attribute of the parameter (section 5.1);
2. a generator registered with #[GeneratorFor] for the parameter's type (section 5.2);
3. the generator derived from the parameter's type (section 4).

Variadic parameters are not supported. Parameters with default values receive generated values like all other parameters.

If no generator can be determined for a parameter, the test is not run and an error is reported for it, just as for an invalid data provider. The message names the parameter and the type that could not be handled.

3.3 Combination with other attributes

• #[DataProvider], #[DataProviderClosure], and #[DataProviderExternal] cannot be combined with #[Property]. Such a test is reported as an error.
• #[TestWith] and #[TestWithJson] specify the explicit examples of a property (section 3.5).
• #[Depends] and its variants cannot be combined with #[Property]. Such a test is reported as an error. The main purpose of #[Depends] is to pass the return value of a test to the tests that depend on it, but the parameters of a property receive generated values. A property that depends on another test would silently receive a generated value where a return value was expected (section 12).
• Other tests may depend on a property. A property passes when all its examples pass. Like a test method with a void return type, it does not pass a return value to the tests that depend on it.
• #[Repeat], --repeat, #[Retry], and --retry do not apply to properties. A warning is emitted when #[Repeat] or #[Retry] is used on a property, as for other methods that are not eligible for repetition or retrying, and --repeat and --retry are ignored for properties without a warning.
  • Retrying a property with the same seed checks the same examples and is pointless. Retrying it with a different seed is harmful: it turns a counterexample that was found into a pass. A property whose outcome does not depend on its arguments alone is detected by the final replay (section 6.5).
  • Repeating a property checks more examples, which the number of examples (section 9) already controls, at a higher cost: each repetition replays the remembered counterexample and is reported as a test of its own. Detecting flaky tests, the purpose of repetition, is covered by the final replay as well.
• Process isolation does not apply to properties in the first iteration (section 12): a property is always run in the main process. A warning is emitted when #[RunInSeparateProcess] or #[RunTestsInSeparateProcesses] would apply to a property. The --process-isolation CLI option and the processIsolation setting of the XML configuration file are ignored for properties without a warning, just as --repeat and --retry are ignored for tests that are not eligible for them.

3.4 Rejecting examples inside the property

TestCase::assume(bool $condition): void discards the current example if $condition is false. A rejected example neither passes nor fails, and it does not count towards the number of examples. Calling assume() in a test that is not a property is reported as an error.

An example for which draw() cannot generate a value, or for which the arguments of a step of a state machine cannot be generated, because no value satisfies a filter, for instance, is rejected as well (sections 3.7 and 3.9).

3.5 Explicit examples

• #[TestWith] and #[TestWithJson] specify explicit examples for a property: arguments for which the property is checked every time it is checked, in addition to the generated examples. A counterexample that was found can be kept this way, so that it guards against a regression (section 8.2). With named arguments, the counterexample shown in part 1 becomes #[TestWith(['money' => new Money(amount: 1, currency: Currency::EUR), 'parts' => 2])].
• The arguments of an explicit example are passed to the parameters by position or by name, like the data of #[TestWith] is passed to a test method. An explicit example must have exactly one argument for each parameter. A property with an explicit example that does not is reported as an error, as is a property with two explicit examples of the same name.
• An explicit example is named like a data set: with the name that is passed to #[TestWith] or #[TestWithJson], or numbered among the explicit examples without a name (#0, #1, ...).
• The explicit examples are checked before any other example, in the order in which they are specified (section 6.2). They are neither generated nor shrunk, they do not depend on the seed, and they do not count towards the number of examples.
• When an explicit example falsifies the property, no further examples are checked. The property is reported with the failure or error of that example, together with its name and its arguments, which are exported like a counterexample (section 8.2). An object among them was not created by a generator, so it is exported with sebastian/exporter.

Property falsified by explicit example "minus one":

    $value = -1;

• An explicit example that is rejected by assume() is reported with a PHPUnit warning, because an explicit example that does not meet the precondition of the property is most likely a mistake. The other examples are checked.
• An explicit example that is skipped, marked incomplete, or aborted because the deadline of the test run has passed ends the check of the property like a generated example (section 6.2).
• An explicit example has no values for draw() and no steps for runStateMachine(). Calling either of them while an explicit example is checked is an error, so that explicit example falsifies the property (sections 3.7 and 3.9).

3.6 Classifying examples and observing values

• TestCase::classify(string $label): void classifies the current example of a property with a label. An example can be classified with several labels. Only the generated examples that are checked are counted, not the rejected ones and not the explicit ones. Calling classify() in a test that is not a property is reported as an error.
• #[RequireClassification(string $label, float $minimumPercentage)], which can be used more than once on a property, requires that at least the given percentage of the examples that are checked are classified with the label. A property that holds but does not meet this requirement is considered risky, with a message that states the percentage that was reached: Only 4.0% of the examples of this property were classified as "negative", but at least 10.0% are required. The label must not be empty, the percentage must be greater than 0 and not greater than 100, and a label can only be required once; otherwise, the property is reported as an error.
• TestCase::observe(string $label, int|float $value): void observes a numeric value of the current example of a property, for instance the size of a generated list. An example can observe several values with the same label. As for classifications, only the generated examples that are checked are counted. The label must not be empty and the value must be finite; calling observe() with an empty label, with NAN or an infinite value, or in a test that is not a property is reported as an error.
• For each label, the values that were observed are summarized by their count, minimum, median, mean, 90th percentile, and maximum. The median and the 90th percentile are determined with the nearest-rank method, so that they are values that were observed: the smallest values that at least half and at least 90 percent of the values are not greater than.
• How many examples were classified with each label and the summary of the values that were observed with each label are part of Test\PropertyChecked (section 8.1), and --display-property-statistics shows them (section 8.3).

3.7 Drawing values inside the property

• TestCase::draw(ValueGenerator $generator, ?string $label = null): mixed generates a value with the generator in the body of a property. It is generic: for a ValueGenerator<T>, it returns a T, so that PHPStan and Psalm know the type of the drawn value.
• A drawn value can depend on the arguments of the property, on values that were drawn before, and on state that the property created, for instance the size of a list that was drawn before or the keys of an array that a parameter received.
• Drawn values come from the choice source that generated the arguments of the example (section 6.1). They are part of its choice sequence, so they are replayed, shrunk, and remembered together with the arguments.
• $label describes the value in the report of the minimal counterexample (section 8.3). It must not be empty.
• When no value can be generated, the example is rejected like an example for which assume() was called with false, and the reason is counted for the health checks (section 6.2).
• Calling draw() in a test that is not a property, or while an explicit example is checked (section 3.5), is reported as an error.

3.8 Targeting scores

Some properties fail only for rare examples: a schedule that is too long for a day, a sum that exceeds a limit, a value in a narrow range. Random examples seldom come close to them. A property can measure how close an example comes with a score, and PHPUnit then searches for examples with higher scores:

#[Property]
public function testScheduleFitsIntoTheDay(Schedule $schedule): void
{
    $this->target('total duration', $schedule->totalDurationInMinutes());

    $this->assertLessThanOrEqual(1440, $schedule->totalDurationInMinutes());
}

• TestCase::target(string $label, int|float $score): void targets a score for the current example of a property. Higher scores are better. A property can target several scores with different labels; the searches for them take turns.
• Once half of the examples of the property have been checked, the other half are spent on searching for examples with higher scores (section 6.2). The examples that are found this way are checked like any other example: one that falsifies the property is shrunk and reported as usual.
• Only the generated examples that are checked are taken into account, not the rejected ones and not the explicit ones.
• The label must not be empty, the score must be finite, and an example can target only one score per label; calling target() otherwise, or in a test that is not a property, is reported as an error.
• The highest score of each label is part of Test\PropertyChecked (section 8.1), and --display-property-statistics shows it (section 8.3).

3.9 Running state machines

• TestCase::runStateMachine(object $machine, int $maximumNumberOfSteps = 50): void runs a state machine in the body of a property. The state machine is an ordinary object, so it receives the system under test, test doubles that setUp() created, and whatever else it needs through its constructor.
• #[Rule] on a public, non-static method of the state machine declares a step that can be taken any number of times, in any order. precondition: names a public, non-static method without required parameters that returns whether the rule can be applied. produces: names a pool to which the value that the rule returns is added.
• #[FromPool('accounts')] on a parameter of a rule passes a value that an earlier step produced for the pool. With consume: true, the value is removed from the pool, for instance when an account is closed. A rule cannot be applied when one of its pools has no value for it.
• The other parameters of a rule receive generated values, from #[ForAll] or #[ForAllExternal] (section 5.1; a closure requires PHP 8.5, as for the parameters of a property), from the generators that are registered for the property with #[GeneratorFor], or from the generators that are derived from their types, as for the parameters of a property (section 3.2).
• #[Initialize] declares a public, non-static method that is called once, before the first step. The methods with #[Initialize] are called in the order in which they are declared. For event-sourced software, they seed the event store with a history, for instance with events of an earlier version for testing upcasters. Their parameters receive generated values, and they can produce values for a pool, but they cannot take values from one.
• #[Invariant] declares a public, non-static method without required parameters that is called after the methods with #[Initialize] and after every step. A method with #[Invariant(atEnd: true)] is only called after the methods with #[Initialize] and after the last step, which is meant for expensive checks such as rebuilding every aggregate from its events.
• While examples are generated at random, the number of steps of an example is chosen uniformly between 1 and the maximum, only rules that can be applied are chosen, and every example enables a random subset of the rules (swarm testing), so that the examples differ in which rules dominate them. No more steps are taken when no rule can be applied.
• Every step is a span of the choice sequence, like an element of a list: a choice whether another step is taken, the index of its rule in the order in which the rules are declared, and its arguments. A value from a pool is drawn as its position counted from the value that was produced last, so that it still refers to the same value when shrinking deletes earlier steps. When a choice sequence is replayed, a rule that cannot be applied is skipped and does not count as a step, so that a shrinking attempt does not change the meaning of the steps after the one it changed. Shrinking deletes steps, prefers rules that are declared earlier, and simplifies arguments with the passes that it has anyway.
• A failure in a rule, an invariant, or a precondition falsifies the example like any other failure (section 6.2). When the system is expected to reject a command, the rule asserts that with try and catch.
• The number of steps that each example took is observed as by observe() with the label steps, and the number of steps that it took with each rule with labels such as steps with deposit(), so that --display-property-statistics shows them (section 8.3).
• A property is considered risky when the state machine took no step in more than half of the examples that ran it:

The state machine of this property took no step in 100 of 100 examples because none of its rules could be applied; make sure that the preconditions of the rules hold and that values are produced for the pools from which they take values

• A state machine that cannot be run is reported as an error of the property, without shrinking: a state machine without a method with #[Rule], a method with more than one of #[Rule], #[Initialize], and #[Invariant], such a method or a precondition that is not public or is static, an invariant or a precondition with required parameters, a precondition that does not exist or does not return a bool, a pool whose name is empty or not a valid variable name, a pool for which no rule or initialization produces values, a method that produces values for a pool but whose return type is void or never, a parameter of an initialization that takes values from a pool, a parameter that takes values from a pool and has #[ForAll] or #[ForAllExternal], a variadic parameter, and a parameter for which no generator can be determined.
• Calling runStateMachine() in a test that is not a property, while an explicit example is checked, more than once in the same example, or with a maximum number of steps that is not positive is reported as an error.

4. Generators derived from types

4.1 Source of the type

• If the method's doc comment has a @param tag (including @phpstan-param and @psalm-param) for the parameter, its type is used. When there is more than one, @phpstan-param takes precedence over @psalm-param, which takes precedence over @param, as in PHPStan.
• Otherwise, the native type declaration is used.
• PHPDoc types are parsed by PHPUnit itself. The parser understands the types listed in section 4.2 and the syntax they need: generic arguments, integer ranges, array shapes, literals, unions, nullable types, parentheses, and T[]. A type it cannot parse is reported like a type that is not supported (section 3.2), but only for a parameter whose generator is derived from its type.
• Class names in PHPDoc types are resolved against the namespace and the use statements of the file that declares the method, the same way PHPStan does. self and static refer to the class that declares the method.
• PHP discards doc comments when OPcache is active and opcache.save_comments is disabled. PHPUnit then reads the doc comment of the property, and of each constructor that objects are derived from, from the source file that declares it, so that the types in @param tags are used regardless of the configuration of OPcache.

4.2 Supported types

Not supported, and requiring #[ForAll], #[ForAllExternal], or #[GeneratorFor]: interfaces (other than DateTimeInterface), abstract classes, classes without a public constructor, object, mixed, callable, Closure, iterable, resource, and template types.

Generated points in time do not depend on the current date. Their choices describe an offset from the Unix epoch, so that a seed and a remembered counterexample (section 7) produce the same points in time on any day and on any machine. Generators::dateTimes() (section 5.3) generates points in time from another range, in other time zones, with microseconds, and biased towards notable points in time.

Classes whose construction is recursive, for example a tree node that takes child nodes, are generated with a smaller probability of recursing and up to a maximum depth. A generator that is registered with #[GeneratorFor] for such a class does the same by passing recursive: true to lists(), arrays(), or nullable() (section 5.3).

5. Explicitly specified generators

5.1 #[ForAll] and #[ForAllExternal]

PHPUnit\Framework\Attributes\ForAll targets parameters. Its argument is either

• a string: the name of a public static method of the test case class that returns a ValueGenerator, or
• a static closure that returns a ValueGenerator (requires PHP 8.5).

PHPUnit\Framework\Attributes\ForAllExternal(class-string $className, string $methodName) targets parameters as well. It names a public static method of another class that returns a ValueGenerator, for instance #[ForAllExternal(MoneyGenerators::class, 'smallAmounts')]. Like #[DataProviderExternal] and #[DependsExternal], it takes the class and the method as two typed arguments, so that PHP checks them when the attribute is instantiated.

The method or the closure is called once per property, before the first example is generated.

For a parameter of a step of a state machine (section 3.9), the string names a public static method of the class that declares the step, and the method or the closure is called whenever the state machine is run, because only then is the class of the state machine known.

The property is not run and an error is reported for it (section 3.2) when

• a parameter has both #[ForAll] and #[ForAllExternal],
• the name of the class or of the method is empty,
• the class cannot be loaded, does not exist, or is an interface or a trait,
• the method does not exist, or
• the method cannot be called on its class without arguments: it is not public and static, it is abstract, or it has required parameters. Parameters with default values are allowed.

5.2 #[GeneratorFor]

PHPUnit\Framework\Attributes\GeneratorFor(class-string $type) targets public static methods of a test case class (including inherited methods and methods declared in traits) that return a ValueGenerator. The generator is used for every value of exactly that type that a property of this test case class needs, including values needed to derive other objects from their constructor. Two methods registered for the same type are reported as an error.

PHPUnit\Framework\Attributes\UseGenerators(class-string $className) targets test case classes and properties and can be used more than once. It uses the generators that the public static methods of another class register with #[GeneratorFor]. This way, generators can be shared between test case classes without a common parent class or a trait, and a library can ship the generators for its value objects as a class.

• A generator that the test case class registers takes precedence over one that a class used for the property registers, which takes precedence over one that a class used for the test case class registers.
• Two generators that are registered for the same type with the same precedence, for instance by two classes that are used for the property, are reported as an error.
• A class that is used with #[UseGenerators] must be a class that can be loaded, not an interface or a trait, and must register at least one generator with #[GeneratorFor]; otherwise, the property is reported as an error.
• A method that is registered with #[GeneratorFor], by the test case class or by a class that is used with #[UseGenerators], must be callable without arguments, like a method that is specified with #[ForAll] or #[ForAllExternal] (section 5.1): a method that is not public and static, that is abstract, or that has required parameters is reported as an error.
• Like other attributes on a class, #[UseGenerators] on a parent class of a test case class is not used.

5.3 The generator API

PHPUnit\Framework\PropertyBased\ValueGenerator<T> is an interface with these methods:

• map(Closure(T): U $mapper): ValueGenerator<U>
• filter(Closure(T): bool $predicate): ValueGenerator<T>: values that do not satisfy the predicate are regenerated. If this fails too often, the example is rejected.
• flatMap(Closure(T): ValueGenerator<U> $mapper): ValueGenerator<U>

PHPUnit\Framework\PropertyBased\Generators is a class with static factory methods:

• integers(int $minimum = PHP_INT_MIN, int $maximum = PHP_INT_MAX)
• floats(float $minimum = -PHP_FLOAT_MAX, float $maximum = PHP_FLOAT_MAX, bool $allowInfinity = false, bool $allowNan = false, bool $excludeMinimum = false, bool $excludeMaximum = false, bool $allowSubnormals = true): with $excludeMinimum or $excludeMaximum, the range does not include the respective bound. Without $allowSubnormals, subnormal floats, whose magnitude is smaller than PHP_FLOAT_MIN, are replaced with 0.0 or -0.0
• booleans()
• strings(?string $alphabet = null, int $minimumLength = 0, ?int $maximumLength = null, ?list<array{int, int}> $codePointRanges = null): UTF-8 strings. $alphabet is a string of characters to choose from. $codePointRanges specifies the characters to choose from as ranges of code points instead, for instance [[0x41, 0x5A], [0x1F600, 0x1F64F]]; each range is chosen from equally often, and surrogates are left out
• bytes(int $minimumLength = 0, ?int $maximumLength = null): arbitrary binary strings
• fromRegularExpression(string $regularExpression): strings that the regular expression, with its delimiters and modifiers, matches as a whole. Lookaround assertions, backreferences, recursion, conditional subpatterns, inline options, and Unicode properties are not supported
• elements(mixed ...$values)
• cases(class-string<UnitEnum> $enumeration)
• constant(mixed $value)
• lists(ValueGenerator $elements, int $minimumSize = 0, ?int $maximumSize = null, bool $recursive = false, bool $unique = false): with $unique, no element equals another one
• arrays(ValueGenerator $keys, ValueGenerator $values, int $minimumSize = 0, ?int $maximumSize = null, bool $recursive = false)
• tuples(ValueGenerator ...$generators)
• records(array<array-key, ValueGenerator> $generators, list<array-key> $optionalKeys = []): keys in $optionalKeys are sometimes omitted
• nullable(ValueGenerator $generator, bool $recursive = false)
• oneOf(ValueGenerator ...$generators)
• frequency(array{positive-int, ValueGenerator} ...$weightedGenerators): like oneOf(), but each generator is chosen with a probability that is proportional to its weight
• deferred(Closure(): ValueGenerator $generator): the generator that the closure returns when the first value is generated, so that a generator can refer to itself
• recursive(ValueGenerator $base, Closure(ValueGenerator): ValueGenerator $extend, int $maximumDepth = 5): the values of $base and the values that $extend builds from them, nested up to $maximumDepth levels
• objects(class-string $class): the generator derived for a class (section 4)
• fromType(string $type): the generator derived for a PHPDoc type

These methods generate values that PHP code handles as formatted strings, and points in time:

• uuids(list<int<1, 8>> $versions = [1, 2, 3, 4, 5, 6, 7, 8], bool $nilAndMax = false): UUIDs (RFC 9562) of the given versions in their canonical lowercase form, with the version and the variant set. The timestamp of versions 1, 6, and 7 is one field, which is laid out as the version requires; everything else is just bits, so a UUID of version 3 or 5 is not the hash of a name. The nil and the max UUID only with $nilAndMax. Simplest: 00000000-0000-1000-8000-000000000000
• ipv4Addresses(string $network = '0.0.0.0/0'): IPv4 addresses of a network, given in CIDR notation, in dotted decimal notation. Half of them are from the special-purpose ranges (RFC 6890) in the network, such as private networks, loopback, link-local, multicast, and the networks for documentation, or right next to one of them. Simplest: the network address
• ipv6Addresses(string $network = '::/0'): IPv6 addresses of a network in their canonical text representation (RFC 5952): lowercase, without leading zeros, the first longest run of zeros compressed to ::, and IPv4-mapped addresses as ::ffff:192.0.2.1. Half of them are from special-purpose ranges, half of them have a run of zeros of any length and at any position. Simplest: the network address
• domainNames(int $maximumLength = 253): domain names of letters, digits, and hyphens in lowercase, with at least two labels of 1 to 63 characters and an alphabetic top-level domain. Labels of 63 characters, numeric labels, many labels, and domain names of the maximum length are generated often. The top-level domain is drawn after the other labels, from the room they leave, so that the simplest domain name of a length has one label of one letter: a.aaaaaaaaa for 11 characters. Simplest: a.aa
• emailAddresses(bool $quotedLocalParts = false, bool $ipAddressLiterals = false): email addresses (RFC 5321) of at most 254 characters, with a local part of at most 64 characters that consists of the characters that RFC 5322 allows in an atom and of dots, and a domain name. These pass filter_var() with FILTER_VALIDATE_EMAIL. With $quotedLocalParts, some local parts are quoted strings, which can contain spaces and escaped quotes; with $ipAddressLiterals, some domains are address literals such as [192.0.2.1] or [IPv6:2001:db8::1]. Simplest: a@a.aa
• urls(list<string> $schemes = ['http', 'https'], bool $userInformation = false): URLs (RFC 3986) with a host that is a domain name, an IPv4 address, or an IPv6 address in brackets, an optional port (biased towards 1, 65535, and default ports), a path (often empty or with a trailing slash), an optional query of key-value pairs, and an optional fragment. Reserved characters, spaces, and characters that are encoded with two, three, and four bytes in UTF-8 are percent-encoded. The URLs are normalized as far as their syntax goes (lowercase scheme and host, uppercase percent-encoding, no . or .. segments) and pass filter_var() with FILTER_VALIDATE_URL. With $userInformation, some URLs have a user name and a password; filter_var() rejects some of these, because it does not accept a percent-encoded byte in the user information whose first hexadecimal digit is a letter, such as %C3. Simplest: http://a.aa
• dateTimes(?DateTimeInterface $minimum = null, ?DateTimeInterface $maximum = null, list<string> $timezones = ['UTC'], bool $microseconds = false): points in time as DateTimeImmutable objects, by default from 0001-01-01 00:00:00 to 9999-12-31 23:59:59 in UTC and without microseconds. Half of them are notable: where 32-bit timestamps overflow, the turn of the millennium, the first second of a month or a year and the second before it, leap days, and the transitions between standard time and daylight saving time of the time zone, with the second before them. Mapping them to DateTime, a string, or a timestamp is up to the property. Simplest: the Unix epoch in the first time zone, or the bound of the range that is closest to it
• timezones(): identifiers of time zones, half of them notable ones such as Asia/Kathmandu (UTC+05:45), Australia/Lord_Howe (daylight saving time of 30 minutes), or Pacific/Apia (which skipped a day), the other half from DateTimeZone::listIdentifiers(). Simplest: UTC

These generators produce canonical spellings only, so that a property can compare a value with what the code under test returns for it. Values that are valid but spelled differently, for instance uppercase UUIDs or IPv6 addresses with leading zeros, are not generated (section 11).

An IP address takes fewer choices than a domain name. The counterexample of a property that does not depend on the host of a URL, or on the domain of an email address with $ipAddressLiterals, can therefore have the host 0.0.0.0 or [::] instead of a.aa.

$recursive is for a collection or a nullable value that can contain values of the generator it belongs to, for instance in a generator that is registered with #[GeneratorFor] for a class whose objects contain other objects of the class. Elements and values are then added less often, so that the generated values stay small, as they do for derived generators (section 4.2).

All methods have generic PHPDoc annotations so that PHPStan and Psalm infer the type of generated values and check the closures passed to map(), filter(), and flatMap(). PHPDoc cannot express every precise type, for instance for records() or for the bounds of integers(); there the annotated type is wider than the generated values. When the precise type matters, it belongs in the PHPDoc of the parameter (section 4).

I named the interface ValueGenerator and not Generator because many test case classes import PHP's built-in Generator class for their data providers.

In the first iteration, custom generators are created only by composing the generators listed above. The low-level interface for drawing choices (section 6.1) is internal. ValueGenerator is not meant to be implemented outside of PHPUnit: an object of any other class that implements it is reported as an error when it is used for a property.

5.4 Distribution requirements

Besides uniformly distributed values, generators must produce boundary values (minimum, maximum, the value closest to zero), small values, and empty collections with a significant probability. For strings, this includes multibyte and combining characters. Distributions only affect which values are generated. They do not affect shrinking.

6. Execution

6.1 Generating an example

• Every random decision a generator makes is a choice: a non-negative integer within a range, where 0 represents the simplest option (for instance, the value closest to zero, the first element, stopping a list). During generation, choices are drawn from the property's random number generator and recorded as the example's choice sequence.
• A choice has at most PHP_INT_MAX + 1 options. An integer range that contains negative and non-negative values is drawn as two choices: the first selects the non-negative (0) or the negative (1) half, the second the value within it. The non-negative half is [0, $maximum]. In the negative half, the choice c in [0, ~$minimum] stands for the value ~c. In both halves, 0 stands for the value closest to zero, and neither half can overflow.
• Generators that consist of several parts (for instance a list element together with the decision to continue the list) mark the part of the choice sequence they consumed as a span. Spans are used for shrinking. A span knows the span it is part of, and it is labelled with how its choices were drawn: the kind of generator, the configuration that determines which choices it draws, and the labels of the generators it is composed of. Generators that are composed and configured the same way, such as two int parameters, therefore label their spans the same way. All levels of a value that recursive() generates share one label, and so do the values of a generator that deferred() makes refer to itself.
• The values that the body of a property draws with draw() (section 3.7) and the steps of the state machine that it runs with runStateMachine() (section 3.9) are drawn from the same choice source, after the arguments of the example.
• When a choice sequence is replayed, choices are read from the sequence instead of the random number generator. Choices beyond the end of the sequence are 0. A choice that exceeds its range is clamped.
• Together with each choice, the largest choice that it could have been is recorded, so that shrinking knows how far a choice can be raised (section 6.3).
• Generating an example stops with a rejection when it exceeds a maximum number of choices.

6.2 Checking a property

• A property is one test. It counts as one test in the result, for --list-tests, --filter, TestDox, and in the test run history.
• First, the explicit examples of the property, if any, are checked (section 3.5). Then the counterexamples stored in the test run history for this property, if any, are replayed (section 7.3). Then examples are generated until the configured number of examples has been checked, a falsified example has been found, or the maximum number of attempts (10 times the number of examples) has been made.
• When the examples of a property target scores (section 3.8), the second half of the examples is not generated at random once half of the examples have been checked. For each label, PHPUnit climbs from the choice sequence of the example with the highest score so far: it changes one of its choices at a time, first to a larger and then to a smaller choice, and doubles the change for as long as the score increases. The choices are changed in a random order that is derived from the property seed, so that the search is reproducible. When a pass over all choices increased the score, another pass follows; when no change to any choice increased it, the example is a local maximum and the search for that label ends, until a random example reaches a higher score. When the searches for all labels have ended, examples are generated at random again.
• No example is generated twice. The choice sequences of the examples that were generated, or rejected while they were generated, including the values they drew and the steps they took, are recorded, and a choice that would lead to one of them again is replaced. When every example that the generators can generate has been generated, for instance for a single parameter of type bool, no further examples are generated. The property is then only considered risky when all of its examples were rejected.
• Every example, every shrinking attempt, and the final replay (section 6.5) is executed as a new instance of the test case class, through the complete test lifecycle described in part 1, using the same mechanism with which RetryTestSuite creates additional attempts.
• An example is falsified when the test status after its lifecycle is failure or error. That includes failed assertions, any Throwable raised by the test method, by setUp(), tearDown(), or the hook methods, unmet test double expectations, and unmet output and exception expectations.
• An example that exceeds the time limit falsifies the property, and the property is reported as an error. (A test that is not a property is only considered risky when it exceeds its time limit.) The report states that the example was aborted after the time limit and shows its arguments like any other counterexample.
• An example that is aborted because the deadline of the test run (--timeout) has passed does not falsify the property, even though it ends with an error. No further examples are checked, no shrinking takes place, and the property is reported with that error like any other test that is aborted by the deadline.
• When an example is marked as skipped or incomplete, the property is skipped or incomplete with the same message, and no further examples are checked.
• Examples are no longer generated when too many of them are rejected before enough of them are checked: by default, once 50 examples were rejected, or 20 examples were rejected because they were too large (they required too many choices or too many nested objects), while fewer than 10 examples were checked (section 9). A property whose examples are mostly rejected would be checked for few examples, if any, and the rejected examples already tell why.
• When fewer examples than configured could be checked because too many examples were rejected, the property is considered risky. The message states how many examples were checked and rejected, why they were rejected, the most frequent reason first, and how fewer examples can be rejected for that reason:

This property was checked for only 1 of 10 examples because generating examples was stopped after 50 examples were rejected by assume() while fewer than 10 examples were checked; generate the values that the property needs instead of rejecting the others

This property was checked for only 0 of 3 examples, 30 examples were rejected (28 by assume(), 2 because no value that satisfies a filter was generated); generate the values that the property needs instead of rejecting the others

• A property whose state machine took no step in most of its examples is considered risky as well (section 3.9).

6.3 Shrinking

• Before its counterexample is shrunk, the example that first falsified the property is replayed four times. A failure that does not depend on the arguments alone, but on timing or on state that examples share, for instance, is recognised this way before the budget for shrinking is spent on it. When none of the replays falsifies the property in the same way, the property is reported as an error stating that it is flaky, with the failure that was originally observed as its cause, and its counterexample is not shrunk. When only some of them do, the counterexample is shrunk, and the report states in how many of the replays the example falsified the property again (section 8.3). An example that exceeded its time limit is not replayed, because every replay would take as long as the time limit, and a replay that is aborted because the deadline of the test run has passed ends the replays.
• A replay whose generators did not generate the same arguments and values from the same choices is not counted. Neither the replays nor the detection of examples that were generated before (section 6.2) work with such generators, so the property is reported with a PHPUnit warning, once, when a replay draws different choices or when generating an example draws a choice from another range than an earlier example did after the same choices: The generators of this property did not generate the same values from the same choices, so its examples may be generated more than once and its counterexamples may not be reproducible; values must not depend on anything but the choices they are generated from.
• When the arguments of the example cannot be generated again for a replay, because a generator throws, the property is reported as an error with that exception.
• Choice sequences are ordered shortlex: a shorter sequence is simpler than a longer one, and among sequences of equal length the lexicographically smaller one is simpler.
• Starting from the choice sequence of the falsified example, the shrinker repeatedly tries simpler sequences and keeps a candidate if replaying it falsifies the property in the same way. Its passes are: deleting spans, also while raising the choice that determines the length of a later group of spans of a kind that the deleted span contains, replacing a span with a shorter span it contains that has the same label, moving values into later values of the same kind, deleting blocks of consecutive choices, setting choices to 0, trying small values for a choice, binary search on a choice, redistributing pairs of nearby choices, lowering a choice while raising a later one to its largest value, lowering a choice while deleting as many of the spans that follow it, also while raising the choice that determines the length of a later group of spans of the same kind, lowering equal values together, and sorting spans that have the same label and are part of the same span. Replacing a span with a shorter one of the same label collapses a recursive value, a tree for instance, onto one of its subtrees in one step instead of one level at a time. A counterexample for a property such as "the sum of the elements of a list does not exceed 1000" has a sum just above 1000 once as many elements as possible were deleted, and then deleting or lowering any element makes the property hold. A value whose span consists of one choice is therefore moved into the later values whose spans have the same label, the last ones first, as far as the largest choice that these could have been allows, so that the earlier values become zero and their elements can be deleted. Values that are adjacent in the choice sequence, the characters of a string for instance, are left out, because moving a value from character to character makes a string more complicated rather than shorter. With int<0, 100> elements, such a counterexample is shrunk to [1, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100]; the same pass shrinks a counterexample for $a + $c > 10 to $a = 1; $c = 10; even when other values are drawn between them. Redistributing moves as much as possible from the first choice of a pair to the second while the property stays falsified, so that a counterexample for a property such as $a + $b > 10 is shrunk to $a = 0; $b = 11; rather than $a = 11; $b = 0;, which no pass that changes one choice at a time can achieve. Lowering equal values together lowers the corresponding choices of all values that were generated from equal choices at the same time, so that a counterexample for a property such as "a list has no duplicates" is shrunk to [0, 0] rather than [-9, -9], because lowering one of two equal values on its own makes them unequal. Lowering a choice by one while raising one of the next three choices to the largest value it can have, which the choice source clamps a larger choice to, shrinks a counterexample for a property such as $a * $b < 1000 with int<0, 100> parameters from $a = 20; $b = 50; to $a = 10; $b = 100;; the other passes then lower the raised choice again. A choice can determine how many values are drawn after it, the size of a list followed by an index into it, for instance. Lowering such a choice alone makes the choices of the elements that are no longer drawn be read as the index, and deleting the choices of elements alone makes the list read the choice of the index as an element. The choice is therefore lowered by an amount while as many of the last spans of a group of spans that follow it, are part of the same span, and have the same label are deleted, so that a property with an int<1, 10> parameter $size that draws a list of exactly $size integers and then an index into it is shrunk to $size = 2, [0, 0], and the index 1. When deleting the last spans does not keep the property falsified, because the value that falsifies it is one of them, the first spans are deleted instead. The amount is doubled after each success. The same pass shortens strings whose length is drawn before their characters, such as the labels of the domain names and the local parts of the email addresses that Generators::domainNames() and Generators::emailAddresses() generate, whose characters are spans for this reason: a counterexample for a property such as "an email address has no +" is shrunk to +@a.aa rather than aaaaaaaaa+@a.aa. What a deleted span contributed to the counterexample can be needed to keep the property falsified, though, the length of a domain name to which a label contributes, for instance. When deleting a span, or lowering a choice while deleting spans, does not keep the property falsified, the choice right before the first span of a later group of spans of the same kind, which determines how many spans that group has, is raised as well: by as many spans as were deleted, or by the number of choices of a deleted span, which the other passes lower again as far as the property stays falsified. A counterexample for a property such as "a domain name has at most 10 characters" is thereby shrunk to a.aaaaaaaaa rather than to aa.aaaaaaaa or a.a.aaaaaaa, because the characters of a label that is shortened or deleted are moved to the top-level domain. Sorting spans of the same label makes the elements of a list, or parameters that are generated the same way, appear in the order that makes the counterexample simplest when their order does not matter. The passes are repeated until none of them succeeds or the budget is exhausted. Each round starts with the passes that made the counterexample simpler most often in the round before, so that the attempts are spent on the passes that are most likely to succeed.
• In the same way means that the Throwable that caused the failure or error has the same class and was raised at the same location. The location is the one PHPUnit reports for the Throwable: the first frame of its stack trace that is not in PHPUnit's own code, or else the file and line of the Throwable itself. The file and line of the Throwable alone would not do, because every failed assertion is raised in PHPUnit's own code. This keeps shrinking from switching to a different bug. A shrinking attempt that falsifies the property in another way is not kept, but it is not forgotten either: for each other way, the simplest choice sequence that was seen is remembered (section 6.5). An attempt that was aborted because the deadline of the test run has passed, or that ended before it was prepared, does not count as another way.
• A candidate falsifies a property that timed out in the same way if it also exceeds the time limit, regardless of where it was interrupted. Because every such attempt costs the full time limit, shrinking after a timeout uses a separate budget of few attempts.
• The budget is the maximum number of shrinking attempts (default 1000, section 9). When it is exhausted, the simplest counterexample found so far is reported, together with a note that shrinking was stopped.
• Because generators do not implement shrinking, composed generators and generators created with map(), filter(), and flatMap() shrink without any additional code.

I chose shrinking on choice sequences, the approach that Hypothesis introduced, because it makes shrinking independent of how a generator is composed and because it turns every counterexample into a list of integers that can be stored (section 7.3). Every shrinking attempt is a complete test, so the number of attempts matters more here than for a library that calls a closure.

6.4 Arguments and test identity

• Examples receive their arguments through a new internal mechanism, not through TestCase::setData(). The Event\Code\TestMethod value object of every example is the property's value object: it has the property's ID and name and no data set.
• As a result, JUnit XML, TeamCity, Open Test Reporting, TestDox, code coverage, the test run history, and the baseline always see a stable identity for a property, regardless of which examples were generated.

6.5 Final replay

After shrinking, the minimal counterexample is replayed once more. The events of this replay are the ones reported for the property. If the replay does not falsify the property, the property is reported as an error stating that it is flaky (its outcome does not depend on its arguments alone), together with the failure that was originally observed. If the arguments of the minimal counterexample cannot be generated again, because a generator throws, the property is reported as an error with that exception.

Then the remembered choice sequence of each other way in which the property was falsified while it was shrunk (section 6.3) is replayed, so that its arguments, drawn values, and steps can be exported. A sequence that does not falsify the property in the same way again, or whose arguments cannot be generated again, is not reported. These counterexamples are the simplest ones that shrinking came across; they are not shrunk on their own. Their events are discarded, like those of shrinking attempts.

6.6 Events of examples

The events of examples, shrinking attempts, and the final replay are collected rather than dispatched, using the mechanism IterativeTestSuite::runCollectingEvents() already provides. What subscribers see is this:

• Exactly one sequence of Test\PreparationStarted ... Test\Finished per property: the one of the final replay for a property that a generated example falsified, the one of the explicit example for a property that an explicit example falsified, the one of the last example otherwise. Its Test\PreparationStarted and Test\Prepared events are those of the first example, so that the duration of the property that loggers and the test run history measure covers all of its examples, its shrinking, and its final replay.
• All issue events (deprecations, notices, warnings, errors, and PHPUnit's own issues) and Test\ConsideredRisky events of the explicit examples and of the examples checked during generation are dispatched within that sequence, so they are aggregated per property exactly as for any other test. Issues triggered during shrinking are not reported.
• The Test\AdditionalInformationProvided events that provideAdditionalInformation() emits are dispatched only for the example whose sequence is reported, and only for a property that did not pass: for a falsified property, they are those of the minimal counterexample or of the explicit example that falsified it. For a property that passed, they would be those of an arbitrary example.
• A Test\ConsideredRisky event or an issue of PHPUnit's own with the same message is dispatched only once per property. Whether a property performed no assertions is decided on the sum over its examples, not for each example.
• The number of assertions reported for the property is the sum over the explicit examples and the examples checked during generation.
• Test\PropertyChecked (section 8.1) is dispatched before Test\Finished. Test\PropertyFalsified is dispatched before the Test\Failed or Test\Errored event of a falsified property.

6.7 Interplay with other features

• Process isolation: not supported in the first iteration. A property is run in the main process (section 3.3).
• Time limits: the time limit for the test's size applies to each example, each shrinking attempt, and the final replay, not to the property as a whole. Without a time limit, an example for which the code under test does not terminate hangs the test run. The documentation of #[Property] therefore recommends using either the --timeout CLI option, which limits the wall-clock time of the entire test run, or the --enforce-time-limit CLI option together with the #[Small], #[Medium], or #[Large] attribute on the test method. Only --enforce-time-limit gives each example its own time limit, which is the time limit that sections 6.2 and 6.3 refer to; --timeout sets a single deadline for the entire test run.
• Code coverage: coverage is collected for the explicit examples and the examples checked during generation and attributed to the property. It is not collected during shrinking and the final replay.
• Execution order, --stop-on-*, --filter, --list-tests: the property is treated as one test.
• TestDox: the property is listed like any other test. The TestDox output for a falsified property includes the minimal counterexample. Placeholders for parameters in #[TestDox] are not replaced for a property, because its name must not depend on the generated examples.

7. Randomness and reproducibility

7.1 Run seed

• Each test run has one seed. It is set with the --random-seed <N> CLI option, which PHPUnit 13.5 introduces (Add the --random-seed <N> CLI option and deprecate --random-order-seed CLI option #7049). Otherwise, the current Unix time is used.
• With the --random-seed-from-git-commit CLI option, the seed is derived from the Git commit that is checked out in the working directory instead: it is the number that the first 15 hexadecimal digits of the hash of the commit make. Every test run of a commit then uses the same seed, so that a build on a continuous integration server that fails because of a counterexample fails again when it is run again, instead of passing with another seed. The commit is determined with sebastian/git-state. --random-seed takes precedence. When no commit is checked out, or Git cannot be run, the option is ignored with a PHPUnit warning, and the current Unix time is used.
• The run seed is the seed for random test order as well as for properties, so that one seed is enough to reproduce a test run. --random-order-seed is deprecated in favour of --random-seed (Add the --random-seed <N> CLI option and deprecate --random-order-seed CLI option #7049).
• The seed is printed with the other runtime information (Random Seed: <N>) when at least one property is checked, when the tests are run in random order, or when --random-seed or --random-seed-from-git-commit is used. A seed that was derived from a commit is printed like any other, so that the test run can be reproduced with --random-seed on another commit, too.

7.2 Property seed

• Each property's random number generator is seeded with a value derived from the run seed and the property's test ID.
• Filtering and execution order therefore do not change the examples a property sees.
• Properties use Random\Randomizer with the Random\Engine\Xoshiro256StarStar engine. Generating examples neither uses nor modifies the global state of mt_rand(), rand(), and the other functions that use PHP's global random number generators.
• When --random-seed or --random-seed-from-git-commit is used or the tests are run in random order, PHPUnit seeds PHP's global random number generators before every test with a seed that is derived from the run seed and the test ID (Seed random number generators using --random-seed <N>, regardless of whether PHPUnit needs random number generation #7051). The examples of a property share its test ID (section 6.4), so code under test that uses these functions gets the same random numbers in every example, every shrinking attempt, and the final replay.

7.3 Remembered counterexamples

• When test run history is recorded (recordTestRunHistory), the choice sequences of the counterexamples of a falsified property are stored in the test run history under the property's test ID: the one of the minimal counterexample first, followed by the one of each other way in which the property was falsified (section 6.5). A choice sequence is a list of integers, so it can be stored whatever the types of the arguments, the drawn values, and the arguments of the steps of a state machine are.
• The next time the property is checked, the stored choice sequences are replayed, in this order, before new examples are generated. They are replayed without drawing from the property's random number generator, so the examples that are generated afterwards are the same as without them. When the bug behind the first sequence has been fixed, the next one still falsifies the property. If the property holds for all of them (and for all newly generated examples), the stored sequences are removed.
• If a generator has changed in the meantime, a stored sequence simply produces a different example. This is harmless.
• When an explicit example falsifies the property, the stored sequences are kept: they were not replayed, because the explicit examples are checked first.

8. Reporting

8.1 Events

Two new events are added. Existing events are not changed.

• PHPUnit\Event\Test\PropertyChecked: the test, the property seed, the number of examples checked, the number of examples rejected, for each label with which examples were classified (section 3.6) the number of examples, for each label with which values were observed (section 3.6) their count, minimum, median, mean, 90th percentile, and maximum, and for each label with which scores were targeted (section 3.8) the highest score. Dispatched for every property. Explicit examples are not counted.
• PHPUnit\Event\Test\PropertyFalsified: the test, the property seed, the number of examples checked before the property was falsified, the number of shrinking attempts, whether the shrinking budget was exhausted, the choice sequence of the minimal counterexample, the minimal counterexample: for each parameter its name and the exported value, the values that it drew with draw(), exported and with their labels, the steps of the state machine that it ran with runStateMachine(), written as PHP code, and, for each other way in which the property was falsified while it was shrunk, a description (the class of the Throwable and where it was raised, or that the time limit was exceeded), its choice sequence, its arguments, its drawn values, and its steps. The event also has how often the example that first falsified the property was replayed before its counterexample was shrunk, and how often it falsified the property again in the same way (section 6.3). When an explicit example falsified the property, the event also has the name of that example, and the counterexample is its arguments; no examples were generated, there were no shrinking attempts, there were no replays, and the choice sequence is empty.

8.2 Exporting counterexamples

• A value that was generated through a class's constructor is exported as a PHP expression that creates it, for instance new Money(amount: 1, currency: Currency::EUR). Enumeration cases are exported as Currency::EUR. Points in time are exported as new DateTimeImmutable('1970-01-01T00:00:00+00:00'), or as new DateTimeImmutable('2020-03-29T03:00:00', new DateTimeZone('Europe/Berlin')) when their time zone has an identifier, with microseconds when they have any. When daylight saving time ends, the local times of the hour before occur twice, and PHP creates the later of the two points in time from them. The earlier one is therefore exported as (new DateTimeImmutable('2020-10-25T02:59:59+02:00'))->setTimezone(new DateTimeZone('Europe/Berlin')). Scalars and arrays are exported as PHP literals.
• An object that was not generated, for instance an event that a rule of a state machine returned, is exported as a PHP expression that creates it when it is an object of a user-defined class without a parent class whose properties are all readonly promoted constructor parameters: new MoneyDeposited(account: new AccountId(value: 'account-1'), amount: 1). Value objects and domain events are usually declared this way, and such an object cannot refer to itself.
• Other objects are exported completely with sebastian/exporter, not in its abbreviated form.
• Class names are written the way the file that declares the property imports them, or fully qualified when it does not import them, so that the counterexample can be pasted into a data provider in that file.
• In Data Provider or Properties? I recommend turning a counterexample into a test with a data provider, so that it guards against a regression. With this output, that is a matter of copy and paste.

8.3 Output and logs

• The default result printer shows, for a falsified property, the number of examples, the number of shrinking steps, and the minimal counterexample as shown in part 1, below the failure message: its arguments, followed by the values it drew and the steps of its state machine. Then it shows the counterexample of each other way in which the property was falsified:

Property falsified after 2 examples, counterexample shrunk in 6 steps:

    $value = 500;

Property also falsified by RuntimeException at /path/to/tests/FooTest.php:26 with this counterexample:

    $value = 251;

• When the minimal counterexample ran a state machine, its steps follow, as PHP code that calls the methods of the state machine, which is referred to as $machine (section 3.9). A value that a step produced is assigned to a variable that is named after its pool and numbered, for instance $accounts1, and later steps refer to it by that variable. The value that another step returned is shown as a comment unless it is null. An invariant is shown only when it failed, with the comment // invariant failed.
• When the example that first falsified the property did not falsify it again in every one of its replays before shrinking (section 6.3), the report says so below the minimal counterexample:

The example that falsified the property falsified it again in only 2 of 4 replays, so its outcome does not depend on its arguments alone.

When it did not falsify the property again in any of them, the property is reported as flaky instead, with its counterexample, which was not shrunk, and the line The example that falsified the property did not falsify it again in 4 replays.

• For a property that an explicit example falsified, it shows the name and the arguments of that example instead (section 3.5).
• JUnit XML appends the same text to the content of the <failure> or <error> element. TeamCity appends it to details. Open Test Reporting includes it in the reason for the failure. TestDox shows it for the property.
• --debug prints Test Property Checked and Test Property Falsified lines.
• With --display-property-statistics, the default and the compact output show for every property how many examples were checked and rejected, the percentage of the examples that were classified with each label, the summary of the values that were observed with each label, and the highest score that was targeted with each label:

1) ScheduleTest::testScheduleFitsIntoTheDay
100 examples checked, 0 rejected

     12.0% empty
      3.0% overnight

    entries: count 100, min 0, median 3, mean 4.27, p90 11, max 25

    total duration: highest targeted score 1395

9. Configuration

XML configuration file, new optional element:

<propertyBasedTesting examples="100"
                      examplesMultiplier="1"
                      maximumShrinkingAttempts="1000"
                      examplesCheckedUntilHealthIsAssumed="10"
                      maximumRejectedExamplesUntilHealthIsAssumed="50"
                      maximumTooLargeExamplesUntilHealthIsAssumed="20"/>

The last three attributes configure when examples are no longer generated because too many of them were rejected (section 6.2): while fewer than examplesCheckedUntilHealthIsAssumed examples were checked, generating stops once maximumRejectedExamplesUntilHealthIsAssumed examples were rejected, or once maximumTooLargeExamplesUntilHealthIsAssumed examples were rejected because they were too large. With examplesCheckedUntilHealthIsAssumed="0", examples are generated regardless of how many were rejected, up to the maximum number of attempts (section 6.2).

CLI options:

• --random-seed <N>: seed for the test run (section 7.1). This option already exists in PHPUnit 13.5, where it seeds random test order and PHP's global random number generators (Add the --random-seed <N> CLI option and deprecate --random-order-seed CLI option #7049, Seed random number generators using --random-seed <N>, regardless of whether PHPUnit needs random number generation #7051)
• --random-seed-from-git-commit: derive the seed for the test run from the Git commit that is checked out (section 7.1), unless --random-seed is used
• --property-examples <N>: default number of examples per property, overriding the configuration file. #[Property(examples: ...)] takes precedence
• --property-examples-multiplier <N>: multiplies the number of examples of every property, including properties with #[Property(examples: ...)], overriding the configuration file. This is meant for running properties with more examples than usual, for instance ten times as many in a nightly build, without losing the relative weight of properties that specify their number of examples. The examples of a property are generated from its seed in the same order, regardless of how many of them are checked, so a test run with a multiplier checks the examples of a test run without it, and more
• --display-property-statistics: display the statistics of the properties (section 8.3), also configured with the displayPropertyStatistics attribute of the <phpunit> element

10. Implementation outline

• Metadata: Metadata\Property, Metadata\GeneratorFor, Metadata\UseGenerators, Metadata\RequireClassification, Metadata\ForAll, which represents both #[ForAll] and #[ForAllExternal], and Metadata\Rule, Metadata\Initialize, Metadata\Invariant, and Metadata\FromPool for state machines. Metadata for parameter attributes is new for the metadata parser.
• Test building: TestBuilder creates a PropertyTestSuite for a property. This class extends IterativeTestSuite. It receives a factory closure that creates fresh test case instances, like the factory used for RetryTestSuite. TestSuiteBuilder in the event system gets a new value object for it: with only its existing support for IterativeTestSuite, a property would be reported as a repeated test method.
• Test case: an internal way to pass generated arguments, and the choice source for drawn values and the steps of state machines, to an example without changing its identity (section 6.4), and assume(), classify(), observe(), target(), draw(), and runStateMachine(). Examples are never run in a separate process.
• Property-based testing internals in PHPUnit\Framework\PropertyBased: the choice source, the generators, the parser for PHPDoc types, type derivation, the targeted search, the shrinker, and counterexample export. State machines are validated and run in PHPUnit\Framework\PropertyBased\StateMachine: a builder validates the attributes of a state machine and creates the generators for the parameters of its steps with the deriver of the property, and a runner draws the steps from the choice source of the example and records them for the report.
• Test runner: code coverage is not started for shrinking attempts and the final replay.
• Test run history: storage of the choice sequences of the counterexamples of a property per test ID.
• Events, loggers, and printers: the new events and their rendering (section 8).
• Dependencies: PHPDoc types are parsed by PHPUnit itself (section 4.1). Class names are resolved with nikic/php-parser, which PHPUnit now requires directly instead of relying on phpunit/php-code-coverage requiring it: its NameResolver provides the namespace and the imports that apply to the class that declares a property, and the same information yields the shortest name of a class for exporting counterexamples (section 8.2). The commit from which --random-seed-from-git-commit derives the seed (section 7.1) is determined with Builder::commit() of sebastian/git-state 1.1, which PHPUnit already requires for the Git information in Open Test Reporting XML logfiles. phpunit/php-code-coverage 14.5, which is not released yet, is required because its CodeCoverage::append() accepts the size of a test: PHPUnit passes the size of a property when it appends the code coverage data of examples that it held back until the property was checked, so that unintentionally covered code is reported according to the size of the property.

11. Out of scope for the first iteration

• Process isolation for properties. Running all examples, the shrinking, and the final replay of a property in one separate process requires collecting events inside the child process, which PHPUnit does not support yet.
• Properties that depend on other tests. This requires deciding how the return values of the tests a property depends on are passed to it in addition to its generated arguments, and whether each example receives its own copy of them.
• Running the steps of a state machine concurrently and checking that the outcome can be explained by some sequential order of the steps. Sequential state machines find the bugs that depend on the order of commands, such as a projection that catches up at the wrong moment, but not race conditions.
• Weights for the rules of a state machine, and a way for a rule to declare that its command is expected to be rejected. Swarm testing varies which rules dominate an example, and a rule can assert a rejection with try and catch.
• Generators for values that are valid but not spelled canonically, for instance uppercase UUIDs, braced UUIDs, or IPv6 addresses with leading zeros, for testing normalization, and generators for values that are almost valid, for testing that a validator rejects them.
• Checking at runtime that values produced by an explicitly specified generator match the parameter's PHPDoc type.
• A public low-level API for custom generators that draw choices directly.
• Explicit examples for properties that draw values with draw() or run a state machine, and a way to replay a counterexample from its choice sequence, for instance an attribute that takes it, which would also cover counterexamples whose objects sebastian/exporter cannot export as PHP code.
• Shrinking the counterexamples of the other ways in which a property was falsified on their own.
• An adapter that makes Eris generators usable as ValueGenerator instances. If the Eris maintainers are interested in this, I would be happy to support it.

12. Backward compatibility

• Adding assume(), classify(), observe(), target(), draw(), and runStateMachine() to TestCase is a backward compatibility break, this is why this issue targets PHPUnit 14