Seed random number generators using --random-seed <N>, regardless of whether PHPUnit needs random number generation #7051

PHPUnit seeds PHP's Mt19937 random number generator, which mt_rand(), rand(), shuffle(), str_shuffle(), and array_rand() use, only when tests are run in random order. When --random-seed <N> (#7049) is used and tests are not run in random order, the seed is not used at all, and PHPUnit triggers a warning.

Code that uses these functions gets different numbers in every test run, so a test failure that depends on these numbers cannot be reproduced. Unlike a Random\Randomizer object, the state of these functions is global to the PHP process and cannot be injected into the code under test. A test can call mt_srand() itself, but then it gets the same numbers in every test run, and the state it leaves behind leaks into the tests that run after it.

PHPUnit 13.5 will therefore:

• Seed PHP's Mt19937 random number generator when --random-seed <N> is used, regardless of whether tests are run in random order or whether any other PHPUnit feature needs random numbers
• No longer trigger the --random-seed is only used when execution order is "random" warning
• Print Random Seed: <N> and write the phpunit:randomSeed element to the Open Test Reporting XML logfile whenever it seeds the random number generator
• Seed the random number generator in the child processes of tests that are run in process isolation, too
• Change the description of --random-seed <N> in the --help output to Use the specified seed for PHP's Mt19937 random number generator (mt_rand(), rand(), shuffle(), ...) and for random test order

The deprecated --random-order-seed <N> option keeps its current behaviour, including the warning when tests are not run in random order. When neither --random-seed nor random order is used, PHPUnit does not seed the random number generator, as before.

This supersedes what #7049 says about --random-seed triggering a PHPUnit warning when tests are not run in random order: --random-seed is always used and never triggers this warning. Only --random-order-seed still does.

When the random number generator is seeded

Today, PHPUnit calls mt_srand() once, in PHPUnit\TextUI\TestRunner::run(), before the tests are sorted. Doing the same for --random-seed would only make a test run reproducible as a whole. The numbers that a test sees depend on everything that consumed numbers from the global state before it:

• The shuffle() call that sorts the tests in random order (PHPUnit\Runner\ExecutionOrder\Stage\Randomize)
• The tests that ran before it in the same process
• PHPUnit itself: the test double generator calls mt_rand() for the class name the first time a test double of a given configuration is created in a process (PHPUnit\Framework\MockObject\Generator\Generator::generateClassName() and the name of intersection interfaces). A test that creates a stub before it calls the code under test therefore gets other numbers depending on whether an earlier test already created the same kind of stub

Two cases are not covered at all: the child processes of tests that are run in process isolation are not seeded, and data provider methods are called while the test suite is loaded, before mt_srand() is called.

This is what --random-order --random-seed=1234 does with PHPUnit 13.5-dev:

• A test that is run on its own using --filter gets other numbers than it does in the full test run
• A test that is run in process isolation gets other numbers in every test run
• A data provider method gets other numbers in every test run
• A test that creates a stub before it calls mt_rand() gets other numbers than the same test without the stub

The goal is to reproduce a failing test, and that usually means running it on its own. Therefore, PHPUnit should seed the random number generator before each test, with a seed that is derived from the seed of the test run and the ID of the test. This is what PHPUnit\Framework\PropertyBased\PropertySeed::derive() already does for the examples of properties (#7008). With this, a test gets the same numbers regardless of filtering, execution order, process isolation, and the tests that ran before it:

• Seed before the before-test methods of a test (seed derived from the test's ID)
• Seed before the before-class methods of a test case class (seed derived from the class name)
• Seed before a data provider method is called (seed derived from the class name and method name of the data provider method)
• Generate the class names of test doubles without consuming numbers from the global state, for instance using random_bytes(), which does not use Mt19937

When the reseeding is done in PHPUnit\Framework\TestCase::prepareTest(), process isolation is covered as well: the child process reaches that method through TestCase::run(), and it is also where the child process calls the before-class methods, so these can be seeded just like when the test is not run in process isolation.

The random order of tests is still computed from the global state that was seeded with the seed of the test run before the tests are sorted, so a given seed results in the same order as before. As the numbers that tests see no longer depend on the shuffle() call, they are different from those in PHPUnit 13.4 for the same seed.

As the seed of a test is derived, mt_rand() in a test does not return the numbers that it returns after mt_srand(<N>). A test that needs these numbers has to call mt_srand() itself. As PHPUnit seeds again before the next test, this no longer affects the tests that run after it.

Other random number generators

mt_srand() is the only seeding function for the functions listed above. srand() is an alias of mt_srand(). PHPUnit would use the default MT_RAND_MT19937 mode, as MT_RAND_PHP has been deprecated since PHP 8.3.

The gmp extension's random number generator is seeded for the same reason as Mt19937: its state is global to the PHP process and cannot be injected into the code under test. Seeding it has no effect on tests that do not use gmp_random_bits() or gmp_random_range(), so the only cost is a call to gmp_random_seed() when the extension is loaded.

Random number generators that PECL extensions provide are out of scope.