Repository navigation
Parallel test execution - #6784
sebastianbergmann wants to merge 120 commits into
Conversation
API Surface ChangesIf any of the additions below are not intended as public API, mark them with New API SurfaceClasses
Methods
Modified API SurfaceMethods
|
9a08909 to
efcd7ee
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6784 +/- ##
============================================
+ Coverage 99.50% 99.55% +0.04%
- Complexity 9988 10899 +911
============================================
Files 968 1005 +37
Lines 30292 32829 +2537
============================================
+ Hits 30142 32682 +2540
+ Misses 150 147 -3 ☔ View full report in Codecov by Harness. |
|
My 2 cents on the topic:
Parallel execution can't and will never be deterministic, a tiny difference between how much a unit lasts results in the following test being run in worker X instead of Y. |
|
Thank you, @Slamdunk, for taking the time to write all of this up. There's a lot of hard-won experience in here, and I appreciate it. I want to set expectations honestly, though: for me, native parallel test execution in PHPUnit is still just an idea, and this pull request is a (remarkably well-working) proof of concept rather than a roadmap commitment. If I ever decide to be serious about this and actually ship it, the feature will have limitations. And those limitations must (and will) be very well documented. Complaints about them will then be kindly refused. 🙂 Even in that case, I neither intend nor expect this to replace dedicated solutions such as ParaTest or Paraunit. The scope I have in mind right now is roughly: PHPUnit supports parallel execution for well-architected test suites that properly deal with resource-usage conflicts like databases, etc. So several of the needs you describe, coordinated bootstrapping, functional/method-level distribution, deterministic replay of a run, are exactly the kind of thing that lives outside that scope and is better served by the tools built specifically for it. That said, your notes are genuinely useful for thinking about where the boundaries of that scope should sit, so thank you again. |
4e03c18 to
5053a2d
Compare
5053a2d to
cf8f973
Compare
|
I have briefly looked into whether the new I/O polling API that PHP 8.6 introduces (RFC: poll_api) could replace the file-based completion polling used by the parallel test runner. The conclusion is that it cannot, because it has the exact limitation that forced the file-based design in the first place: on Windows, it cannot wait on
However:
In other words, Therefore, the uniform file-based completion polling stays. This decision should be revisited only if a future PHP version ships a poll handle that can wait on pipes or process handles on Windows, which is what this feature actually needs. |
cf8f973 to
1e97c6b
Compare
|
I see that it's up to the user to select how many worker to use. The You might be interested in giving it a try, so PHPUnit can provide |
I am aware of |
04cb38d to
43341d8
Compare
908d59e to
4b04c6a
Compare
2c7a98a to
c034df8
Compare
|
e9e9866 to
7c6f829
Compare
…cted for from a parallel worker, instead of the whole code coverage object with its filter and target map
…r class in the main process, and ship the passed tests of a parallel worker without their return values, so that a return value that cannot be serialized, or cannot be unserialized in the main process, no longer breaks a parallel run
…its standard input, which is closed, so that a test that reads its standard input, or starts a process that reads it, does not wait forever for input
…orker process with n instead of PHP_EOL, and start a PHP process instead of cat in the standard input test, so that both pass on Windows
…worker on Windows, as the other JUnit XML tests with errors are, because the logger writes the line ending of the platform after the name of a test
…r its workers, and report the test that a worker was running when the deadline passed as aborted, so that --timeout ends a parallel run at the deadline as it ends a sequential run
…g or reading them, instead of suppressing the errors, because an error handler that the bootstrap script registers is called for suppressed errors, too, and may abort the test run
…l run stops between two tests when the results call for it, as a sequential run does, and close the test suites that a unit left open when the run stops while the unit is executing
…f killing their workers, so that the running test finishes and tearDown(), tearDownAfterClass() and the --CLEAN-- section of a PHPT test still run, as they do when a sequential run stops
…unning only on Linux and macOS, because on Windows reading the output of the section's child process blocks until the process has exited
…unning a unit that was not retried once the next unit is dispatched to it, instead of leaving the worker dead and reporting the units that never ran as crashed once every worker has died
… per appearance instead of folding all of its tests into one unit, so that setUpBeforeClass() and tearDownAfterClass() run once per appearance, as they do in a sequential run
… when the next unit is dispatched to it instead of right after it completes a unit, so that --recycle-workers-after also takes effect across <testsuite> elements
…sts that are still executing when an exception ends a parallel run, so that the run neither waits for the workers to finish their units nor leaves the units' files and PHPT child processes behind
…ly test class it holds in a parallel run, instead of mistaking the <testsuite> for the class's own test suite, so that the suite hierarchy matches that of a sequential run
…picked, such as with --run-test-id, when tests are run in parallel, instead of all of them
…attempt of a PHPT test throws inside the generator that orchestrates them, so that the events collected for the repetition or attempt are forwarded and the collection of events ends before the exception propagates
…shutdown failed in the parallel workers once, delivered when a worker is stopped rather than with its first unit, and reject a subscriber of an unknown type that such an extension registers in a worker, as the child process of a test that runs in a separate process does
…ess has died, so that the events of a test that finished just before the worker died are reported instead of being lost
…ond instead of treating its duration as unknown when the units of a parallel run are ordered for dispatch, so that units of sub-millisecond tests are no longer dispatched before the units of slow tests
…arallel run instead of from the callbacks through which the worker pool and the PHPT runner hand over finished units, and advance both while a test that runs in a separate process waits for its child process, so that idle workers are given new units and the durations of PHPT tests are not inflated
…lel run be destructed as soon as the test has run, as in a sequential run, by removing the tests from the test suite once the run has been planned and letting go of the units that run in the main process before their tests are run
…nly when a test could actually be started, instead of in every polling round, most of which find every slot taken
… when the process budget it shares with the worker pool has room for fewer tests than its concurrency, so that the longest PHPT test is no longer started only after the tests the ordered output waits for have run out
… instead of pipes, so that polling a --SKIPIF-- or --CLEAN-- section that runs in a child process no longer blocks the parallel run on Windows, where reading from a pipe waits until the child has ended
…ack and unpack the 64-bit length prefix of the frames in which the workers of a parallel run stream their events
…n, or in its separate processes, have no worker identity, so that a fixture that falls back to a value when PHPUNIT_WORKER_ID is not set does not fall back to the ID of a worker that runs alongside them
…events of a retried test are collected in a worker of a parallel test run, so that the expectations for deprecations of an attempt are verified against the deprecations triggered in that attempt, as they are in the main process
…ate process instead of matching the metadata from which the test builder already configured the test, so that the parallel test runner decides this exactly as the sequential test runner does
7cc85bf to
ea792e6
Compare
PHPUnit can now execute a test suite across several worker processes concurrently instead of one test after another. Parallel execution is opt-in via a new
--parallel=<n>command-line option and changes nothing when it is not used: the sequentialTextUI\TestRunnerremains the default andTextUI\ParallelTestRunneris selected only when<n> > 1.Native parallelism here is a generalization of process isolation rather than a new subsystem: a worker reconstructs and runs a unit of work and ships its outcome home in the very same serialized envelope that process isolation already uses, and the parent replays that envelope through its normal event pipeline. As a result the parent process remains the single source of truth for all output, logging, results, and code coverage, which are produced exactly as in sequential mode.
How it works
#[BeforeClass]/#[AfterClass]and intra-class ordering. ADataProviderTestSuite, and theIterativeTestSuitethat carries the repetitions of a repeated test or the attempts of a retried test, travel to the worker as atomic members of their class' unit, so their suite envelopes nest in logger output exactly as they do sequentially.<testsuite>elements of an XML configuration are run one after another, just as in sequential mode: only tests that belong to the same top-level test suite ever run concurrently.Runner\Parallel\WorkerPoolownsnpersistent workers and pulls from a dynamic work-stealing queue, so the load self-balances against stragglers. A worker signals completion through the filesystem, which the parent polls;stream_select()is not used, because it does not work on the workers' output pipes on Windows.Runner\Parallel\Schedulerdispatches units in the order of the durations recorded by the test run history, longest first, so that the longest-running work starts as early as possible instead of becoming the straggler the pool waits for. A unit with no recorded duration goes first. Only the dispatch order is affected; results are released in suite order either way.Runner\Parallel\ResultAggregatorbuffers each unit and releases it only once every preceding unit (in suite order) has been released, so the event stream, and therefore every output format, including the default progress output, is byte-for-byte what sequential mode produces. Streamed events of the unit that is next in suite order are forwarded immediately; those of later units are buffered until their turn.PHPUnit\Framework\TestCaseinstances and cannot be reconstructed in a worker (and nesting their child processes inside workers hung on Windows), soRunner\Parallel\PhptRunnerruns them side by side in the main process, each as its own child process.ProcessBudget, so--parallel <n>never executes more than<n>units at once, no matter how a chunk is composed.Running in the main process
Some tests cannot, or must not, run in a worker. These run in the main process at their correct suite position — the aggregator invokes them while releasing results, so global ordering is preserved — and their execution there is ordinary sequential execution:
#[DoNotRunInParallel]— a new attribute, valid on classes and methods, for tests that must not run alongside others (for instance because they share a machine-global resource). Such a unit runs alone: the worker pool and the PHPT runner first finish what they are executing and start nothing new until it is done.#[RunInSeparateProcess],#[RunTestsInSeparateProcesses], or global--process-isolation) — a shared worker cannot provide isolation, but the main process spawns the isolated child as usual.#[Depends]— the result they depend on is produced by a different unit and is only visible in the main process, once that unit has been released.serialize()throw) as well as resources (whichserialize()silently degrades to0).TestCasenor a PHPT test.Because a work unit is a whole test class, a single test method carrying
#[DoNotRunInParallel], requiring isolation, depending on another class, or providing non-serializable data takes its entire class out of the parallel phase.PHPT tests declare their concurrency constraints with a
--CONFLICTS--section instead, since a PHPT file cannot carry PHP attributes: while a test holding conflict keyKruns, no other test declaringKis started, and the reserved keyallmeans the test runs entirely on its own.Robustness
The worker process running X ended unexpectedly), the remaining units are redistributed across the surviving workers, and the run accounts for all of them.--stop-on-*. As soon as the collected results call for a stop, the aggregator releases nothing further and the units still executing are terminated (a grace period, then a forced kill). A PHPT test whoseFILEsection was terminated still runs itsCLEANsection, and one terminated duringSKIPIFdoes not start itsFILEsection.New public API
--parallel=<n>CLI option (also listed in--help); a value that is not a positive integer is ignored with a test runner warning.#[PHPUnit\Framework\Attributes\DoNotRunInParallel](TARGET_CLASS | TARGET_METHOD), with full metadata-layer support.--CONFLICTS--PHPT section, as supported by the PHP project'srun-tests.php: one conflict key per line,#starts a comment, blank lines are ignored, andallis reserved.PHPUNIT_WORKER_ID— the small, stable ordinal (0,1,2, ...), ideal for indexing a fixed set of pre-provisioned resources. A worker restarted after a crash keeps its ordinal.PHPUNIT_WORKER_TOKEN— a value of the form<id>_<random>that is unique across workers and across runs, for resources that must not collide with those left behind by a previous run. A restarted worker gets a fresh token.Event\TestRunner\ChildProcessReason::ParallelWorker, so that the events about child processes say why one was used.Shared with the sequential runner
Rather than duplicating the sequential runner, behavior the two must perform identically was extracted so it is expressed exactly once:
TextUI\TestRunnerLifecycle— the test runner lifecycle both runners drive.Event\Dispatcher\CollectionWindowand the facade's dispatcher selection — the window during which events are collected instead of dispatched.Framework\TestRunner\ChildProcessBootstrapplus two templates — the boot code every kind of child process shares; the configuration and source map are written once per run for all of them, so isolated processes and parallel workers cannot drift apart.Framework\TestRunner\ChildProcessResultEnvelope— the result envelope's encoding and decoding, shared by its consumers.TestRunner\ExecutionFinished, instead of every time an outermost test suite finishes.Results for PHPUnit's own test suite
Measured on a 6-core machine:
--parallel 10--testsuite unit(5813 tests)--testsuite end-to-end(1257 tests)Both suites report the same numbers of tests, failures, and skipped tests in both modes, stable across runs. The end-to-end suite benefits the most because it is dominated by child processes rather than by CPU work in the main process.
Which worker runs which test class is not deterministic; the work-stealing scheduler assigns classes by timing, but the aggregated output is identical regardless.
Notes and limitations (possible follow-ups)
#[DoNotRunInParallel].