Test Impact Analysis by sebastianbergmann · Pull Request #6919 · sebastianbergmann/phpunit

sebastianbergmann · 2026-08-24T10:25:50Z

The changes proposed here implement test impact analysis: PHPUnit records which source files each test depends on, and can then run only the tests a change can affect.

Closes #6897.

What it does

Recording

--record-test-impact-data, or recordTestImpactData="true", records while the tests run which source files each test depends on. The data goes into its own file in the cache directory, separate from the test run history. --do-not-record-test-impact-data turns recording off for a single run, and so does --no-coverage.

Recording needs a cache directory, a code coverage driver that supports line coverage, and <source>. When one of them is missing, PHPUnit warns that test impact data will not be recorded and runs the tests anyway.

Querying

--list-tests-that-depend-on src/Money.php queries what was recorded, without running a single test:

Recorded at 2026-10-04 07:16:57 UTC from what the tests executed and, for a test that ran in a process of its own, what that process loaded.

Tests that depend on /…/src/Money.php as it is now:
 - Example\InvoiceTest::testTotalIsTheSumOfItsItems

Tests that depend on an earlier version of /…/src/Money.php:
 - Example\FormatterTest::testFormatsAmount
 - Example\SumTest::testAdds#0

The two lists are kept apart on purpose: the first describes the code as it is now, the second describes tests recorded against a version of that file which no longer exists. For a file that was executed outside of any test, the answer is that every test depends on it.

Selecting

--only-impacted runs only the tests a change can affect:

Impact:        3 of 4 tests can be affected by what changed; 1 test is not run
Recorded:      2026-10-04 07:16:47 UTC

OK (3 tests, 3 assertions)

The first line says how many tests were left out, the second when what the selection rests on was recorded, and the summary reports the tests that actually ran. Nothing pretends that the test that was left out passed. When the tests are filtered as well, with --filter or --group for instance, the line does not say how many tests are not run, because the filter decides that afterwards.

When no test can be affected by what changed, no test is run. That is not an empty test suite, and --fail-on-empty-test-suite does not make such a run fail.

PHPUnit works out what changed by comparing what is there now against what it recorded. --impacted-by src/Money.php --impacted-by src/Service tells it instead and it implies --only-impacted. What you name is the change set: the recorded hashes are not consulted at all. That matters on a fresh checkout, where comparing against the recording reports everything that changed since, while git diff --name-only reports what you actually did. A relative path is resolved against the working directory, as version control names it. A directory stands for everything that was recorded beneath it, and a file in a directory that a test declared as a fixture selects that test.

Because naming paths one at a time does not compose with the tool that knows what changed, --impacted-by-file reads a list, one path per line, - being standard input:

$ git diff --name-only | phpunit --impacted-by-file -

An empty list is an answer rather than a missing one: nothing changed, so nothing that depends on code runs. A list that cannot be read is not an empty list, and PHPUnit stops rather than guessing.

--explain-impacted says, without running a single test, which tests a test run would select and why:

Recorded at 2026-10-04 07:16:47 UTC from what the tests executed and, for a test that ran in a process of its own, what that process loaded.

3 of 4 tests can be affected by what changed.

3 tests depend on something that changed:
 - Example\FormatterTest::testFormatsAmount
   /…/src/Money.php
 - Example\InvoiceTest::testTotalIsTheSumOfItsItems
   /…/src/Money.php
 - Example\SumTest::testAdds#0
   /…/src/Money.php

It asks the same question a test run asks, with the same input, so what it reports is what such a test run does. The other reasons it reports are that a test has never been recorded, that it did not pass when it was last run, that it depends on a test that can be affected by what changed, that another test that is run depends on it, and that it is not a test method and can never be recorded.

What is recorded

For each test, PHPUnit records:

the files in <source> that the test executed
the files the test is made of: the file of its test class, of its parent classes, of the traits it uses, and of the class its data provider belongs to
the fixtures it declares with #[UsesFixture] or registers with registerFixture()
when it ran in a process of its own, every file in <source> that this process loaded, whether a line of it was executed or not

A test run that consists of a single test counts as a process of its own. Loading is what shows that a test uses an interface, an enum case or a class constant, none of which executes a line, but only a process that runs a single test shows which test loaded what.

What is executed outside of any test is recorded as well, for the tests it can affect:

what a data provider executes, for the tests that are made from the data it provides
what the methods that are called before the first and after the last test of a test class execute, for the tests of that class
everything else, such as what the bootstrap script, an extension, or loading the test files executes, for every test

The repetitions of a repeated test and the attempts of a retried test are recorded as the test they are repetitions and attempts of. A PHPT test is never recorded.

Files are named relative to the directory of the configuration file, or relative to the working directory when there is none, so that what is recorded in one checkout can be used in another.

Two ways of maintaining the data

Observation needs code coverage data collection and works no matter what a project declares. --derive-test-impact-data-from-coverage-targets is the alternative for projects that already take coverage targets seriously: it works out what each test depends on from the #[CoversClass] and #[UsesClass] attributes it declares, without code coverage data collection, and without the tests having been run even once.

Two settings decide whether that is sound. requireCoverageMetadata makes it complete: the mode is refused unless code coverage metadata is required for all tests. beStrictAboutCoverageMetadata makes it correct: under it, a test that executes code it does not declare is already marked risky, so declarations are a superset of execution. PHPUnit warns when the mode is used without the strict check, because then nothing has ever verified the declarations.

What was recorded in one way is of no use to a test run that records in the other. Such a test run discards it instead of adding to it, and a selection based on it runs every test.

Measurements

All from one real project of 177 tests that enables both requireCoverageMetadata and beStrictAboutCoverageMetadata.

	wall clock
the suite, no coverage driver	5.44s
recording derived from declarations	5.47s
a coverage run, no recording	33.5s
the same coverage run, recording	87.8s

Recording by observation costs 2.6× on a run that was already collecting coverage, and all of it lands in the tests declaring #[CoversNothing], which have to be collected for and are usually the slow ones. Deriving from declarations is free.

For 165 of 165 tests that declare targets, the declarations covered everything the test was observed to execute, with no exceptions beyond #[CoversNothing]. The price is a coarser selection:

changed file	from what tests executed	from what tests declare
the matrix class	129 of 177	138
the tuple class	123	164
the world class	16	25
an output mapper	6	6

Declaration is never below observation on any file, and costs between nothing and half as many tests again. Five of the 177 tests declare no targets, are never recorded in that mode, and therefore always run: that is the floor.

Selecting after editing two source files gave 26 of 177 whether the change set was worked out from the hashes or piped in from git diff --name-only. With nothing edited, --only-impacted selected 0 of 177 in 0.14 seconds. The data is about 150 to 350 bytes per test, 27 KiB for this project.

What you can rely on

Everything the analysis has no reliable information about causes the test to run:

a test that was never recorded, such as a PHPT test, or a test that declares #[CoversNothing] when the data is derived from code coverage targets
a test whose own code changed, or whose data provider changed
a test that did not pass when it was last run
a test that depends, through #[Depends], on a test that can be affected by what changed: what it is given is not something it executes
a test another selected test depends on through #[Depends]

And these cause every test to run:

nothing has been recorded yet, or what was recorded is no longer usable
what was recorded comes from the other way of maintaining the data
a file changed that was executed outside of any test
a source file or a watched file changed that no test is recorded as depending on
a source file is there that was not there, or was not first-party code, when the recording was made
a watched file is there that was not there, or was not watched, when the recording was made
a path that was named, with --impacted-by or in a list, is not among the files that were recorded, or no test is recorded as depending on it

What was recorded is no longer usable when it was recorded with another version of PHPUnit or of PHP, or when one of these changed since: a setting in the configuration that can change what code a test executes, a bootstrap script, what is first-party code, or composer.lock. A PHP setting given with -d counts as configuration. A setting that only changes how results are reported does not.

That matters more than it sounds: what was recorded describes one state of the world, and when that state is not what it was, the honest answer is not that some entries are stale but that none of them can be trusted.

Nothing is kept forever, either. A test run forgets every entry it did not record itself, but only when that run ran every test there is. A run that filtered, that selected by impact, or that stopped at the first failure recorded nothing for the tests it did not run, and nothing tells those apart from tests that are gone, so such a run adds to what is known and takes nothing away. For the same reason, a run that did not run every test does not record a source file that no test refers to as it is now: it did not assess a change to that file, so the next selection still runs every test because of it, and that run records it.

--only-impacted refuses to run at all without a cache directory, or when the test run history is turned off, rather than quietly being less careful.

What it cannot see

Observation shows what a test executes, and some dependencies execute nothing:

Using a class constant, an enum case, an interface, or the default value of an inherited property executes no line of the file that declares it, so a test that only does that is not recorded as depending on that file. As long as no test executes a line of the file, a change to it runs every test. Once another test does, a change to it only runs that other test. A test that runs in a process of its own is the exception, because what its process loaded is recorded.
A system under test that runs in another process, behind an HTTP API or a browser, is not observed at all. registerFixture(), or #[UsesFixture] naming its source directory, is how it gets recorded.
A file outside <source> that a test reads, and that no test declares or registers, is only noticed when it is watched, and then a change to it runs every test.

I decided against closing these gaps with static analysis of test code, helper code or production code. It would be a game of cat and mouse, and PHP's dynamic features would keep it from ever being complete. Running every test on the main branch catches what a selection missed.

`#[UsesFixture]`

Neither observation nor declaration can see a data file: reading invoices.csv is not executing code, and a coverage target names code units. A new attribute closes that:

#[UsesFixture('../fixtures/invoices.csv')]
public static function provideInvoices(): array

The path is relative to the file the attribute is written in. The attribute goes on a test class, a test method, or a data provider method: the last is the one that earns its keep, because a provider shared by many tests declares the file once and every test using it inherits the dependency. On a test class, it counts for the tests of every class that extends it. Directories work too, watched as a whole, so adding a file to one counts as a change. A path that does not exist produces a warning and is ignored, the same treatment a coverage target that cannot be used gets. Unlike coverage targets, nothing can ever verify this attribute; it is a promise the project makes to itself.

`registerFixture()`

An attribute needs the path when the test is written. When it is only known while the test runs, which templates were rendered or which tables were queried, for instance, the test can register it:

$this->registerFixture(__DIR__ . '/../resources/views/invoice.blade.php');

What is registered is recorded together with what the test executed, also when the test runs in a process of its own. A relative path is resolved against the working directory at the time of the call, and a path that does not exist produces a warning. Nothing is recorded for a test that is skipped or incomplete, registering does nothing when test impact data is not recorded, and deriving the data from code coverage targets ignores it.

Watched files

When PHPUnit worked out what changed by itself, a change to a file outside <source> that no test was recorded as depending on did not run a single test, while naming the same file with --impacted-by ran every test. A test helper or a configuration file was invisible that way. Such files can be watched:

<testImpactAnalysis>
    <watch>
        <directory>config</directory>
        <directory suffix=".yaml">config/packages</directory>
        <file>bootstrap/app.php</file>
    </watch>
</testImpactAnalysis>

A watched file is treated like a file in <source> that no test executed. Changing it, removing it, or adding a file where files are watched runs every test. If a test is recorded as depending on the file, only that test is run. <directory> takes prefix and suffix as it does in <source>, and the suffix defaults to .php.

The PHP files in the directories of the test suites are watched without any configuration, except for what a test suite excludes and for the test files themselves. A test file that changed is recorded for its tests already. A test file that was added only brings tests that are run because nothing is known about them, and watching it would make every new test file a reason to run everything.

Decisions

Recording is never automatic, not even when coverage is already being collected. A suite whose slow tests declare #[CoversNothing] would otherwise get 2.6× slower unasked.
Tests declaring #[CoversNothing] are collected for in observation mode, with the targets bypassed so nothing reaches the coverage report. The alternative, treating them as affected by everything, would put a floor under every selected run.
Impact data is never filtered using coverage targets; the report always is. The two disagree by design. Before this, what the coverage report kept was missing 75% and 94% of the dependencies tests really have on the two suites I measured, because the report is narrowed to what a test declares. That is why the data is collected before the narrowing, which needed php-code-coverage 14.4.0.
What is recorded in one checkout can be used in another. Files are named relative to the directory of the configuration file, so the data can be restored from a CI cache into a checkout somewhere else. It is still discarded on a PHPUnit or PHP version change, a patch release of PHP included.
A test's own files are recorded in the impact data rather than checked against the test index at selection time. The index is written at a different moment, so a run that refreshed the index without recording impact data would report a changed test as unchanged.
The assumption hash covers the <source> declarations, not the expanded file list. Hashing the list would throw everything away each time a class is added to src/. A file that was added runs every test once instead: it may be found by code that scans a directory, without any file that changed leading to it.
--only-impacted, --impacted-by and --impacted-by-file are command line options only. A switch that makes every run partial does not belong in a file a project commits.
What is watched is configured in phpunit.xml, not added by an extension. Which files belong to a project is the project's decision, as it is for <source>.
No extension decides which tests a change nothing was recorded for can affect. Running every test is how the analysis says that it does not know. Letting an extension narrow that would replace the answer with a heuristic, such as table names read from migrations with regular expressions, and a miss would not show in the output.
Recording by many independent processes that share one cache directory is out of scope. One PHPUnit process per test, each recording what it ran, makes every one of them read, merge and write the whole file while it holds the lock, and a recording that is only ever made of partial runs never records the source files no test refers to. This will be addressed together with native parallel test execution.

github-actions · 2026-08-24T10:32:54Z

API Surface Changes

If any of the additions below are not intended as public API, mark them with @internal in the docblock.

New API Surface

Classes

PHPUnit\Framework\Attributes\UsesFixture — class PHPUnit\Framework\Attributes\UsesFixture
PHPUnit\Metadata\UsesFixture — class PHPUnit\Metadata\UsesFixture extends PHPUnit\Metadata\Metadata
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges — class PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges
PHPUnit\TextUI\Configuration\TestImpactAnalysis — class PHPUnit\TextUI\Configuration\TestImpactAnalysis

Methods

PHPUnit\Event\Code\TestMethod::idWithoutRepetitionAndAttempt — public function idWithoutRepetitionAndAttempt(): string
PHPUnit\Framework\Attributes\UsesFixture::__construct — public function __construct(string $path)
PHPUnit\Framework\Attributes\UsesFixture::path — public function path(): string
PHPUnit\Framework\TestCase::registerFixture — final protected function registerFixture(string ...$paths): void
PHPUnit\Metadata\Metadata::usesFixtureOnClass — static public function usesFixtureOnClass(string $path, string $declaringFile): PHPUnit\Metadata\UsesFixture
PHPUnit\Metadata\Metadata::usesFixtureOnMethod — static public function usesFixtureOnMethod(string $path, string $declaringFile): PHPUnit\Metadata\UsesFixture
PHPUnit\Metadata\Metadata::isUsesFixture — public function isUsesFixture(): bool
PHPUnit\Metadata\MetadataCollection::isUsesFixture — public function isUsesFixture(): self
PHPUnit\Metadata\UsesFixture::path — public function path(): string
PHPUnit\Metadata\UsesFixture::declaringFile — public function declaringFile(): string
PHPUnit\TextUI\Configuration\Configuration::recordTestImpactData — public function recordTestImpactData(): bool
PHPUnit\TextUI\Configuration\Configuration::deriveTestImpactDataFromCoverageTargets — public function deriveTestImpactDataFromCoverageTargets(): bool
PHPUnit\TextUI\Configuration\Configuration::testImpactAnalysis — public function testImpactAnalysis(): PHPUnit\TextUI\Configuration\TestImpactAnalysis
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges::__construct — public function __construct(PHPUnit\TextUI\Configuration\FilterDirectoryCollection $includeDirectories, PHPUnit\TextUI\Configuration\FilterFileCollection $includeFiles, PHPUnit\TextUI\Configuration\FilterDirectoryCollection $excludeDirectories, PHPUnit\TextUI\Configuration\FilterFileCollection $excludeFiles)
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges::includeDirectories — public function includeDirectories(): PHPUnit\TextUI\Configuration\FilterDirectoryCollection
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges::includeFiles — public function includeFiles(): PHPUnit\TextUI\Configuration\FilterFileCollection
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges::excludeDirectories — public function excludeDirectories(): PHPUnit\TextUI\Configuration\FilterDirectoryCollection
PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges::excludeFiles — public function excludeFiles(): PHPUnit\TextUI\Configuration\FilterFileCollection
PHPUnit\TextUI\Configuration\TestImpactAnalysis::__construct — public function __construct(PHPUnit\TextUI\Configuration\FilterDirectoryCollection $watchedDirectories, PHPUnit\TextUI\Configuration\FilterFileCollection $watchedFiles, PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges $addedFilesAreReachedThroughChanges)
PHPUnit\TextUI\Configuration\TestImpactAnalysis::watchedDirectories — public function watchedDirectories(): PHPUnit\TextUI\Configuration\FilterDirectoryCollection
PHPUnit\TextUI\Configuration\TestImpactAnalysis::watchedFiles — public function watchedFiles(): PHPUnit\TextUI\Configuration\FilterFileCollection
PHPUnit\TextUI\Configuration\TestImpactAnalysis::addedFilesAreReachedThroughChanges — public function addedFilesAreReachedThroughChanges(): PHPUnit\TextUI\Configuration\AddedFilesAreReachedThroughChanges

Modified API Surface

Methods

PHPUnit\TextUI\Configuration\Configuration::__construct

- public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $recordTestRunHistory, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testRunHistoryFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, bool $coverageHtmlClassView, bool $coverageHtmlFileView, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageJsonl, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnSelfDeprecation, bool $failOnDirectDeprecation, bool $failOnIndirectDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnSelfDeprecation, bool $doNotFailOnDirectDeprecation, bool $doNotFailOnIndirectDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireCoverageMetadataOnSmallTests, bool $requireCoverageMetadataOnMediumTests, bool $requireCoverageMetadataOnLargeTests, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomSeed, bool $seedRandomNumberGenerators, int $repeat, int $retry, int $timeout, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold, bool $warnWhenPhpIsNotConfiguredForDevelopment, bool $cacheTestIndex)
+ public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $recordTestRunHistory, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testRunHistoryFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, bool $coverageHtmlClassView, bool $coverageHtmlFileView, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageJsonl, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnSelfDeprecation, bool $failOnDirectDeprecation, bool $failOnIndirectDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnSelfDeprecation, bool $doNotFailOnDirectDeprecation, bool $doNotFailOnIndirectDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireCoverageMetadataOnSmallTests, bool $requireCoverageMetadataOnMediumTests, bool $requireCoverageMetadataOnLargeTests, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomSeed, bool $seedRandomNumberGenerators, int $repeat, int $retry, int $timeout, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold, bool $warnWhenPhpIsNotConfiguredForDevelopment, bool $cacheTestIndex, bool $recordTestImpactData, bool $deriveTestImpactDataFromCoverageTargets, TestImpactAnalysis $testImpactAnalysis)

codecov · 2026-08-24T12:21:43Z

Codecov Report

❌ Patch coverage is 99.84970% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.53%. Comparing base (6d6ba6b) to head (4e4979e).
⚠️ Report is 10420 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/TextUI/Application.php	98.97%	3 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #6919       +/-   ##
=============================================
+ Coverage     82.95%   99.53%   +16.58%     
- Complexity     4754    10858     +6104     
=============================================
  Files           279     1000      +721     
  Lines         13566    32521    +18955     
=============================================
+ Hits          11253    32370    +21117     
+ Misses         2313      151     -2162

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

sebastianbergmann · 2026-09-06T15:07:44Z

What is recorded for a test today

Three things:

the source files the test executed, as observed through the code coverage driver and restricted to what <source> says is first-party code (CodeCoverage::recordTestImpactDataFor());
the paths the test declares with #[UsesFixture];
the files the test class itself is made of: its own file, its parent classes, its traits, and the classes its data providers are methods of (TestFiles::of()).

The gap

Code that a test executes, that is not first-party code under <source>, and that is not a parent class, trait or data-provider class of the test, is in none of those three. The obvious example is a test helper:

src/Calculator.php          <- <source>
tests/CalculatorTest.php
tests/Support/Helper.php    <- executed by the test, in neither <source> nor the test class

public function testAdds(): void
{
    $this->assertSame(Helper::expectedSum(), (new Calculator)->add(1, 2));
}

What gets recorded for that test is src/Calculator.php and tests/CalculatorTest.php. tests/Support/Helper.php is not recorded, so changing it changes nothing that was recorded:

$ # edit Helper.php so that the test would now fail
$ phpunit --only-impacted
Impact:        0 of 1 tests can be affected by what changed; 1 test is not run

No tests executed!            # exit code 0

$ phpunit
FAILURES!
Failed asserting that 3 is identical to 4.

That is the bad shape: the suite reports success while a change that breaks it goes unrun. It is not a stale-data problem; the recording is up to date and correct about everything it covers.

What this is not

Not the bootstrap gap: That one was the same finding's other half and is fixed: Assumptions now hashes the bootstrap scripts, so changing one discards the whole recording. That mechanism does not generalise to helpers and discarding everything whenever any test file changes would defeat the point.
Not a problem for --impacted-by / --impacted-by-file: Naming the helper falls back to running everything, because the path is not among the files that were recorded:
```
Impact:        every test is run: .../tests/Support/Helper.php is not among the files that were recorded
```
Only the automatic mode (--only-impacted, which works out for itself what changed) is silently wrong.

Workaround

Adding the helper directory to <source> closes the gap for that project:

Impact:        1 of 1 tests can be affected by what changed; 0 tests are not run

The cost is that the helpers then count as first-party code for code coverage too, and show up in coverage reports.

Directions considered, none chosen (yet)

Record every file the test loaded: Complete and mechanical, but a test would then depend on the framework, vendor code and its whole autoload graph: very large recordings, and selections that rarely narrow.
Record from a declared test-code root: A new configuration element, or reusing the test suite directories. Bounded and predictable, but it needs new configuration and a rule for what belongs in it.
Extend #[UsesFixture], or add something like it, to cover code: Cheapest to build and the most precise, but opt-in per test, so the default behaviour stays as it is today; which is exactly what this note is about.

The first two change what "what a test depends on" means, so they want a deliberate decision rather than being settled on the way past.

sebastianbergmann · 2026-09-10T06:20:17Z

When --only-impacted runs more tests than you expected, the run tells you how many tests were selected, but not why any particular one is among them:

Impact:  44 of 2849 tests can be affected by what changed; 2805 tests are not run

Being affected by a change is only one of the reasons a test is run. A test is also run when nothing is known about it, and that number can dominate the selection without anything saying so. Read as it stands, the line above claims something about 44 tests that is only true of a few of them.

The new --explain-impacted CLI I just implemented reports which tests can be affected by what changed and why each of them can be, and it does not run anything:

$ phpunit --explain-impacted

Recorded from what the tests executed.

5 of 2849 tests can be affected by what changed.

2 tests depend on something that changed:
 - App\Tests\Api\ThemeControllerTest::testGetTheme
   src/Http/Controllers/ThemeController.php
 - App\Tests\Api\ThemeControllerTest::testGetThemeNotFound
   src/Http/Controllers/ThemeController.php

3 tests have never been recorded:
 - App\Tests\Models\RegionTest::testHasFallback
 - App\Tests\Models\RegionTest::testHasNoFallback
 - App\Tests\Services\ExportServiceTest::testExports

A test is reported under one of these:

depends on something that changed: the file it executed, or that it declares it uses, is not what it was. This is the reason you are looking for, and the file that caused it is named.
has never been recorded: nothing is known about the test. It is new, or it was skipped or marked incomplete when it last ran, and a test that executes nothing has nothing to record. Anything unknown is run rather than silently left out.
did not pass when it was last run: what was recorded describes a run that did not get through the test.
is depended upon by another test that is run: it cannot be left out without the test that depends on it being skipped.
is not a test method and can never be recorded: PHPT tests, for example.

That second reason is usually the surprising one. Tests that are always skipped never record anything, so they stay in the selection no matter what you change. They cost almost nothing to run, but until now there was no way to see that this is what was happening.

The option reports exactly what a run with --only-impacted would select, so you can use it to understand a selection before spending the time on it. It also takes the paths you already know changed:

$ git diff --name-only | phpunit --explain-impacted --impacted-by-file -

sebastianbergmann · 2026-09-21T12:28:47Z

This should be merged after #6784. It merges cleanly into main on its own, but once the parallel test execution has been merged there will be two functional problems that a textual conflict resolution does not solve:

1. The test selection must reach the parallel test runner

This branch passes the tests selected by --only-impacted from Application into TestRunner::run(), which hands them to TestSuiteFilterProcessor::process().

The parallel branch replaces the body of TestRunner::run() with TestRunnerLifecycle, which calls TestSuiteFilterProcessor::process() without a selection, and adds ParallelTestRunner, which uses the same lifecycle and does not accept a selection at all. Application chooses between the two runners based on numberOfParallelWorkers().

If the conflict is resolved by simply keeping both sides, --only-impacted --parallel=N runs every test: the selection is computed and printed, but never applied. The selection has to be threaded through TestRunnerLifecycle::run() into TestSuiteFilterProcessor::process(), and ParallelTestRunner::run() needs to accept and forward it just like TestRunner::run() does. With that in place, selection under --parallel=2 matches the sequential result on the end-to-end fixtures of this branch.

2. The test impact data recorded inside a worker must reach the parent

Test impact data is recorded per test in CodeCoverage::testImpactData() of the process that runs the test. For tests that run in a separate process, this branch ships what was recorded as testImpactData in the child's result envelope (method.tpl) and records it in the parent in ChildProcessResultProcessor.

The parallel branch's workers do not do this. src/Runner/Parallel/templates/worker.tpl ships only codeCoverage, events, and passedTests in the envelope of a unit, and ResultAggregator only calls ChildProcessResultEnvelope::mergeCodeCoverage(). Everything a worker records is lost. On the tests/end-to-end/cli/test-impact-data fixture, a sequential run records three tests, while the same run with --parallel=2 records only IsolatedCalculatorTest::testAddsInAnotherProcess, which survives because it goes through the separate-process path. Since a run that records overwrites the cache, a parallel run also destroys what a previous sequential run recorded, so subsequent --only-impacted runs select more tests than necessary.

To fix this:

worker.tpl must include what was recorded (CodeCoverage::instance()->testImpactData()->recorded()) in the unit's result envelope, in the same way method.tpl does.
The recording loop that this branch adds to ChildProcessResultProcessor should move into ChildProcessResultEnvelope::mergeCodeCoverage() (or a sibling method), so that both the separate-process path and ResultAggregator record it into the parent's CodeCoverage::testImpactData().
TestImpactData needs a way to forget what was recorded (the interface currently only has record() and recorded()). A persistent worker runs many units, and worker.tpl already clears the code coverage and calls PassedTests::reset() after shipping a unit's envelope. Without a reset, every envelope ships the data of all units the worker has run so far.
Until this is done, --record-test-impact-data combined with --parallel should be refused, like the other invalid option combinations this branch already rejects, so that a parallel run cannot silently overwrite the cache with incomplete data.

theofidry · 2026-09-25T12:30:30Z

@sebastianbergmann I have been trying it out and I found out a couple of blockers. The main one currently is the re-usability of the execution results across generated configurations.

As of now PHPUnit is invalidating the execution data based on a hash containing the following pieces of information:

hash of the PHPUnit configuration file
hash of the bootstrap files
source scope

For the initial test run, Infection applies various transformations to the PHPUnit configuration file. A non-exhaustive list:

Make the paths absolute
Add the source directories if absent.
Add the random ordering.
Enable stopOnDefect.
Disable colours and stderr redirection.
Removes custom printer classes, logging configuration and coverage reports.
Add coverage report if relevant.

However, there is no expected change from the way the dependencies are resolved (except maybe the source scope). Not being able to re-use previously generated execution results would make the feature unusable for infection.

Also ideally we should be able to re-use the cache across runs. Indeed, if there is no execution result at first then all the tests are executed, but in the sub-sequent run (even if the first run was done by infection), we should be able to use the results previously generated.

I also think it would be worth to notify the user about whether we are doing a cold run (i.e. missing or outdated execution results) vs. a hot run.

I still have to play around with how PHPUnit resolves the tests too.

A side question, if you consider the following:

infection tests/CalculatorTest.php # Infection will only execute that test, and mutate the covered code
infection src/Calculator.php # Infection will use TIA
infection src/Calculator.php tests/CalculatorTest.php ... # ???

Intuitively, as a user not familiar with infection internals but aware of the usage of TIA, would it make sense to you that we stick to executing the tests given, i.e. not use TIA here (but we will limit the mutations to src/Calculator.php, not all the code covered by the tests)?

sebastianbergmann · 2026-09-26T08:42:32Z

@theofidry Thank you for trying this with Infection, this is exactly the kind of feedback I was hoping for.

Re-using the data across generated configurations

You were right: hashing the configuration file was far too coarse. The data is no longer discarded when the configuration file changes, but only when a setting changes that can change what code a test executes: the <php> settings (from the XML or from the command line, -d and --include-path), which test suite is bootstrapped by which script, the extensions that are loaded, processIsolation, backupGlobals, and backupStaticProperties. Colours, output, logging, code coverage reports, stopOn*, failOn*, execution order, and so on are ignored. Together with the bootstrap script, the <source> declarations, and composer.lock, which were already covered, that is what the data rests on. So data recorded by Infection's initial run should now be usable by a plain phpunit run and vice versa, as long as both use the same cache directory and the same PHPUnit and PHP versions.

Two things worth checking on Infection's side:

A relative cacheDirectory is resolved against the directory of the configuration file. If the generated configuration lives in a temporary directory and the path is not made absolute, the two runs will look in different caches.
If Infection's initial run uses a bootstrap script of its own rather than the project's, the data will be discarded, because the contents of the bootstrap scripts are part of what it rests on.

Cold vs. hot run

When there is data but it cannot be used, --only-impacted, --explain-impacted, and --list-tests-that-depend-on now say why, for instance "every test is run: the configuration changed since the test impact data was recorded", instead of claiming that nothing was recorded. On a hot run, the Impact: line says how many tests were selected, and a Recorded: line says when the data used was recorded.

Your side question

I agree with your intuition: when tests are named explicitly, run those. Mutating only src/Calculator.php and running only the named tests that cover it already gives you the intersection, so there's nothing for TIA to add there. PHPUnit behaves the same way: when a filter is combined with impact selection, the filter narrows what was selected. It may be worth warning when a named test does not cover a named source file, since every mutant would then be reported as not covered.

Looking forward to what you find when you look at how the tests are resolved.

jarstelfox · 2026-09-28T21:58:14Z

Thanks for working on this! I've been trying it against iFixit's PHP application: about 10,800 tests, including WebDriver tests and some HTTP API tests whose code runs in a web server rather than in PHPUnit's process.

Dependency edge cases and bugs

I found a few bugs/edge cases to consider. Mostly with dependencies not being picked up / tests not running when they should:

If a test only reads a class constant (like Rates::US), it technically depends on that file but executes no lines in the Rates class. This means the file is not recorded as a dependency
If a test relies on state set by any framework bootstrapping code, and doesn't call that code itself, then it technically depends on the file that code is in, but executes no lines in it during the test, because the bootstrap ran before the test started. This means the file is not recorded as a dependency. For example, we point to bootstrap="bootstrap.php" which is a small init file:

require_once 'vendor/autoload.php';
(new \iFixit\Framework())->minimallyInitialize();

PHPUnit hashes the bootstrap file, so changing it discards the recording, but it doesn't track the code it calls, like minimallyInitialize() in iFixit/Framework.php.

If a test gets its data from a data provider that calls other code, and the test doesn't call that code itself, then it technically depends on the file that code is in, but executes no lines in it during the test, because PHPUnit runs data providers while it collects the tests, before any test starts. This means the file is not recorded as a dependency. Example:

final class ProvidedTest extends PHPUnit\Framework\TestCase {
   public static function cases(): array { return Cases::all(); }
   #[PHPUnit\Framework\Attributes\DataProvider('cases')]
   public function testIt(string $case): void { $this->assertContains($case, ['a', 'b']); }
}

PHPUnit records the test class the provider is in, so changing the provider itself is caught, but it doesn't track the code the provider calls, like Cases::all() in src/Cases.php.

If a test file requires another file whose top-level code sets state the test relies on, and the test doesn't run that file's code again itself, then it technically depends on the required file, but executes no lines in it during the test. PHPUnit loads test files while it collects the tests, before any test starts, and the required file's code runs then. This means the file is not recorded as a dependency. Example:

<?php
// file whose only line is Registry::$items[] = 'loaded';
require_once __DIR__ . '/../src/TopLevel.php';

final class RequiresTopLevelTest extends PHPUnit\Framework\TestCase {
   public function testIt(): void { $this->assertSame(['loaded'], Registry::$items); }
}

PHPUnit records the test file itself, so changing the require_once line is caught, but it doesn't track the file being required, i.e., src/TopLevel.php

Helpful Improvements

When looking at adding this to our CI pipeline (GH Actions) I noticed we couldn't reuse the TIA data across checkouts. We currently have multiple action runners on a server we own.

If the test impact data is recorded in one checkout and used from another, then PHPUnit can't use it, because the data file names every source file by its absolute path.

We'd want to record in a job on the main branch and select tests in pull request jobs, which run on other runners with their own checkouts. For example, a recording made in /home/runner-a/src refers to /home/runner-a/src/Libs/Utils.php, not to the copy a job in /home/runner-b/src has checked out.

I'm fairly sure storing paths relative to the project root, like Libs/Utils.php, would fix this.

Here is an easy repro script

#!/bin/bash

# Records tiny tests one per process with --record-test-impact-data, then
# asks PHPUnit which tests a change to each file can affect.
#
#   PHPUNIT=/path/to/vendor/bin/phpunit bash test-impact-gaps.sh

set -e -o pipefail -o nounset

phpunit=$(realpath "${PHPUNIT:-vendor/bin/phpunit}")
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir"
mkdir -p src/iFixit tests

cat >phpunit.xml <<'XML'
<?xml version="1.0"?>
<phpunit bootstrap="tests/bootstrap.php" cacheDirectory="cache" colors="false">
  <testsuites><testsuite name="all"><directory>tests</directory></testsuite></testsuites>
  <coverage includeUncoveredFiles="false"/>
  <source><include><directory suffix=".php">src</directory></include></source>
</phpunit>
XML
cat >tests/bootstrap.php <<'PHP'
<?php
spl_autoload_register(function (string $class): void {
   $file = __DIR__ . '/../src/' . str_replace('\\', '/', $class) . '.php';
   if (is_file($file)) {
      require $file;
   }
});
(new \iFixit\Framework())->minimallyInitialize();
PHP
cat >src/Rates.php <<'PHP'
<?php
final class Rates {
   public const US = 1;
   public static function us(): int { return self::US; }
}
PHP
cat >src/iFixit/Framework.php <<'PHP'
<?php
namespace iFixit;

final class Framework {
   public static string $mode = '';
   public function minimallyInitialize(): void { self::$mode = 'strict'; }
   public static function label(): string { return 'framework'; }
}
PHP
cat >src/Cases.php <<'PHP'
<?php
final class Cases {
   public static function all(): array { return [['a'], ['b']]; }
}
PHP
cat >src/Registry.php <<'PHP'
<?php
final class Registry {
   public static array $items = [];
}
PHP
cat >src/TopLevel.php <<'PHP'
<?php
Registry::$items[] = 'loaded';
PHP

# "test id|test file|test body"; each test gets a file of its own.
# CallsFrameworkTest, CallsCasesTest and IncludesTopLevelTest aren't checked
# below. They run a line of Framework.php, Cases.php and TopLevel.php inside
# the test, so PHPUnit knows those files: a change to one then shows
# PHPUnit's pick instead of "runs everything".
tests=(
   'CallsRatesTest::testIt|tests/CallsRatesTest.php|$this->assertSame(1, Rates::us());'
   'ReadsConstantTest::testIt|tests/ReadsConstantTest.php|$this->assertSame(1, Rates::US);'
   'CallsFrameworkTest::testIt|tests/CallsFrameworkTest.php|$this->assertSame("framework", \iFixit\Framework::label());'
   'ReadsBootStateTest::testIt|tests/ReadsBootStateTest.php|$this->assertSame("strict", \iFixit\Framework::$mode);'
   'CallsCasesTest::testIt|tests/CallsCasesTest.php|$this->assertCount(2, Cases::all());'
   'IncludesTopLevelTest::testIt|tests/IncludesTopLevelTest.php|require __DIR__ . "/../src/TopLevel.php"; $this->assertSame(["loaded"], Registry::$items);'
)
for test in "${tests[@]}"; do
   IFS='|' read -r id file body <<<"$test"
   printf '<?php\nfinal class %s extends PHPUnit\\Framework\\TestCase {\n   public function testIt(): void { %s }\n}\n' "${id%%::*}" "$body" >"$file"
done
cat >tests/ProvidedTest.php <<'PHP'
<?php
final class ProvidedTest extends PHPUnit\Framework\TestCase {
   public static function cases(): array { return Cases::all(); }
   #[PHPUnit\Framework\Attributes\DataProvider('cases')]
   public function testIt(string $case): void { $this->assertContains($case, ['a', 'b']); }
}
PHP
cat >tests/RequiresTopLevelTest.php <<'PHP'
<?php
// file whose only line is Registry::$items[] = 'loaded';
require_once __DIR__ . '/../src/TopLevel.php';

final class RequiresTopLevelTest extends PHPUnit\Framework\TestCase {
   public function testIt(): void { $this->assertSame(['loaded'], Registry::$items); }
}
PHP

record() {
   if ! XDEBUG_MODE=coverage php "$phpunit" --record-test-impact-data --record-test-run-history --run-test-id "$1" "$2" >recording.log 2>&1; then
      echo "Could not record $1:" >&2
      cat recording.log >&2
      exit 1
   fi
}
for test in "${tests[@]}"; do
   IFS='|' read -r id file _ <<<"$test"
   record "$id" "$file"
done
record 'ProvidedTest::testIt#0' tests/ProvidedTest.php
record 'RequiresTopLevelTest::testIt' tests/RequiresTopLevelTest.php

# "changed file|a test that depends on it|how"
cases=(
   'src/Rates.php|CallsRatesTest::testIt|calls a method in it'
   'src/Rates.php|ReadsConstantTest::testIt|only reads a class constant'
   'src/iFixit/Framework.php|ReadsBootStateTest::testIt|relies on what bootstrap.php ran'
   'src/Cases.php|ProvidedTest::testIt#0|gets its data from a provider'
   'src/TopLevel.php|RequiresTopLevelTest::testIt|relies on code a required file ran'
)
for case in "${cases[@]}"; do
   IFS='|' read -r changed id how <<<"$case"
   echo "$changed" >changed.txt
   explanation=$(XDEBUG_MODE=off php "$phpunit" --impacted-by-file changed.txt --explain-impacted 2>&1)
   if grep -qF -- " - $id" <<<"$explanation"; then
      verdict=picked
   elif grep -q '^Every test is run' <<<"$explanation"; then
      verdict='picked (runs everything)'
   else
      verdict=MISSED
   fi
   printf '%-26s %-30s %-38s %s\n' "$changed" "$id" "$how" "$verdict"
done

sebastianbergmann · 2026-09-29T05:53:18Z

@jarstelfox Thank you for trying this on a test suite of that size, and for the script: it reproduced every case on the first run, and I kept running it while I worked on them.

Three of the four cases you reported are now handled, and sharing a recording between checkouts works. The first case, reading a class constant, remains a limitation, and I explain why below.

Every path in the test impact data is now relative to the directory of the configuration file, or to the working directory when there is no configuration file. Changing only the file names would not have been enough: the recording was discarded before any file name was looked at, because the directories configured in <source> were part of what it was checked against, and they were absolute paths, too. PHPUnit then claimed that "what is first-party code changed", which was misleading. The same applied to the bootstrap script of a test suite, to include paths, and to directories named with #[UsesFixture].

For recording on main and selecting in pull request jobs, I recommend --only-impacted over passing git diff --name-only to --impacted-by-file. It compares every file against the hash it had when it was recorded, so the recording does not have to be made from the commit the pull request branched off: whatever changed on main since then only makes the selection larger. Keep in mind that a recording is only used with the same version of PHPUnit and the same version of PHP, down to the patch release, that it was recorded with. The format of the file has changed as well, so the first run with the new version records from scratch.

Your cases 2, 3 and 4 have one thing in common: the code runs before the test starts, and PHPUnit only recorded what was executed while a test ran. When it records test impact data, PHPUnit now keeps the code coverage driver running from before the bootstrap script is loaded until the recording is written, and pauses it only while a test runs. What a test executes is recorded for that test, as before. What is executed outside of the tests is recorded for the tests it can affect:

what a data provider executes, for the tests that are made from the data it provides
what is executed while the tests of a test class are run, setUpBeforeClass() and tearDownAfterClass() included, for the tests of that class
everything else, the bootstrap script and loading the test files among it, for every test

A change to a file in the last group makes PHPUnit run every test, and --explain-impacted names the file. With your script, ReadsBootStateTest, ProvidedTest and RequiresTopLevelTest are now selected. The second item also covers state that setUpBeforeClass() prepares, which your list did not include but which has the same shape.

For your suite, this means that every file in which minimallyInitialize() executes code is a file every test depends on, and a change to one of them runs the whole suite. I think that is the correct answer: the state those files set up is the state every test starts in. Only executable lines count, as they do for what a test executes, so a class that the bootstrap script merely loads is not among them.

Recording has become cheaper along the way. PHPUnit used to load its own code before it told the code coverage driver which code is first-party code, and Xdebug decides whether it collects code coverage for a file when that file is compiled. PHPUnit's own code was therefore collected while every test ran, only to be discarded. PHPUnit now tells the driver first. In a synthetic project with 2,000 tests, recording with Xdebug took 8.2 seconds before these changes and takes 6.7 seconds now, although it records more. I have not tried any of this with PCOV yet.

While working through your report I found a case you did not list, and that one was a bug. A test that depends on another test with #[Depends] is given what that test returns, but it does not execute that test's code. When only the test it depends on could be affected by a change, PHPUnit ran that test but not the test that depends on it. That is fixed.

Case 1 stays a limitation. Reading Rates::US executes no line in Rates.php, and the same is true for an enumeration case, an interface, or the default value of a property declared in a parent class. Static analysis of test code, test helpers and production code could find some of these dependencies, but I do not want to go down that road. PHP makes it impossible to be complete, think of $class::US, constant(), new $className, or service definitions in configuration files, and every heuristic would leave the next gap. I would rather have a mechanism whose limits can be stated in a sentence than one whose limits keep moving.

Where a test has a dependency that executing it cannot show, #[UsesFixture] declares it. The path is relative to the file the attribute is written in. With

#[UsesFixture('../src/Rates.php')]
final class ReadsConstantTest extends TestCase

a change to Rates.php selects ReadsConstantTest as well. I would use the same attribute for your WebDriver and HTTP API tests: the code they test runs in the web server, where PHPUnit sees none of it, so they can name the directory that code is in, and a change to any file in it will select them.

Because you record on main with a complete run anyway, a test that the selection misses in a pull request still fails there. It fails later than it should, but it does not go unnoticed.

If you can run the branch against your suite again, I would like to know how often a change runs every test because of what the bootstrap script executed, and how long recording takes now.

jarstelfox · 2026-09-29T22:14:41Z

Thanks for the quick turnaround! I reran the whole suite at 18b2323. Here are the two numbers you asked for, and then three changes that would help with the cases that are left.

How often the bootstrap makes a change run every test

In our suite, 101 files execute outside of any test. I replayed the files changed by the last 50 merges to our main branch through --impacted-by-file, and only 3 of those merges ran every test because they changed one of those files (one merge changed 11 of them, and two changed iFixit/Sites/Setting.php). All 3 also changed a file the recording has no record of, so they would have run every test anyway.

That was the common case: 38 of the 50 ran every test for a path the recording has no record of, like a doc, a config file, a SQL migration, a frontend source, or a PHP file no test loads. The other 9 selected a median of 1.5% of the tests, and at most 6.7%.

I used --impacted-by-file instead of --only-impacted because we record one test per process, and --only-impacted then runs every test. More on that under 1.

How long recording takes

The unit suite, 2,582 tests in one process with Xdebug 3.4.1:

	`09b54d8`	`18b2323`
empty code coverage cache	55.3 s	51.4 s
warm cache	31.4 s	27.3 s
of that, after the last test (cold / warm)	1.4 s / 1.4 s	13.2 s / 2.7 s

The tests themselves got a lot faster (PHPUnit's Time: went from 53.9 s to 38.2 s cold), but about 13 s of a cold run now happens after the last test. Without recording, PHPUnit reports 2.9 s at either commit with Xdebug off.

The whole suite is 10,839 tests, 552 of them WebDriver tests, recorded one PHPUnit process per test, 8 at a time. That took 73.6 minutes with --record-test-impact-data, and 27.5 minutes without it. Those were two runs on a shared server, so the ratio is rough.

1. Count the files a test's process loaded, when the test runs alone in it

Case 1 turned out to be bigger than constants. On the real suite, the recording also missed tests that call a method the class inherits, read the class by reflection, or mock it. None of them execute a line of the file they depend on.

The one that worried me is a test that checks who may open each of our admin pages. Many of those pages inherit hasPermission() from a base class, so the test runs no line of their own files. If I give one of those pages a hasPermission() that returns true, the test fails, but the recording links that page's file to a single other test, so a PR making that change wouldn't select it. You mentioned that a missed test still fails on main after the full run, but for us that's after review and merge, so the page would already be open on main.

These are the same kind as the cases you listed (enum cases, interfaces, a parent's property defaults), and I agree static analysis isn't the way to go. But loading gives a runtime rule whose limit fits in a sentence: a test depends on every file its process loaded. There's no analysis, and it catches the dynamic cases too, since $class::US, constant() and new $className all load the file. What it can't see is files read as data, like YAML service definitions, and other processes.

It only works when a test runs alone in its process, since PHP loads a file once per process. That's already how we record (one --run-test-id per process, which is how our own record gets each test's get_included_files()), and it's what process isolation does. Files the bootstrap loads would count for every test, the same way what it executes does now.

For the two concerns you listed for this idea, very large recordings and selections that rarely narrow, here are our numbers, leaving vendor/ out. Our record has 2.56 million pairs of a test and a file it loaded, against 1.50 million pairs of a test and a file it executed in PHPUnit's recording of the same run, so it would be about 1.7 times as big. And a one-file change runs 5.4% of our tests on average going by loaded files, against 22.8% going by the recording. That's because the recording runs every test for the 859 of 4,594 loaded files that no test executed a line of, and for the 101 that executed outside of any test.

Where the recording does know a file, it picks fewer tests (206 against 294 per file, over 3,612 files), and the difference is exactly the tests that loaded the file without executing it. Across the 50 merges, letting the recording decide instead of loaded files skipped only 1 to 10 more test methods, on 4 of them, and every one of those tests used the changed file in one of the ways above. Because of that, we went back to our own record for every test.

One more thing for recordings made one test per process: they never list the <source> files, so any source file no test executed looks new, and --only-impacted runs every test. With the reproducer below and nothing changed, it says:

Every test is run: src/Registry.php was not there, or was not first-party code, when what is known was recorded

After one run of every test instead, the same check says 0 of 16 tests can be affected by what changed.

2. A way to add files to the running test's record

For the WebDriver and HTTP API tests, you suggested #[UsesFixture] naming the server code's directory. That works, but then any change to the server's code runs every one of those tests. We record what the server loaded for each test instead. The test sends a token with its requests (a cookie in the browser, a cookie header on the HTTP client), the server writes its get_included_files() under that token when each request ends, and the test collects them before it finishes. Over the same 50 merges, when a merge selected, it ran a median of 9.3% of our browser test methods.

All we'd need from PHPUnit is a way to add files to what's recorded for the test that's running, from the test or from an extension, before PHPUnit stores it. Like #[UsesFixture], but at runtime:

// A sketch, not a real API: in tearDown(), or an extension subscribed to Test\Finished
TestImpact::addDependencies(...$filesTheServerLoadedForThisTest);

Then the recording would cover these tests too, and we could let PHPUnit decide every test instead of keeping our own record alongside it.

A bigger version would be PHPUnit shipping the server side too, the way phpunit-selenium collects code coverage from a web server: a PHPUNIT_SELENIUM_TEST_ID cookie, prepend.php and append.php for the server, and phpunit_coverage.php to hand the data back. Sending the token with each request would stay with the test suite either way, since only it knows its HTTP clients.

3. Inherit a class-level `#[UsesFixture]` from parent classes

If 2 is out of scope, #[UsesFixture] is what's left for these tests, and right now it has to go on every test class: a class-level #[UsesFixture] on a parent class is ignored, without a notice. Our 172 WebDriver test classes all extend one base class, directly or through one of three subclasses. #6998 will make a class-level #[Group] and friends apply to subclasses in PHPUnit 14, and doing the same for #[UsesFixture] would let us declare it once.

The reproducer, with the new cases

#!/bin/bash

# Records tiny tests one per process with --record-test-impact-data, then
# asks PHPUnit which tests a change to each file can affect.
#
#   PHPUNIT=/path/to/vendor/bin/phpunit bash test-impact-gaps.sh
#
# Every test on a row below depends on that row's file, except on the last
# row, so both columns should say yes, and the last row no. The last column
# is get_included_files() in the test's own process, which bootstrap.php
# writes at shutdown.

set -e -o pipefail -o nounset

phpunit=$(realpath "${PHPUNIT:-vendor/bin/phpunit}")
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
cd "$dir"
mkdir -p src/iFixit tests loaded

cat >phpunit.xml <<'XML'
<?xml version="1.0"?>
<phpunit bootstrap="tests/bootstrap.php" cacheDirectory="cache" colors="false">
  <testsuites><testsuite name="all"><directory>tests</directory></testsuite></testsuites>
  <coverage includeUncoveredFiles="false"/>
  <source><include><directory suffix=".php">src</directory></include></source>
</phpunit>
XML
cat >tests/bootstrap.php <<'PHP'
<?php
spl_autoload_register(function (string $class): void {
   $file = __DIR__ . '/../src/' . str_replace('\\', '/', $class) . '.php';
   if (is_file($file)) {
      require $file;
   }
});
(new \iFixit\Framework())->minimallyInitialize();

// What this test's process loaded, for the last column below.
if (($id = getenv('TEST_ID')) !== false) {
   register_shutdown_function(function () use ($id): void {
      $src = realpath(__DIR__ . '/../src') . '/';
      $loaded = [];
      foreach (get_included_files() as $file) {
         if (str_starts_with($file, $src)) {
            $loaded[] = 'src/' . substr($file, strlen($src));
         }
      }
      file_put_contents(__DIR__ . '/../loaded/' . preg_replace('/[^A-Za-z0-9_]/', '_', $id), implode("\n", $loaded) . "\n");
   });
}
PHP
cat >src/Rates.php <<'PHP'
<?php
final class Rates {
   public const US = 1;
   public static function us(): int { return self::US; }
}
PHP
cat >src/iFixit/Framework.php <<'PHP'
<?php
namespace iFixit;

final class Framework {
   public static string $mode = '';
   public function minimallyInitialize(): void { self::$mode = 'strict'; }
   public static function label(): string { return 'framework'; }
}
PHP
cat >src/Cases.php <<'PHP'
<?php
final class Cases {
   public static function all(): array { return [['a'], ['b']]; }
}
PHP
cat >src/Registry.php <<'PHP'
<?php
final class Registry {
   public static array $items = [];
}
PHP
cat >src/TopLevel.php <<'PHP'
<?php
Registry::$items[] = 'loaded';
PHP
cat >src/Page.php <<'PHP'
<?php
abstract class Page {
   // The default gate: a page opens for nobody unless it says otherwise.
   public function hasPermission(): bool { return false; }
}
PHP
cat >src/ReportPage.php <<'PHP'
<?php
final class ReportPage extends Page {
   public function title(): string { return 'report'; }
}
PHP
cat >src/Mailer.php <<'PHP'
<?php
class Mailer {
   public function send(string $to): bool { return $to !== ''; }
}
PHP
cat >src/Seeds.php <<'PHP'
<?php
final class Seeds {
   public static array $rows = [];
   public static function plant(): void { self::$rows = ['seeded']; }
   public static function count(): int { return count(self::$rows); }
}
PHP

# "test id|test file|test body"; each test gets a file of its own.
# CallsFrameworkTest, CallsCasesTest, IncludesTopLevelTest, CallsSeedsTest,
# CallsReportPageTest and CallsMailerTest aren't checked below. They run a
# line of Framework.php, Cases.php, TopLevel.php, Seeds.php, ReportPage.php
# and Mailer.php inside the test, so PHPUnit knows those files: a change to
# one then shows PHPUnit's pick instead of "runs everything".
tests=(
   'CallsRatesTest::testIt|tests/CallsRatesTest.php|$this->assertSame(1, Rates::us());'
   'ReadsConstantTest::testIt|tests/ReadsConstantTest.php|$this->assertSame(1, Rates::US);'
   'CallsFrameworkTest::testIt|tests/CallsFrameworkTest.php|$this->assertSame("framework", \iFixit\Framework::label());'
   'ReadsBootStateTest::testIt|tests/ReadsBootStateTest.php|$this->assertSame("strict", \iFixit\Framework::$mode);'
   'CallsCasesTest::testIt|tests/CallsCasesTest.php|$this->assertCount(2, Cases::all());'
   'IncludesTopLevelTest::testIt|tests/IncludesTopLevelTest.php|require __DIR__ . "/../src/TopLevel.php"; $this->assertSame(["loaded"], Registry::$items);'
   'CallsSeedsTest::testIt|tests/CallsSeedsTest.php|$this->assertSame(0, Seeds::count());'
   'CallsReportPageTest::testIt|tests/CallsReportPageTest.php|$this->assertSame("report", (new ReportPage())->title());'
   'InheritsGateTest::testIt|tests/InheritsGateTest.php|$this->assertFalse((new ReportPage())->hasPermission());'
   'ReflectsPageTest::testIt|tests/ReflectsPageTest.php|$this->assertTrue((new ReflectionClass(ReportPage::class))->isSubclassOf(Page::class));'
   'CallsMailerTest::testIt|tests/CallsMailerTest.php|$this->assertTrue((new Mailer())->send("a"));'
   'MocksMailerTest::testIt|tests/MocksMailerTest.php|$mailer = $this->createMock(Mailer::class); $mailer->method("send")->willReturn(true); $this->assertTrue($mailer->send("a"));'
)
for test in "${tests[@]}"; do
   IFS='|' read -r id file body <<<"$test"
   printf '<?php\nfinal class %s extends PHPUnit\\Framework\\TestCase {\n   public function testIt(): void { %s }\n}\n' "${id%%::*}" "$body" >"$file"
done
cat >tests/DeclaresRatesTest.php <<'PHP'
<?php
#[PHPUnit\Framework\Attributes\UsesFixture('../src/Rates.php')]
final class DeclaresRatesTest extends PHPUnit\Framework\TestCase {
   public function testIt(): void { $this->assertSame(1, Rates::US); }
}
PHP
cat >tests/ProvidedTest.php <<'PHP'
<?php
final class ProvidedTest extends PHPUnit\Framework\TestCase {
   public static function cases(): array { return Cases::all(); }
   #[PHPUnit\Framework\Attributes\DataProvider('cases')]
   public function testIt(string $case): void { $this->assertContains($case, ['a', 'b']); }
}
PHP
cat >tests/RequiresTopLevelTest.php <<'PHP'
<?php
// file whose only line is Registry::$items[] = 'loaded';
require_once __DIR__ . '/../src/TopLevel.php';

final class RequiresTopLevelTest extends PHPUnit\Framework\TestCase {
   public function testIt(): void { $this->assertSame(['loaded'], Registry::$items); }
}
PHP
cat >tests/UsesClassSetupTest.php <<'PHP'
<?php
final class UsesClassSetupTest extends PHPUnit\Framework\TestCase {
   public static function setUpBeforeClass(): void { Seeds::plant(); }
   public function testIt(): void { $this->assertSame(['seeded'], Seeds::$rows); }
}
PHP

record() {
   if ! TEST_ID=$1 XDEBUG_MODE=coverage php "$phpunit" --record-test-impact-data --record-test-run-history --run-test-id "$1" "$2" >recording.log 2>&1; then
      echo "Could not record $1:" >&2
      cat recording.log >&2
      exit 1
   fi
}
for test in "${tests[@]}"; do
   IFS='|' read -r id file _ <<<"$test"
   record "$id" "$file"
done
record 'DeclaresRatesTest::testIt' tests/DeclaresRatesTest.php
record 'ProvidedTest::testIt#0' tests/ProvidedTest.php
record 'RequiresTopLevelTest::testIt' tests/RequiresTopLevelTest.php
record 'UsesClassSetupTest::testIt' tests/UsesClassSetupTest.php

# "changed file|a test|how the test uses the file"
cases=(
   'src/Rates.php|CallsRatesTest::testIt|calls a method in it'
   'src/Rates.php|ReadsConstantTest::testIt|reads a class constant'
   'src/Rates.php|DeclaresRatesTest::testIt|reads the constant, declares #[UsesFixture]'
   'src/iFixit/Framework.php|ReadsBootStateTest::testIt|relies on what bootstrap.php ran'
   'src/Cases.php|ProvidedTest::testIt#0|gets its data from a provider'
   'src/TopLevel.php|RequiresTopLevelTest::testIt|relies on code a required file ran'
   'src/Seeds.php|UsesClassSetupTest::testIt|relies on what setUpBeforeClass() ran'
   'src/ReportPage.php|InheritsGateTest::testIt|calls a method the class inherits'
   'src/ReportPage.php|ReflectsPageTest::testIt|reads the class by reflection'
   'src/Mailer.php|MocksMailerTest::testIt|mocks the class'
   'src/Mailer.php|CallsRatesTest::testIt|does not use it'
)
printf '%-26s %-30s %-44s %-18s %s\n' 'changed file' 'test' 'how the test uses the file' 'PHPUnit picks it' 'its process loaded it'
for case in "${cases[@]}"; do
   IFS='|' read -r changed id how <<<"$case"
   echo "$changed" >changed.txt
   explanation=$(XDEBUG_MODE=off php "$phpunit" --impacted-by-file changed.txt --explain-impacted 2>&1)
   if grep -qF -- " - $id" <<<"$explanation"; then
      picked=yes
   elif grep -q '^Every test is run' <<<"$explanation"; then
      picked='yes (runs all)'
   else
      picked=no
   fi
   if grep -qxF "$changed" "loaded/${id//[^A-Za-z0-9_]/_}"; then
      loaded=yes
   else
      loaded=no
   fi
   printf '%-26s %-30s %-44s %-18s %s\n' "$changed" "$id" "$how" "$picked" "$loaded"
done

# A recording made one test per process never lists the <source> files, so a
# source file no test executed (Registry.php) looks new to --only-impacted.
echo
echo "--only-impacted, with nothing changed since the recording:"
XDEBUG_MODE=off php "$phpunit" --only-impacted --explain-impacted 2>&1 | grep -E '^Every test is run|can be affected' | sed "s#$(pwd -P)/##"

Its output at 18b2323:

changed file               test                           how the test uses the file                   PHPUnit picks it   its process loaded it
src/Rates.php              CallsRatesTest::testIt         calls a method in it                         yes                yes
src/Rates.php              ReadsConstantTest::testIt      reads a class constant                       no                 yes
src/Rates.php              DeclaresRatesTest::testIt      reads the constant, declares #[UsesFixture]  yes                yes
src/iFixit/Framework.php   ReadsBootStateTest::testIt     relies on what bootstrap.php ran             yes (runs all)     yes
src/Cases.php              ProvidedTest::testIt#0         gets its data from a provider                yes                yes
src/TopLevel.php           RequiresTopLevelTest::testIt   relies on code a required file ran           yes (runs all)     yes
src/Seeds.php              UsesClassSetupTest::testIt     relies on what setUpBeforeClass() ran        yes                yes
src/ReportPage.php         InheritsGateTest::testIt       calls a method the class inherits            no                 yes
src/ReportPage.php         ReflectsPageTest::testIt       reads the class by reflection                no                 yes
src/Mailer.php             MocksMailerTest::testIt        mocks the class                              no                 yes
src/Mailer.php             CallsRatesTest::testIt         does not use it                              no                 no

--only-impacted, with nothing changed since the recording:
Every test is run: src/Registry.php was not there, or was not first-party code, when what is known was recorded

sebastianbergmann · 2026-09-30T05:58:05Z

@jarstelfox Thank you for running the branch against your suite again, and for the numbers. All three of your requests are implemented. With your reproducer, PHPUnit now picks the test on every row except the last one, which is the row where it should not.

Recording your suite with one --run-test-id process per test, eight of them at a time, is a setup I will not support, though. I explain why below, and what still works for you.

What a test's process loaded

When a test runs in a process of its own, PHPUnit now records every first-party file that process loaded for it, in addition to what the test executed. A test runs in a process of its own when it runs in process isolation, or when it is the only test of the test run. Your --run-test-id processes are the second case.

This covers calling a method a class inherits, reading a class by reflection, creating a test double for it, reading one of its constants, and $class::US, constant(), and new $className, because all of these load the file that declares the class. With your reproducer, InheritsGateTest, ReflectsPageTest, MocksMailerTest, and ReadsConstantTest are now selected.

PHP loads a file only once per process. In a process that runs many tests, a file is loaded by whichever test needs it first, and that says nothing about the tests that come after it. That is why this only applies to a test that has a process to itself. The rule now reads: a test that runs in a process of its own depends on every first-party file that process loaded, and any other test depends on the first-party files it executed a line of. vendor/ is not first-party code, so it is not recorded.

For such a test, this changes what I wrote last time: a class the bootstrap script merely loads now counts, because it was loaded in the process the test ran in.

Adding what a test depends on while it runs

A test can now register files and directories it depends on while it runs:

abstract class WebDriverTestCase extends TestCase
{
    #[After]
    protected function recordWhatTheServerLoaded(): void
    {
        $this->registerFixture(...$this->filesTheServerLoadedFor($this->token));
    }
}

What is registered is recorded for the test together with what it executed and what it declares with #[UsesFixture]. The registration has to happen while the test runs: in the test method, in setUp() or tearDown(), in a #[Before] or #[After] method, or in anything these call. An #[After] method of a parent class is run for every test of every class that extends it, so it only has to be written once.

A relative path is resolved against the working directory at the time of the call. A path that does not exist triggers a warning, and so does an empty path. Nothing happens when no test impact data is recorded. For a test that is skipped or marked incomplete, nothing is recorded at all, so what it registered is dropped as well. Registering works for a test that runs in process isolation, too.

Paths are recorded as they are named, just like those of #[UsesFixture], so leave out vendor/: a change there comes with a change to composer.lock, which already discards everything that was recorded.

There is no way to register paths from an extension. Test\Finished is emitted after what the test depends on has been recorded, and a subscriber only gets to see a value object, not the test case object. The test sends the token with its requests anyway, so it is the place that knows what the server loaded. PHPUnit will not ship the server side of this: how requests reach a web server is too specific to each project.

`#[UsesFixture]` on a parent class

A class-level #[UsesFixture] attribute on a parent class now counts for the tests of every class that extends it. The path is resolved relative to the file the attribute is written in, which is the file of the parent class. Your 172 WebDriver test classes can declare it once, on the class they extend.

While working on this, I found that #[UsesFixture('')] resolved to the directory of the file the attribute is written in. An empty path now resolves to nothing and triggers the warning about a fixture that does not exist.

Where the time went

The 13 seconds your cold run spends outside of the Time: that PHPUnit reports are not spent after the last test, but before the first one. Time: only covers running the tests. With an empty code coverage cache, the static analysis of your source files now runs while the tests are loaded, because what is executed outside of the tests is collected at every data provider, and those files then do not have to be analysed again while the tests run.

In a synthetic project with 3,000 classes and 3,000 tests that use data providers, the tests were loaded after 1.9 seconds before these changes and after 30.3 seconds now, while Time: went from 48.2 seconds to 17.1 seconds, and the whole run from 51.3 seconds to 48.3 seconds. The work was moved, not added, but it now happens before the first test runs, while there is no progress to show.

The difference between 73.6 and 27.5 minutes for the whole suite has a different cause, which leads to the next section.

Custom setups for parallel test execution are out of scope

A recording is made by one test run. Splitting a suite across independent PHPUnit processes that record into the same cache directory, one --run-test-id per process for instance, is out of scope for test impact analysis.

Only a test run that runs every test there is can record the source files that no test refers to, and only such a test run can forget the entries of tests that no longer exist. None of your processes is such a test run. That is why --only-impacted runs every test for you: every source file that no process loaded is one that nothing is known about. With your reproducer, --only-impacted now selects 1 of 17 tests, the data set of ProvidedTest that the script never records, but only because every source file in it is loaded by one of its processes.

Every one of these processes also reads, merges, and rewrites the whole data file while it holds the lock on it. For a file the size of yours, 10,839 tests with about 1.5 million pairs of a test and a file, that takes 0.92 seconds per process on my machine, compared to 0.11 seconds for a small file. Eight processes that each hold the lock that long end up waiting for each other. I expect that to be most of the difference, but I have only measured it on my machine.

Nothing stops you from recording that way, and what is recorded for each test is correct. --impacted-by-file works with such a recording, as you found. But I will not change how the data is stored to make this setup work. When PHPUnit ships its own support for running tests in parallel, one test run will be in charge of the recording again, no matter how many processes run the tests, and that is when I will address this.

Updated decisions

Recording is never automatic, not even when coverage is already being collected. A suite whose slow tests declare #[CoversNothing] would otherwise get 2.6× slower unasked.
Tests declaring #[CoversNothing] are collected for in observation mode, with the targets bypassed so nothing reaches the coverage report. The alternative — treating them as affected by everything — would put a floor under every selected run.
Impact data is never filtered using coverage targets; the report always is. The two disagree by design. Before this, what the coverage report kept was missing 75% and 94% of the dependencies tests really have on the two suites I measured, because the report is narrowed to what a test declares. That is why the data is collected before the narrowing, which needed php-code-coverage 14.4.0.
What a test depends on is observed, not worked out. There is no static analysis of test code, test helpers, or production code. PHP makes it impossible to be complete, and every heuristic would leave the next gap. What observation cannot show is declared with #[UsesFixture] or registered with registerFixture().
What is executed outside of the tests counts for the tests it can affect. What a data provider executes counts for the tests made from the data it provides, what is executed while the tests of a test class are run counts for the tests of that class, and everything else, the bootstrap script and loading the tests among it, counts for every test. A change to a file in the last group runs every test.
A test that runs in a process of its own depends on every first-party file that process loaded. That is a test in process isolation, or the only test of a test run. For every other test, what it executed is what counts, because in a process that runs many tests, which test loaded a file depends on the order they run in.
What only the test knows while it runs is registered by the test. registerFixture() adds files and directories to what is recorded for the running test. PHPUnit does not collect anything from other processes itself.
A class-level #[UsesFixture] on a parent class counts for the classes that extend it, and its path is relative to the file of that parent class.
A test's own files are recorded in the impact data rather than checked against the test index at selection time. The index is written at a different moment, so a run that refreshed the index without recording impact data would report a changed test as unchanged.
The assumption hash covers the <source> declarations, not the expanded file list. Hashing the list would throw everything away each time a class is added to src/, and buys nothing: a new file can only be reached if an existing one changed.
The data can move between checkouts, but not between versions. Every path in it is relative to the directory of the configuration file, so what one CI runner recorded can be used by another. It is discarded when the version of PHPUnit or of PHP, down to the patch release, is not the one it was recorded with.
A recording is made by one test run. Splitting a suite across independent PHPUnit processes that record into the same cache directory is out of scope. Such runs never record the source files that no test refers to, never forget the entries of tests that no longer exist, and each of them rewrites the whole data file while it holds the lock on it. Running a suite in parallel will be covered by PHPUnit's own support for parallel test execution.
--only-impacted, --impacted-by and --impacted-by-file are command line options only. A switch that makes every run partial does not belong in a file a project commits.

…g can be affected by what changed

…ommand line

…well

…from

…estores it too

…ge driver is available

…erage metadata is required for all tests

… and why each of them can be

…ing that none was recorded

…e a test executes changed, not whenever the configuration file changed

…onfiguration file so that what is recorded in one checkout can be used in another

… as the paths it is asked about are

…hanged

…est impact data, and run every test when one of them changed

…and the methods called before and after the tests of a test class execute, for the tests it can affect

…r the tests of every class that extends it

…, so that they are recorded as test impact data together with what it executed

…tribute is written in, and quote the path in the warnings about fixtures that do not exist

…cess of its own loaded, whether a line of them was executed or not

… includes what that process loaded

…ing on changed or was added

…ys that an added file can only affect a test through a file that was changed to use it

sebastianbergmann self-assigned this Aug 24, 2026

sebastianbergmann added type/enhancement A new idea that should be implemented feature/test-runner CLI test runner labels Aug 24, 2026

sebastianbergmann force-pushed the feature/test-impact-analysis branch from c866112 to a703d1f Compare August 24, 2026 10:31

sebastianbergmann force-pushed the feature/test-impact-analysis branch from ec50b37 to bdf152b Compare August 27, 2026 04:52

theofidry mentioned this pull request Aug 27, 2026

PHPUnit orchestration: known issues and discussion notes infection/infection#2874

Open

sebastianbergmann force-pushed the feature/test-impact-analysis branch 3 times, most recently from 248a176 to 5aad389 Compare September 6, 2026 15:01

sebastianbergmann force-pushed the feature/test-impact-analysis branch from 40fc113 to 5a7b2ae Compare September 9, 2026 05:50

sebastianbergmann added this to the PHPUnit 13.5 milestone Sep 19, 2026

sebastianbergmann force-pushed the feature/test-impact-analysis branch from fef6829 to 09b54d8 Compare September 20, 2026 06:38

sebastianbergmann mentioned this pull request Sep 21, 2026

Parallel test execution #6784

Draft

sebastianbergmann mentioned this pull request Sep 24, 2026

Test Impact Analysis #6897

Closed

theofidry mentioned this pull request Sep 24, 2026

feat(telemetry): Add OpenTelemetry observability support infection/infection#3108

Open

13 of 28 tasks

sebastianbergmann force-pushed the feature/test-impact-analysis branch from 5f4f80f to 52211f0 Compare September 29, 2026 05:14

sebastianbergmann force-pushed the feature/test-impact-analysis branch from 18b2323 to e93d35c Compare September 30, 2026 05:56

sebastianbergmann force-pushed the feature/test-impact-analysis branch 2 times, most recently from 9a1c62d to 9a5f363 Compare October 2, 2026 04:58

sebastianbergmann added 29 commits October 16, 2026 10:23

Do not treat having no test to run as an empty test suite when nothin…

ad99ae9

…g can be affected by what changed

Look for composer.lock above the configuration file as well

382b1ca

Do not derive test impact data when recording it is declined on the c…

c950e24

…ommand line

Hold the test impact data file while it is read, added to, and written

b63814c

Do not say how many tests are not run when the tests are filtered as …

c6c7f7d

…well

Do not let a path that is worked out climb out of the root it starts …

6d36678

…from

Reject an empty value for --list-tests-that-depend-on

d23d290

Restore what a test changed in --CLEAN--, so that a test that fails r…

9b8430a

…estores it too

Do not record test impact data when no code coverage is collected

ecfed7a

Make test portable

077330d

Report that test impact data will not be recorded when no code covera…

8117d10

…ge driver is available

Only derive test impact data from code coverage targets when code cov…

f4056d3

…erage metadata is required for all tests

Add option for reporting which tests can be affected by what changed,…

c075b56

… and why each of them can be

Add tests

a276796

Report when the test impact data that is used was recorded

c85e720

Report why the recorded test impact data is not used instead of claim…

4fe4a68

…ing that none was recorded

Only discard test impact data when a setting that can change what cod…

9115463

…e a test executes changed, not whenever the configuration file changed

Name files in the test impact data relative to the directory of the c…

151cf19

…onfiguration file so that what is recorded in one checkout can be used in another

Read the base directory with the directory separator of the platform,…

741ccdc

… as the paths it is asked about are

Also run a test that depends on a test that can be affected by what c…

2ac13f2

…hanged

Keep the source files that were executed outside of any test in the t…

fa11e0d

…est impact data, and run every test when one of them changed

Record what the bootstrap script, loading the tests, data providers, …

a71a02a

…and the methods called before and after the tests of a test class execute, for the tests it can affect

Let a class-level #[UsesFixture] attribute on a parent class count fo…

53a80a5

…r the tests of every class that extends it

Let a test register files and directories it depends on while it runs…

915f89d

…, so that they are recorded as test impact data together with what it executed

Do not take an empty #[UsesFixture] path to mean the directory the at…

3e375d2

…tribute is written in, and quote the path in the warnings about fixtures that do not exist

Record the source files that the process of a test that runs in a pro…

44ea689

…cess of its own loaded, whether a line of them was executed or not

Say that what is recorded for a test that ran in a process of its own…

67dadb2

… includes what that process loaded

Run every test when a watched file that no test is recorded as depend…

6038921

…ing on changed or was added

Do not run every test when a file is added where the configuration sa…

4e4979e

…ys that an added file can only affect a test through a file that was changed to use it

sebastianbergmann force-pushed the feature/test-impact-analysis branch from 5c9e134 to 4e4979e Compare October 16, 2026 08:41

Navigation Menu

Repository navigation

Test Impact Analysis - #6919

Test Impact Analysis#6919
sebastianbergmann wants to merge 79 commits into
mainfrom
feature/test-impact-analysis

sebastianbergmann commented Aug 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 24, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 24, 2026 •

edited

Loading

Uh oh!

sebastianbergmann commented Sep 6, 2026

Uh oh!

sebastianbergmann commented Sep 10, 2026

Uh oh!

sebastianbergmann commented Sep 21, 2026

Uh oh!

theofidry commented Sep 25, 2026 •

edited

Loading

Uh oh!

sebastianbergmann commented Sep 26, 2026

Uh oh!

jarstelfox commented Sep 28, 2026 •

edited

Loading

Uh oh!

sebastianbergmann commented Sep 29, 2026 •

edited

Loading

Uh oh!

jarstelfox commented Sep 29, 2026

Uh oh!

sebastianbergmann commented Sep 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sebastianbergmann commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

Recording

Querying

Selecting

What is recorded

Two ways of maintaining the data

Measurements

What you can rely on

What it cannot see

#[UsesFixture]

registerFixture()

Watched files

Decisions

Uh oh!

github-actions Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

API Surface Changes

New API Surface

Classes

Methods

Modified API Surface

Methods

Uh oh!

codecov Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sebastianbergmann commented Sep 6, 2026

What is recorded for a test today

The gap

What this is not

Workaround

Directions considered, none chosen (yet)

Uh oh!

sebastianbergmann commented Sep 10, 2026

Uh oh!

sebastianbergmann commented Sep 21, 2026

1. The test selection must reach the parallel test runner

2. The test impact data recorded inside a worker must reach the parent

Uh oh!

theofidry commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sebastianbergmann commented Sep 26, 2026

Re-using the data across generated configurations

Cold vs. hot run

Your side question

Uh oh!

jarstelfox commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency edge cases and bugs

Helpful Improvements

Uh oh!

sebastianbergmann commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jarstelfox commented Sep 29, 2026

How often the bootstrap makes a change run every test

How long recording takes

1. Count the files a test's process loaded, when the test runs alone in it

2. A way to add files to the running test's record

3. Inherit a class-level #[UsesFixture] from parent classes

Uh oh!

sebastianbergmann commented Sep 30, 2026

What a test's process loaded

Adding what a test depends on while it runs

#[UsesFixture] on a parent class

Where the time went

Custom setups for parallel test execution are out of scope

Updated decisions

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

sebastianbergmann commented Aug 24, 2026 •

edited

Loading

`#[UsesFixture]`

`registerFixture()`

github-actions Bot commented Aug 24, 2026 •

edited

Loading

codecov Bot commented Aug 24, 2026 •

edited

Loading

theofidry commented Sep 25, 2026 •

edited

Loading

jarstelfox commented Sep 28, 2026 •

edited

Loading

sebastianbergmann commented Sep 29, 2026 •

edited

Loading

3. Inherit a class-level `#[UsesFixture]` from parent classes

`#[UsesFixture]` on a parent class