Test Impact Analysis

Summary

PHPUnit should be able to work out which tests can possibly be affected by a change to the code under test, and offer to run only those tests.

The information needed for this already exists inside PHPUnit: when code coverage is collected, PHPUnit knows exactly which source files each individual test executed. Today that knowledge is used only to produce coverage reports and is thrown away at the end of the run. Kept around, it answers a question developers ask constantly: "I changed this file, what do I need to run?"

Motivation

The larger a test suite becomes, the less it is actually used sometimes.

A suite that takes ten minutes is not run after every change; it is run before a commit, or before a push, or not at all until CI complains. The feedback loop that unit testing exists to provide gets stretched until it stops being a loop. Developers work around this by hand: running a single test class, guessing at a --filter pattern, remembering which directory "probably" covers the thing they touched. Those guesses are quick, and they are wrong often enough to matter and miss the test three layers away that also exercised the changed line.

PHPUnit is in a better position to answer this than the developer is, because it has observed what every test actually executed rather than what someone assumes it executed.

Where this idea comes from

The idea is old and well studied, under the name regression test selection. Gregg Rothermel and Mary Jean Harrold formalised it in the 1990s and gave the field the property that still matters most: a selection technique is called safe when it selects every test that could expose a fault introduced by the change. Their techniques achieved that by comparing control flow graphs before and after a change. Everything since has been a trade between how safe a technique is and how cheap it is to run.

The name "test impact analysis" comes from industry tooling rather than from the literature. Microsoft shipped a feature under that name in Visual Studio 2010, and Paul Hammant's 2017 article The Rise of Test Impact Analysis carried it into wider use.

What made it practical was lowering the resolution. Ekstazi (ISSTA 2015) showed that tracking dependencies at file granularity, observed dynamically while tests run, is coarse enough to be cheap to collect and precise enough to be useful, which is the granularity this proposal assumes. A different branch of the field, beginning with Meta's predictive test selection in 2018, drops dependency tracking entirely and predicts from historical run data using machine learning. That is a different feature and is not what is proposed here.

Most ecosystems now have something in this space, whether built into the test runner, the build tool, or the CI vendor.

Proposed Feature

Two capabilities, both opt-in:

Recording

During a run in which code coverage is collected, PHPUnit records which source files each test depends on and stores that alongside the other information it already stores between runs. This costs nothing beyond the coverage collection that is already happening.

Selection

On subsequent runs, PHPUnit can compare the current state of the source files against what was recorded, determine which tests can possibly be affected, and run only those.

From the command line this might look like:

$ phpunit --only-impacted

with the set of changed files determined by PHPUnit itself, or supplied explicitly so that it composes with whatever the developer already uses to answer "what changed":

$ phpunit --impacted-by src/Service/Invoice.php
$ git diff --name-only | phpunit --impacted-by-file -

And a read-only form that answers the question without running anything:

$ phpunit --list-impacted-tests --impacted-by src/Service/Invoice.php

Use Cases

Keeping the feedback loop short while working

A developer editing one class re-runs the suite after every save. Instead of ten minutes, they wait for the handful of tests that touch what they just changed. The suite becomes something used continuously rather than something run at checkpoints.

Pre-commit and pre-push checks

A hook that runs the full suite gets disabled within a week. A hook that runs the tests affected by the staged changes stays enabled, because it finishes in seconds and still catches the mistakes that matter.

Fast signal on pull requests

CI runs the affected tests first and reports within a minute, then runs the full suite. Reviewers get an early answer without giving up the guarantee that the complete suite ran before merge.

Understanding the impact before making a change

Before touching a class in unfamiliar code, a developer asks which tests cover it. Today the answer requires generating a coverage report and reading it backwards. This turns it into one command, and the answer is derived from what the tests really executed rather than from #[CoversClass] attributes that may be incomplete or absent.

Spotting untested changes

A change to a source file that maps to zero tests is worth knowing about. It means either that the file is untested, or that whatever exercises it does so in a way coverage did not observe. Both are useful signals during review.

Working productively in large or inherited suites

In a codebase where the full suite takes long enough that nobody runs it locally, this restores the ability to test at all during development, without requiring the suite to be reorganised or split first.

What users must be able to rely on

This feature trades completeness for speed, and it is only worth having if that trade is made honestly and visibly.

It is never enabled by default. Selecting a subset of tests is a decision the user makes deliberately, per run or per configuration.
The report never claims a test passed when it was not run. Selecting fewer tests must change the number of tests reported, not fabricate results for the rest. A run of 42 selected tests reports 42 tests and says plainly how many were left out and why. Skipping the rest is not the same as passing them, and the output must not blur the two.
When in doubt, the test runs. Anything PHPUnit does not have reliable information about is treated as affected: tests it has never seen before, tests whose own file changed, tests that did not pass on the previous run, and tests that other tests depend on.
When the ground shifts, everything runs. A change to the PHPUnit configuration, to the project's dependencies, or to the PHP version invalidates what was recorded, and the full suite runs.
The selection is explainable. Users can ask which tests would be selected, and why, without executing anything.
CI is not the target. The intended use is shortening the local feedback loop and getting early signal. It does not replace running the full suite before a change is merged or released, and the documentation should say so directly.

Known limitations

Impact analysis is based on observed execution sees PHP code. It does not see everything a test depends on: fixture files, templates, schema definitions, environment configuration, data files, code reached only through reflection, or a test that asserts something is absent. A change to any of those may affect tests that this feature will not select.

This is a real limitation, not a bug to be fixed later, and users need to understand it before they rely on the feature. It also argues for an extension point: PHPUnit itself cannot know that a change to a template or a migration should re-run a particular group of tests, but the frameworks and tools built on top of PHPUnit do, and they should be able to contribute that knowledge.

Non-Goals

Replaying previous results. Reporting cached outcomes for tests that were not executed would make the suite faster in exactly the way that matters least and misleading in the way that matters most.
Version control integration. PHPUnit should determine what changed by looking at the files it already knows about, and should accept an explicit list of changed files from the user. It should not need to understand branches, rebases, or any particular VCS.
Framework-specific knowledge. Rules about templates, routes, migrations or asset pipelines belong in extensions, not in PHPUnit.

Relationship to work already underway

Two features currently in development change what this one would have to build, and one of them changes whether it is worth building at all.

Caching what a test file contains (#6863, `feature/test-index`)

Selecting fewer tests does not by itself make a run fast. Before PHPUnit can decide that a test has nothing to contribute, it has to load the file that test lives in, and loading test files is most of the cost of a narrow run. That work measured a --group run that selects 98 of a suite's tests at 981 ms without the index and 242 ms with it — the saving is almost entirely in not loading files.

An impact-selected run has exactly that shape: a few dozen tests out of many hundreds. Without the index, the time saved by not running the unaffected tests would be spent discovering them anyway, and the feature would underdeliver on the one thing it promises. These two belong together: "affected by what changed" is a third way of establishing that a file has nothing to contribute, alongside the group and name criteria that work already handles.

It also settles several things this feature would otherwise have to invent, and settles them the same way:

Deciding whether remembered knowledge is still valid: An index entry is used only while every source file it was derived from still hashes the same, parent classes and traits included. Impact data needs precisely that rule on the test side, and should use the same mechanism rather than a second one.
Hashing contents rather than trusting timestamps, for reasons that apply here unchanged.
Failing open: A file that cannot be read or loaded is never remembered, so a later run loads it again rather than skipping past it.
Being explicit that not loading a file is not the same as skipping a test. That work states the distinction outright: a skipped test is part of the suite and is reported; a test in a file that was not loaded is neither. That is the same honesty requirement stated above, already articulated.

Parallel test execution (#6784, `feature/parallel-test-execution`)

Workers ship their results home to the parent process, which merges each unit's coverage into its own and remains the single source of truth for output, logging, results and coverage. Per-test attribution survives that merge, so recording the data this feature needs works under --parallel without anything extra.

Selection size should inform how many workers are started: Starting ten workers to run six selected tests loses more than it saves. The two features are both about making short runs short, and they have to agree on that.
Work is distributed one test class at a time: When only some tests of a class are affected, the unit sent to a worker has to be the selected subset, not the class.
Tests that must run in the main process (those marked as not parallelisable, those requiring process isolation, those whose data cannot be serialised, PHPT files declaring a conflict) must still have their dependencies recorded, or they will silently become tests that are never selected again.
Scheduling: That work notes that recorded durations could seed a longest-first ordering and are not used for it yet. Durations and impact data are both per-test knowledge kept between runs; if this feature lands, they should feed the same scheduler rather than two.

Taken together the three describe one coherent end state: the index avoids loading what cannot matter, impact analysis decides what does matter, and parallel execution runs what is left across all available cores.

Open questions

Is the dependency data PHPUnit can record actually correct enough?

When a test declares #[CoversClass] or #[UsesClass], PHPUnit deliberately narrows the coverage it records to the declared targets. That is the right behaviour for a coverage report and the wrong behaviour for impact analysis: a test that exercises a helper class but does not declare it would have no recorded dependency on that helper, and would therefore not be selected when the helper changes. Silently.

So recording would have to capture what the test executed, independently of what it declares it covers — meaning the impact data and the coverage report would disagree with each other by design. Is that acceptable? If not, this feature cannot be built on observed execution at all, and everything below is moot.

Is requiring a coverage driver a price users will actually pay?

Recording needs pcov or Xdebug, and a recording run is meaningfully slower than a normal run. Users without a driver installed locally get nothing. How many are in that position, and does the feature still pay for itself once the cost of periodically re-recording is counted?

What defines the set of files this feature watches?

<source> is the obvious boundary, but it describes first-party PHP code for the purpose of coverage, not "everything a test might depend on". Changes outside it would be invisible. Is <source> the right boundary, does this need its own configuration, or is the honest answer that anything outside <source> forces a full run?

What is the contract when the selection is empty?

A change that maps to no tests currently has no defined outcome. Is that a successful run, a warning, or a failure? The same question applies when the recorded data is stale or missing: does PHPUnit run everything, or refuse and tell the user to re-record? Both are defensible; picking one late would be a BC problem.

How large is the recorded data, and where does it live?

A per-test-to-per-file map for a suite of any size is substantially larger than what PHPUnit keeps between runs today, and larger than a test index entry. Does it sit alongside that data in the cache directory, or does its size argue for something else? Is it per-machine, or something a team would want to share? If shared, it becomes a format with compatibility obligations.

What happens the first time it selects wrong?

A false negative here does not look like a bug — it looks like a green suite. The failure mode is "PHPUnit told me my tests passed and they did not", which is the single worst bug report this project can receive, and it will arrive regardless of how the feature is documented. Is there a formulation of the output that makes the incompleteness impossible to miss?

Which existing features must work with it on day one?

#[Depends], data providers, process isolation, --order-by, groups and PHPT tests. Some of these are straightforward, some are not.

The harder version of this question is one of sequencing rather than compatibility: this feature needs the test index to deliver what it promises, and should agree with parallel execution about how short runs are made short. Does it wait for both, or is there a first version that stands on its own?

Should recording happen automatically whenever coverage is collected, or only when explicitly requested?

Should the recorded information live with the existing per-test data kept between runs, or separately, given that it is substantially larger?

What is the right output when a change affects no tests — success, or a warning that nothing was verified?

Should a run that selects a subset record dependency data for the tests it did run, or is recording only meaningful for a run of the whole suite?

When only some tests of a class are affected, is the class still the right unit of work to hand to a parallel worker?

Did some exploratory work for this in the past as part of my 'run (probable) defects first'. Looking forward to your work on this.
(and hopefully I have the opportunity again in future to join the party)

It was one of the reasons I introduced the state cache: keep track of costs (time, CPU) of tests to be used to faster and cheaper pipelines, but also what tests are affected by changes to which files. Very interesting puzzle to explore. Hope you have some fun with it, too.

What made it practical was lowering the resolution. Ekstazi showed that tracking dependencies at file granularity, observed dynamically while tests run, is coarse enough to be cheap to collect and precise enough to be useful, which is the granularity this proposal assumes

Eventhough I never submitted a draft or proposal, this worked surprisingly well. My guess is you already did some exploratory coding, yes? :)

Just joining the conversation to mention that obviously that would be tremendously useful for infection.

I recently tackled this after Pest v5 as I still use PHPUnit heavily. I'd love to see this in core. But until then some may find this extension helpful: https://github.com/jasonmccreary/phpunit-tia

Very nice. This will make such a big difference for framework developers. Hypervel's test suite is ~30k now so even with paratest, it's pretty painful. If you need any beta testers please let me know - happy to help.

@jasonmccreary

@jasonmccreary I was not aware of your extension until yesterday, same for what Pest has. What I wrote up yesterday in #6897 (comment) is based on ideas I have had ever since I read Paul Hammant's 2017 article.

@sebastianbergmann

@sebastianbergmann yeah, I just let AI crank something out after Laracon. Excited to see what's in core as it will improve the DX (via options and clearer test output). 👍

I had a better read and I would like to provide some input/feedback. Note that it is as much from a personal usage, not strictly from Infection PoV, although I will note how we can leverage this in infection too.

Is the dependency data PHPUnit can record actually correct enough?

I am not 100% certain of what you had in mind there, but I think there is two distinct very different approaches and each have their use case.

Work from coverage attributes

Since PHPUnit knows all of the tests code and source code (via the configuration), it can generate a graph of dependencies based on the covers attributes. From this, it can derive what tests to execute for a given change.

This is not a perfect TIA. It has quite a few shortcomings, namely tests that have #[CoversNothing] will be considered as "not exercising any code" which is not true. But it has one huge benefit: this can be done before any test is run and without any coverage data. This means the feedback loop can be incredibly fast, even if no test was ever executed before. For infection for example, this would fit in perfectly when executing it against a specific selection:

bin/infection src/Differ/Differ # TIA reduces the tests to execute immediately
# without TIA: will need to execute the whole test suite once first

Whereas currently you're more likely to get better results from specifying the tests instead:

bin/infection tests/Differ

Work with coverage data (without covers attributes)

If a coverage driver is used to construct a dependency graph of what source code tests exercise, I would argue it should NOT be restricted to the covers attributes. Indeed, if you go through the trouble of doing that, it is because the previous solution is not good enough, for example because there is a test with #[CoversNothing] that exercise the changed source code that you want to be executed. For certain the feedback loop will be slower there, but I presume the idea is that it would remain much faster than running the whole test suite.

For infection, this would fit perfectly still for users using --with-uncovered (which allows to mutate code that is not necessarily covered by tests).

If it is restricted by the covers attributes, then IMO it is a much more expensive and convoluted approach that the first one, and would make the scenario described just above useless, both with and without infection.

Is requiring a coverage driver a price users will actually pay?

Hard to say without benchmarks but I assume leveraging a coverage driver would be the simplest way to get this working. So at least if I'm asked whether it is a worthwhile trade to get that feature done? Absolutely.

What happens the first time it selects wrong?

I do not know about that, but I think it would be very helpful to have a "dry run", for either debugging or reporting purposes.

Should recording happen automatically whenever coverage is collected, or only when explicitly requested?

Assuming there is very little extra cost, IMO it would be reasonable to have that by default and allow one to disable it if they wish via a configuration setting.

What is the right output when a change affects no tests — success, or a warning that nothing was verified?

Success (IMO).

One additional scenario to consider

How to handle partial runs. For example:

phpunit # runs the whole test suite, collect data for TIA
# change some source code
phpunit tests/.../*Test.php # manually ran PHPUnit against one test or a subset for some reasons
phpunit --only-impacted # will the previous command have corrupted or updated the data?

I have done the exploratory work that the first open question ("Is the dependency data PHPUnit can record actually correct enough?") above asks for. To answer it, I implemented a throwaway spike.

The spike records, for each test, two sets of source files:

executed — the files the test actually executed, observed by the coverage driver
reported — the files the code coverage report attributes to the test today

The first set is what test impact analysis would have to be built on. The second is what PHPUnit keeps today. The question is how far apart they are.

Getting at the first set required a change to phpunit/php-code-coverage. Inside CodeCoverage::append(), the raw data object that CodeCoverage::stop() returns is mutated in place: the code coverage filter is applied to it, then the executable-lines filter, then the ignored-lines filter, and then it is narrowed to the code coverage targets of the test. Or cleared outright when the test declares that it covers nothing. By the time PHPUnit sees the data, what the test executed is no longer in it. For the spike I added an opt-in to php-code-coverage that keeps a copy of the data as it is after the code coverage filter and before the target filter, along with an accessor for it. It is off by default and costs nothing when it is off.

Two suites were measured, on PHP 8.5 with Xdebug:

PHPUnit's own unit suite — 5,791 tests, 738 source files executed. Useful for scale, but atypical in one important way: the code under test is the test runner, so every test executes a large part of it.
sebastianbergmann/raytracer — 177 tests, 34 source files, requireCoverageMetadata="true", so every test declares its code coverage targets. This is the shape of a normal project that follows the best practice to declare code coverage targets.

Is the dependency data PHPUnit can record actually correct enough?

Not as it is recorded today. Yes, if it is collected before the target filter is applied.

	raytracer	PHPUnit itself
observed (test, file) dependencies	824	406,674
of those, absent from the report data	74.8 %	94.0 %
tests whose two sets are identical	31.1 %	3.4 %
files executed per test (median)	3	68
files reported per test (median)	1	1
tests that executed code but have nothing reported	5 (2.8 %)	182 (3.1 %)
source files with tests observed but none reported	0	25

The two projects lose their dependencies in different ways, and both ways matter.

In PHPUnit's own suite, 25 source files have tests that execute them and no test that the report associates with them at all. A change to src/Event/Value/Test/TestDox.php is executed by 337 tests and would select none.

In raytracer, no file is invisible in that sense — every class has a test that declares it — but the recorded sets are far too narrow:

source file	tests, observed	tests, report
`src/math/Matrix.php`	129	25
`src/Color.php`	91	7
`src/material/Material.php`	76	13
`src/PointLight.php`	24	1
`src/shapes/ShapeCollection.php`	17	1

A change to PointLight.php affects 24 tests. The report data knows about one of them. Selecting on that basis would report a green suite while 23 tests that exercise the changed code were never run, which is the failure mode the issue text calls the worst bug report this project can receive.

So the answer to the open question is that impact data and coverage report data have to disagree with each other by design, and the disagreement is not something to be reconciled later. They answer different questions: the report answers what a test declares it covers, impact analysis answers what a test executed.

The target filter is not the only obstacle

There is a second one, on the PHPUnit side. TestRunner::run() asks Metadata\Api\CodeCoverage::shouldCodeCoverageBeCollectedFor(), which returns false for a test that declares #[CoversNothing], and the driver is then never started for that test. There is no data to un-filter. Recording dependencies for those tests means collecting for them and keeping the result out of the report, which is what the spike does by passing false for the targets to CodeCoverage::stop(). That change alone accounts for the entire runtime cost measured below.

Keeping the two apart

Two scenarios have to work, and they place different demands on the filtering.

Run tests and collect impact data, without generating a code coverage report. No report is built, so nothing has to be narrowed to code coverage targets, and the map from targets to code units does not have to be built at all. What PHPUnit does with those targets beyond the report (validating them, and the check for unintentionally covered code that makes a test risky) hangs off the same machinery and would have to be decided on separately for this mode. The driver still has to run, which is the whole cost of this mode. This is the mode the feature is actually for: the developer running a suite repeatedly does not want an HTML report each time. Note that PHPUnit does not activate code coverage at all today unless a report is configured or an extension asks for it, so this mode needs the runner to activate the driver for impact analysis on its own account. I have not measured a record-only run: both measurements below generate a report, so they overstate the cost of this mode by whatever report generation costs.

Run tests, collect impact data, and generate a code coverage report. One driver run feeds two consumers whose filtering rules are opposites. Because the target filter is applied in place, inside php-code-coverage, the split has to happen there too: the data is handed to the impact recorder before applyCoversAndUsesFilter() and before the clear() that a test declaring CoversNothing triggers, and the report continues to be built from the narrowed data afterwards. Doing it anywhere further out is not possible as by then the information is gone.

That gives three invariants that should hold whatever the eventual design:

The code coverage report is always filtered using code coverage targets, unless the user disables targeting explicitly with --disable-coverage-targeting.
Impact data is never filtered using code coverage targets, in either scenario, and --disable-coverage-targeting therefore must not have an effect on it. That option changes the report, never the impact data.
Collecting impact data must not change the report. In the spike it does not: the text report for tests/unit/Metadata, including uncovered files, is identical with and without recording, down to the last line.

There is another consequence: a test that is risky, incomplete or skipped, and a test that declares CoversNothing, contribute nothing to the coverage report. But they executed code and that information must contribute to the impact data.

What it costs

Recording itself is free. Collecting for tests that do not collect today is not.

	no coverage	coverage	coverage + recording
PHPUnit `unit` suite	4.7 s	49.4 s	50.6 s (+2.4 %)
raytracer, whole suite	5.6 s	33.6 s	87.9 s (+161 %)
raytracer, `unit` suite (172 tests, all declare targets)		0.8 s	0.8 s
raytracer, `integration` suite (5 tests, all `CoversNothing`)		32.8 s	86.9 s

The whole of raytracer's 54 seconds is in those five tests. They are the scene-rendering tests, they are slow, and they run without the driver today precisely because they declare that they cover nothing.

This is not an accident of one project. #[CoversNothing] tends to end up on the broad, slow, end-to-end tests, so "collect for them too" and "recording is nearly free" pull against each other, and PHPUnit's own suite hides the tension because its CoversNothing tests happen to be cheap.

The alternative is to record nothing for such tests and treat them as affected by everything instead. That is just as safe, and free at recording time, but it is the wrong trade. Without a coverage driver, raytracer's five CoversNothing tests take 5.5 s and its other 172 tests take 0.2 s. Treating them as always affected would put a floor of 5.5 s under every impact-selected run, however small the change — five tests would take the feature's entire promise away. Collecting for them costs 54 s once, on a run that records, and runs that record are rare compared to runs that select.

So: collect for tests that declare CoversNothing, and keep what is collected out of the coverage report. That is what the spike does, and it is what the measurements above include.

It does mean that recording is not free for every suite, which bears on another of the open questions: recording should be something the user asks for, not something that happens automatically whenever code coverage is collected. A suite shaped like raytracer's would otherwise see every coverage run get 2.6× slower without having asked for anything.

The driver remains the price of admission in any case: 10.5× on PHPUnit's suite, 6× on raytracer, before any recording.

How large the data is

	raytracer (177 tests)	PHPUnit (5,791 tests)
as recorded, file paths interned	25 KiB	2.0 MiB
gzipped	3 KiB	85 KiB
as a reverse map, file → tests	20 KiB	2.4 MiB
without interning the paths	65 KiB	31 MiB
per test	143 bytes	351 bytes

Roughly 150 to 350 bytes per test, and the difference between the two is mostly PHPUnit's self-hosting. For comparison, the test run history file for PHPUnit's own suite is 769 KiB. So this is larger than what is kept between runs today, but of the same order, and it can sit next to it. Interning the file paths is not optional at this size.

Would it pay off

A change to one source file selects, from observed data, a median of 15 of raytracer's 177 tests (8.5 %) and a median of 15 of PHPUnit's 5,791 (0.3 %, with 501 of 738 files selecting under 1 % of the suite). PHPUnit's maximum is the entire suite, which is again due to self-hosting: a change to the runner really does affect every test.

The report data selects less, a median of 4 tests for raytracer, which is exactly the point. It is smaller because it is wrong.

Two limitations that showed up

Seven of the 34 files in raytracer's <source> are executed by no test at all. One of them is genuinely untested; the other six are exception classes, whose files contain no executable lines at all and can therefore never be observed. A change to any of them selects nothing, from either data source, and no amount of observation fixes it, because there is nothing to observe. Such a file has to be treated as unknown, and unknown has to mean that everything runs.

The <source> boundary is not really a policy choice either. The Xdebug driver installs the list of files from <source> into Xdebug itself via xdebug_set_filter(), so nothing outside it is ever reported; pcov collects everything and intersects afterwards. Anything outside <source> is invisible to impact analysis unless the driver is asked to collect more, and with Xdebug that is not free. The honest answer to that open question is the one already suggested in it: a change outside <source> forces a full run.

What this settles

The data PHPUnit can record is good enough, but only if it is collected before the code coverage targets are applied, and only if it is also collected for tests that declare CoversNothing. Two changes, not one.
The recording itself is not what costs; the driver is. Collecting for tests that declare CoversNothing can be expensive, and is still worth it: the cost falls once on a run that records rather than on every run that selects.
Recording should be requested explicitly rather than happening whenever code coverage is collected, for the same reason.
The size is manageable and can live alongside the data PHPUnit already keeps between runs.
Impact data and the coverage report will disagree, permanently and by design. That has to be documented rather than fixed.

Superseded by #6919.

Test Impact Analysis #6897

Description

Summary

Motivation

Where this idea comes from

Proposed Feature

Recording

Selection

Use Cases

Keeping the feedback loop short while working

Pre-commit and pre-push checks

Fast signal on pull requests

Understanding the impact before making a change

Spotting untested changes

Working productively in large or inherited suites

What users must be able to rely on

Known limitations

Non-Goals

Relationship to work already underway

Caching what a test file contains (#6863, feature/test-index)

Parallel test execution (#6784, feature/parallel-test-execution)

Open questions

Is the dependency data PHPUnit can record actually correct enough?

Is requiring a coverage driver a price users will actually pay?

What defines the set of files this feature watches?

What is the contract when the selection is empty?

How large is the recorded data, and where does it live?

What happens the first time it selects wrong?

Which existing features must work with it on day one?

Should recording happen automatically whenever coverage is collected, or only when explicitly requested?

Should the recorded information live with the existing per-test data kept between runs, or separately, given that it is substantially larger?

What is the right output when a change affects no tests — success, or a warning that nothing was verified?

Should a run that selects a subset record dependency data for the tests it did run, or is recording only meaningful for a run of the whole suite?

When only some tests of a class are affected, is the class still the right unit of work to hand to a parallel worker?

Activity

epdenouden commented on Aug 6, 2026

epdenouden commented on Aug 6, 2026

theofidry commented on Aug 6, 2026

jasonmccreary commented on Aug 6, 2026

binaryfire commented on Aug 7, 2026

sebastianbergmann commented on Aug 7, 2026

jasonmccreary commented on Aug 7, 2026

theofidry commented on Aug 7, 2026

Work from coverage attributes

Work with coverage data (without covers attributes)

One additional scenario to consider

sebastianbergmann commented on Aug 23, 2026

Is the dependency data PHPUnit can record actually correct enough?

The target filter is not the only obstacle

Keeping the two apart

What it costs

How large the data is

Would it pay off

Two limitations that showed up

What this settles

sebastianbergmann commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Caching what a test file contains (#6863, `feature/test-index`)

Parallel test execution (#6784, `feature/parallel-test-execution`)