perf(stats): reduce allocations and comparator-cache churn in stats generation - #21506
Conversation
Cut per-item work in the stats factory extractors that run per module, reason and chunk origin on large builds: - assetsByChunkName: iterate the file->chunks maps directly instead of spreading both into one intermediate array of entries. - module$visible: read pre/post-order index once each instead of twice. - moduleReason: hoist originModule/resolvedOriginModule/dependency and compute the readable identifier once instead of twice. - chunkOrigin: compute the module identifier once instead of twice. - moduleTraceItem: collect dependencies into a Set in one pass instead of chaining spread/filter/map plus a second Set.
Profiling detailed stats on a 2000-module build showed ~30% of stats time in StatsFactory._create and a large chunk in comparator cache churn: - StatsFactory sorted via concatComparators(...c, keepOriginalOrder(items)). keepOriginalOrder returns a fresh closure each call, so the concat cache always missed and allocated a new WeakMap per sort. Replace with an inline stable sort that applies the comparators then an original-index tiebreaker, keeping identical ordering (and stability on engines without a stable sort) without touching the caches. - sortByField rebuilt a fresh compareSelect selector per sort; memoize per field so the selector cache stays warm. - module.reasons and the module sorters rebuilt their compareSelect closures on every sort (reasons runs once per module). Build the pure reason comparators once at module scope and cache the moduleGraph-bound module comparators per moduleGraph. Cuts ~9% off stats generation and ~27% of its retained heap on the benchmark; stats snapshots are unchanged.
…bject The module, module$visible and moduleReason extractors built a fully-typed literal and then Object.assign'd it onto the target. That allocates an intermediate object and copies every field on the hottest per-module and per-reason paths. Assign the fields straight onto the target in the same order (identical output); moduleReason self-time dropped ~60% in profiling.
Extend the direct field assignment already used by the asset extractors to chunk, chunkOrigin, moduleIssuer and profile, dropping the intermediate typed literal + Object.assign copy. Output is identical; this trims one object allocation and copy per chunk/origin/issuer/profile item, helping chunk- and asset-heavy builds that the module-heavy path doesn't stress.
🦋 Changeset detectedLatest commit: fd274ea The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
This PR is packaged and the instant preview is available (4ae3518). Install it locally:
npm i -D webpack@https://pkg.pr.new/webpack@4ae3518
yarn add -D webpack@https://pkg.pr.new/webpack@4ae3518
pnpm add -D webpack@https://pkg.pr.new/webpack@4ae3518 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #21506 +/- ##
=======================================
Coverage 93.57% 93.57%
=======================================
Files 619 619
Lines 73232 73296 +64
Branches 21105 21104 -1
=======================================
+ Hits 68526 68588 +62
- Misses 4706 4708 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…rift generate-types hoists the helper's JSDoc block onto the next emitted public interface when the function sits between the module typedefs and the class, which made types.d.ts differ from the committed copy and failed lint:special. Moving the helper below the class (before module.exports) keeps its JSDoc from attaching to a public type; behavior and output are unchanged.
Merging this PR will improve performance by 25.42%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Memory | benchmark "many-chunks-esm", scenario '{"name":"mode-production","mode":"production"}' |
9.6 MB | 7.6 MB | +25.42% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/stats-performance-memory-gjiwr0 (fd274ea) with main (61d4136)
Types CoverageCoverage after merging claude/stats-performance-memory-gjiwr0 into main will be
Coverage Report |
|
Re the CodSpeed memory report: I believe the
None of these grow with graph/output size. Everything else in the diff either removes allocations (direct field assignment instead of a temp object + Happy to dig in further if it reproduces on a re-run, but it looks like environment noise to me. Generated by Claude Code |
|
Note on the red Generated by Claude Code |
|
The
That test passes cleanly under Node here ( Generated by Claude Code |
Summary
Stats generation runs on every build and, on large projects, spends almost all of its time in
StatsFactoryand the per-module/per-reason extractors. Profiling a 2000-module build with detailed stats showed a throwawayWeakMapbeing allocated on every sort —concatComparators(..., keepOriginalOrder(items))never hits its cache because the tiebreaker closure is fresh each call — plus per-item churn from rebuilt comparators, redundant module-graph lookups, and intermediate objects in the extractors. This PR removes those without changing output: ~20% faster stats generation and ~27% less retained heap on that benchmark (toJson/toString). Refs n/a.What kind of change does this PR introduce?
perf
Did you add tests for your changes?
No — this is a behavior-preserving refactor; the output is byte-identical and already covered by the existing 156 stats snapshots (
StatsTestCases), and the watch/hot suites (2635 tests) pass unchanged.Does this PR introduce a breaking change?
No.
If relevant, what needs to be documented once your changes are merged or what have you already documented?
n/a
Use of AI
AI (Claude) was used to profile the stats phase (CPU and heap), locate the hotspots, and implement and verify these optimizations; every change was validated against the existing stats, watch, hot, and compiler test suites with byte-identical output.
Generated by Claude Code