Repository navigation
Concatenated modules have nondeterministic ordering when side effect analysis is on #15866
Description
Bug report
Minimal repro here: https://github.com/pgoldberg/concatenate-modules-bug
Discussed in: #15844
What is the current behavior?
When module dependencies internally export * and then re-export in the root index.js, webpack has nondeterministic output, because modules can be concatenated in different orders across separate runs. This can be "resolved" by turning off optimization.sideEffects.
I've dug a bit and found the following:
In ConcatenatedModule.js, the references are sorted here, and if the sourceOrder is the same, it falls back to the original order in the references array, which is not always the same and causes the modules to get concatenated in a different order. I was get the builds to always produce the same output by adding some hacky sorting logic there to sort the modules by reference.connection.dependency.loc.start.line/referenceconnection.dependency.loc.start.column (discussed here).
Moving up to the SideEffectsFlagPlugin, the modules array received from the optimizeDependencies compilation hook is not always in the same order, which seems to be what causes the moduleGraph.outgoingConnections to be out of order in ConcatenatedModule.
Digging more, it seems like this is a race condition in Compilation.js that's causing the modules to get processed out of order. I was able to see that factory.create here sometimes finishes first for inner_dep2 instead of inner_dep in the minimal repro, and that's when the output ends up changing to concatenate inner_dep2 and its dependencies before inner_dep, because inner_dep2/index.js gets added to Compilation.modules before inner_dep/index.js, which then causes the downstream issues with SideEffectsFlagPlugin and ConcatenatedModule.
If the current behavior is a bug, please provide the steps to reproduce.
Minimal repro here: https://github.com/pgoldberg/concatenate-modules-bug
What is the expected behavior?
Builds always produce the same output when run against the same input.
Other relevant information:
webpack version: 5.72.1
Node.js version: 16.13.0
Operating System: macOS
Additional tools:
Activity
vankop commentedon May 27, 2022
I have checked this for side effects and it works with side effects. So problem only for
sideEffects=false( or when webpack detects no side effects ) and this affects only hash..vankop commentedon May 27, 2022
@pgoldberg Could you try fix above ☝️?
pgoldberg commentedon May 27, 2022
Yep this does fix the issue!
The only potential issue I see is that
compilation.modulesis still out of order, so if some external plugin also relies on the order ofcompilation.modules, then it would also have an issue related to this. Would it make more sense to sortcompilation.modulesupstream of here? Ifcompilation.modulesalways has deterministic ordering, then module concatenation will always have deterministic ordering (as far as I can tell). I know that factorizing in different orders shouldn't lead to different concat results, but it looks like it does because if you follow the callbacks up from thefactory.createcallback incompilation.factorizeModule, you eventually get tocompilation.addModule, which results in the module that finishes factorizing first getting inserted to thecompilation.modulesset first.Or is
compilation.modulesnot intended to have a reliable order?vankop commentedon May 28, 2022
yes, they not intended.