MultiCompiler didn't release memory after one compiler is finished.

Bug report

What is the current behavior?
I use MultiCompiler configs.
If i run them one by one, Webpack can build them all.

const config = {}
webpack(config)

If i run them together, OOM appeares.

const configs = [config1, config2, config3, config4]
webpack(configs)

If the current behavior is a bug, please provide the steps to reproduce.
MultiCompiler uses more memory than single compiler.

What is the expected behavior?
Each compiler should work separately. when one is finished, who's memory should be released.

Other relevant information:
webpack version: 5.70.0
Node.js version: 14.17.1
Operating System: macOS 11.6
Additional tools: none

Part of configuration
entries (each compiler): 2 + 159 + 230
cache: dsiabled
mode: 'production'
loaders: babel / postcss / vue-laoder
minimizer: terser / css-minimizer-webpack-plugin
devtool: false
target: ['web', 'es5']

in multicompiler mode next compilation could depends on previous one https://github.com/webpack/webpack/blob/main/schemas/WebpackOptions.json#L396 . but maybe this could be improved.

@vankop

in multicompiler mode next compilation could depends on previous one

@vankop Is there any way to stop such behavior because in my scenario, there is no dependencies between multicompilers

not sure.. lets keep tracking issue for now.

The strange thing is even all assets are emited. OOM still appeares at last.

@Edge00

@Edge00 Can you provide OOM message?

@alexander-akait

@alexander-akait Sure

<--- Last few GCs --->

[94438:0x1048db000]   239481 ms: Scavenge (reduce) 3886.2 (4100.5) -> 3886.0 (4103.8) MB, 11.5 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure
[94438:0x1048db000]   239494 ms: Scavenge (reduce) 3887.0 (4108.3) -> 3886.9 (4109.5) MB, 11.9 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure
[94438:0x1048db000]   239510 ms: Scavenge (reduce) 3888.1 (4102.8) -> 3888.0 (4106.0) MB, 12.2 / 0.0 ms  (average mu = 0.090, current mu = 0.033) allocation failure


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10130d5e5 node::Abort() (.cold.1) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 2: 0x1000b2289 node::Abort() [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 3: 0x1000b23ef node::OnFatalError(char const*, char const*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 4: 0x1001f68c7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 5: 0x1001f6863 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 6: 0x1003a47e5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 7: 0x1003a628a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 8: 0x1003a19b5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
 9: 0x10039f2e0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
10: 0x1003ad9ea v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
11: 0x1003ada71 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
12: 0x100376cce v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
13: 0x10078f54d v8::internal::IncrementalStringBuilder::Extend() [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
14: 0x1004bd866 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
15: 0x1004bf3dd v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
16: 0x1004c37c4 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
17: 0x1004c07ee v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
18: 0x1004ba503 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
19: 0x1002a0253 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]
20: 0x100a82099 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/qiongen/.nvm/versions/node/v14.17.1/bin/node]

@Edge00

@Edge00 Cache enabled? What is mode? If you can will be great to provide configuration

@alexander-akait

@alexander-akait Sorry. The configuration are generated by some complicated logic. I supplemented some configuration information on the issue description. If you need more information please let me know. Thank you!

This issue had no activity for at least three months.

It's subject to automatic issue closing if there is no activity in the next 15 days.

Issue was closed because of inactivity.

If you think this is still a valid issue, please file a new issue with additional information.

@vankop

@vankop Still valid?

fix: also release per-module presentationalDependencies and build-pha…

fix: drop NormalModule._source after seal, simplify the release helper

revert: stop nulling NormalModule._source — it breaks the cache and s…

fix: release per-child Compilation heap pressure in MultiCompiler

docs(AGENTS): tighten brevity and add perf/memory guidance

fix: release per-child Compilation heap pressure in MultiCompiler

Navigation Menu

Repository navigation

MultiCompiler didn't release memory after one compiler is finished. #15521

Bug report

42 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

MultiCompiler didn't release memory after one compiler is finished. #15521

Description

Bug report

Activity

vankop commented on Mar 12, 2022

Edge00 commented on Mar 12, 2022

vankop commented on Mar 12, 2022

Edge00 commented on Mar 12, 2022

alexander-akait commented on Mar 13, 2022

Edge00 commented on Mar 13, 2022

alexander-akait commented on Mar 13, 2022

Edge00 commented on Mar 13, 2022

webpack-bot commented on Jun 12, 2022

webpack-bot commented on Jun 27, 2022

alexander-akait commented on Jun 28, 2022

42 remaining items

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions