Repository navigation
NgccProcessor slowing down subsequent builds with AngularCompilerPlugin #15078
Description
I'm using the AngularCompilerPlugin for a medium-sized project and I've observed a slowdown in build times after enabling Ivy which appears to come from ngcc. When Ivy is enabled the AngularCompilerPlugin runs the NgccProcessor which is supposed to compile the node_modules for Ivy on the first run, however in my case any subsequent builds are slowed down as well.
I have a hacky way to disable the NgccProcessor which I've used to measure the differences in build times. Here's what it looks like:
class MyAngularCompilerPlugin extends AngularCompilerPlugin {
public apply(compiler: Compiler): void {
super.apply(compiler);
compiler.hooks.environment.tap('angular-compiler', () => {
(this as any)._compilerHost.ngccProcessor = undefined;
});
}
}Here are the timings for 5 production builds on my own machine. Note that these times are after ngcc has been run once over the node_modules in a postinstall script.
// Without ngcc (80737ms on average)
84520ms
72547ms
81733ms
85043ms
79844ms
------------------------
// With ngcc (94208ms on average)
93741ms
93714ms
92922ms
97220ms
93444ms
The 15s slower build time for a production build isn't a big deal locally, however the difference becomes even larger when running it on our CI server. For our production builds we need to run about 30 Webpack builds (10 at a time in parallel), about 20 which go through the AngularCompilerPlugin. Disabling the NgccProcessor on our CI server reduced our build time from around 10 minutes to about 6 minutes. Furthermore keeping the NgccProcessor led to some flakiness in our builds, presumably because 10 different ngcc processes were trying to access file at the same time. Our build script looks something like this:
- Run
npm ci. - Run
ngccprogrammatically as follows:
const ngcc = require('@angular/compiler-cli/ngcc');
ngcc.process({
basePath: path.join(projectRoot, 'node_modules'),
compileAllFormats: false,
propertiesToConsider: ['browser', 'module', 'main'],
createNewEntryPointFormats: true
});- Run
webpack --mode=production30 times by splitting them into chunks of 10 that are run concurrently. We have some logic that tries to make sure that we always run the maximum number of builds, but it stays the same no matter whetherngccis enabled so it shouldn't have an effect on the times.
I can see a couple of ways to get around this:
- Short term: add an option to the
AngularCompilerPluginconfig that allows users that know what they're doing to disable theNgccProcessor. - Long term: rework either
ngccorNgccProcessorso that it doesn't have to spend time analyzing thenode_modulesafter it has been run already.
cc @filipesilva
Activity
alan-agius4 commentedon Jul 15, 2019
@crisbeto, which version of @angular/compiler-cli are you using?
@petebacondarwin was your PR for improving NGCC performance merged and released?
7 remaining items
gkalpak commentedon Jul 16, 2019
FYI, we've run into the same issue with the docs examples (which essentially share the same
node_modules/directory via symlinks). See angular/angular#30593 for details on the problem and the solution.petebacondarwin commentedon Jul 16, 2019
Actually I think that we don't need the flag... if the node_modules have already been compiled by ngcc then the CLI+ngcc should not need to write any files, so there should be no flakes........
gkalpak commentedon Jul 16, 2019
Yeah, exactly, that's how we solved it in angular/angular#30593 for angular.io docs examples 😉
No flakes (of that particular kind 😞) since 💪
alan-agius4 commentedon Jul 16, 2019
@crisbeto, seeing all the comments above. I don’t think there is anything to action.
Feel free to ping me on slack if you feel otherwise.
crisbeto commentedon Jul 19, 2019
@petebacondarwin @alan-agius4 I'm still seeing the flaky behavior where
ngccthrowsUnexpected end of JSON inputafter I updated to8.2.0-next.2.gkalpak commentedon Jul 20, 2019
@crisbeto, are you explicitly running
ivy-ngccupfront or implicitly via the cli?crisbeto commentedon Jul 20, 2019
I'm running it once up-front like this after I
npm install:Afterwards it gets run implicitly by the
NgccProcessorin theAngularCompilerPlugin.gkalpak commentedon Jul 20, 2019
Could it be that those properties do not correspond to the es2015 format (which the cli wants to compile)?
crisbeto commentedon Jul 20, 2019
That call goes through fine. What I'm running into is that when I'm running 10 different Webpack builds in parallel, once every 10 or 15 times it flakes with
Unexpected end of JSON input. My best guess is that it's due to multiple ngcc instances trying to read the same files at the same time.gkalpak commentedon Jul 20, 2019
I am not implying the call doesn't go through fine 😄 Here is what I mean:
The
Unexpected end of JSON inputcan happen when apackage.jsongets corrupted. This in turn can happen if multiple ngcc instances try to update apackage.jsonsimultaneously. Ngcc updatespackage.jsons to add info about which formats have been processed.Based on the above, the trick to avoid the error is to ensure that ngcc will not need to update any
package.jsonduring the concurrent cli builds. The way to achieve this is to ensure that all necessary formats will have been compiled in the initial standalone ngcc run.So, you need to ensure that ngcc will have processed all formats that the cli will request afterwards.
crisbeto commentedon Jul 20, 2019
That makes sense, although from looking at the logs it doesn't seem like it's doing any more compilation, because I'm not seeing any more of those
Compiling something as ...messages.petebacondarwin commentedon Jul 21, 2019
It is worth taking a look at the package.json after the original ngcc run and then after the CLI compilation?
We should note that multiple properties in
package.jsonmap to the same code format. E.g.moduleandfesm5andesm5are the same "format".So although you have built the
esm5format in the first run of ngcc (via themoduleproperty), there might be a case where the CLI attempts to build an additional entry point, which happens to map to a format that has already been built. In that case you would not see a "Compiling..." message but ngcc might still write to the package.json to say that an entry-point has been processed.angular-automatic-lock-bot commentedon Sep 9, 2019
This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.
Read more about our automatic conversation locking policy.
This action has been performed automatically by a bot.