Repository navigation
NGCC is non-hermetic and produces different output depending on source location in file system #39293
Description
🐞 bug report
Affected Package
angular/compiler-cli
Description
NGCC produces different output - specifically, in sourcemap transformation - depending on source location in file system.
I have two copies of the same project checked out in two places:
D:/myproj
D:/myproj-other/myproj
If I compare node_modules/@angular/animation between the two (this is just an example), the "sources" fields on the source maps in node_modules/@angular/animations/fesm2015 (again, just an example) are different
D:/myproj-other/myproj/node_modules/@angular/animations/fesm2015/animations.js.map
{"version":3,"file":"animations.js","sources":["../../../../../../packages/animations/src/animation_builder.ts","../../../../../../packages/animations/src/animation_metadata.ts","../../../../../../packages/animations/src/util.ts","../../../../../../packages/animations/src/players/animation_player.ts","../../../../../../packages/animations/src/players/animation_group_player.ts","../../../../../../packages/animations/src/private_export.ts","../../../../../../packages/animations/src/animations.ts","../../../../../../packages/animations/public_api.ts","../../../../../../packages/animations/index.ts","../../../../../../packages/animations/animations.ts"],"names":
...
D:/myproj/node_modules/@angular/animations/fesm2015/animations.js.map
{"version":3,"file":"animations.js","sources":["../../../../../packages/animations/src/animation_builder.ts","../../../../../packages/animations/src/animation_metadata.ts","../../../../../packages/animations/src/util.ts","../../../../../packages/animations/src/players/animation_player.ts","../../../../../packages/animations/src/players/animation_group_player.ts","../../../../../packages/animations/src/private_export.ts","../../../../../packages/animations/src/animations.ts","../../../../../packages/animations/public_api.ts","../../../../../packages/animations/index.ts","../../../../../packages/animations/animations.ts"],"names":[
...
Note that the number of "../" is different between them.
the __ivy_ngcc_bak file shows the original:
{"version":3,"file":"animations.js","sources":["../../../../../../packages/animations/src/animation_builder.ts","../../../../../../packages/animations/src/animation_metadata.ts","../../../../../../packages/animations/src/util.ts","../../../../../../packages/animations/src/players/animation_player.ts","../../../../../../packages/animations/src/players/animation_group_player.ts","../../../../../../packages/animations/src/private_export.ts","../../../../../../packages/animations/src/animations.ts","../../../../../../packages/animations/public_api.ts","../../../../../../packages/animations/index.ts","../../../../../../packages/animations/animations.ts"],
...
Unfortunately, this difference is significant if you want to do any sort of build caching between machines as the hashes of the input files of any dependencies on ngcc-processed node_modules are different. This has caused me a lot of pain as I am trying to set up Bazel in our system. I had problems with cache misses between our build servers and developer machines that appears to be caused by this. As a temporary workaround I have had to check in our node_modules, along with ngcc generated files.
I don't have a firm grasp on the details of this bug, but it appears to me arise from the following code in:
node_modules/@angular/compiler-cli/src/ngtsc/sourcemaps/src/source_file.js
var sourceMap = {
version: 3,
file: this.fs.relative(sourcePathDir, this.sourcePath),
sources: sources.map(function (sf) { return _this.fs.relative(sourcePathDir, sf.sourcePath); }),
names: names,
mappings: sourcemap_codec_1.encode(mappings),
sourcesContent: sources.map(function (sf) { return sf.contents; }),
};The line
sources: sources.map(function (sf) { return _this.fs.relative(sourcePathDir, sf.sourcePath); }),ends up forming a relative path from
D:/myproj/path/to/file
to
D:/packages/path/to/file
Obviously this is going to be different if the first path is
D:/my/nested/checkout/myproj/path/to/file
🌍 Your Environment
Angular Version:
10.1.5
Anything else relevant?
Windows
Activity
petebacondarwin commentedon Oct 15, 2020
Thanks for this detailed report @matthewjh - I will take a look at it tomorrow.
matthewjh commentedon Oct 15, 2020
Thank you, @petebacondarwin :)
petebacondarwin commentedon Oct 16, 2020
So as far as I can tell, this is kind of an issue with the source-maps generated by the Angular project in the first place.
Since the file paths of original source files are pointing way way up the tree, (e.g.
../../../../../../packages/...) then if the location of the source-map itself is not deep enough in the tree then of course it will produce different relative paths.This is equivalent to doing:
This should put you in
/a/b/xright?But if you did:
Then you end up in
/xI appreciate that is not very helpful. So I will take a look into whether we can make ngcc's source-map handling more resilient to this...
petebacondarwin commentedon Oct 16, 2020
So I guess, if the original source path points so far up the tree that we hit the root, then we should assume that this file never exists, and so just leave the path alone rather than modifying it to be relative to the current source-map. What do you think?
matthewjh commentedon Oct 16, 2020
Does that mean that if I checkout the source code at a sufficiently deep path that running ngcc on these files does not hit the root, I may end up with different source files?
I'm assuming by root you mean the root of the file system tree, i.e.
C:/
or
D:/
Is there a way to invoke that logic (fallback to the original) more generally in the event that the path "breaks out" of the current workspace/project subtree? So in D:/my-path/my-proj/node_modules/thing, ../../../ would break out of my-proj and the source map path would left alone. But in D:/my-path/my-proj/node_modules/thing/a/b/c, ../../../ would be fine.
petebacondarwin commentedon Oct 16, 2020
So, yes I did mean
C:/etc when I said root.I think we have a problem here generally, in that even if we identified files that broke out of the root, and "fixed" the paths to them. If you did move to a deeper path, then this logic might not kick in and then we would be left with different paths again.
(I think this might have been what you are saying).
Perhaps a better approach is to identify when we have been given the source content inline (as in this case) and then just always modify the file path to something consistent (e.g. just the basename, stripping off any leading path segments).
petebacondarwin commentedon Oct 16, 2020
I created a possible fix.... #39302
petebacondarwin commentedon Oct 19, 2020
Tracking the underlying issue for this in #14021.
I think there is still room for ngcc (and our source-map merging tooling) to be more resilient to these dodgy source paths anyway. So I will keep working on this in #39302
matthewjh commentedon Oct 19, 2020
Thanks, @petebacondarwin, for looking at this so quickly. For what it's worth I have tested your fix with our codebase, and there are no longer differences between machines / checkout locations, meaning build caching works perfectly. 👍
angular-automatic-lock-bot commentedon Nov 19, 2020
This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.
Read more about our automatic conversation locking policy.
This action has been performed automatically by a bot.