localized components containing i18n tags will produce identical chunk hashes when building with --localize

🐞 Bug report

Command (mark with an `x`)

Description

I have an application that is being translated in several (21 at the moment) languages using the Angular i18n mechanism. I'm compiling the app AOT, therefore I need a separate build for each locale. To achieve this I setup my Azure pipeline to build the app using multiple agents in parallel, one for each locale, and it works like a charm. However, this is very resource-consuming, so I'm updating to Angular 9 in order to use the --localize option, which perfectly fits my needs.
After the update I can build the app once and translate it to all languages in a few seconds, which is amazing.
I am now facing this issue: when I build my application for all locales, the resulting chunk hashes are identical. As a result, browser cache is not invalidated and files are not re-requested, so the app is not translated.

🔬 Minimal Reproduction

clone this repo
run npm install
run ng build --localize
check the dist folder: file content for main.js is different for each locale (template text is translated), while the hashes are the same

🌍 Your Environment


Angular CLI: 9.1.0
Node: 10.19.0
OS: win32 x64

Angular: 9.1.0
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.901.0
@angular-devkit/build-angular     0.901.0
@angular-devkit/build-optimizer   0.901.0
@angular-devkit/build-webpack     0.901.0
@angular-devkit/core              9.1.0
@angular-devkit/schematics        9.1.0
@ngtools/webpack                  9.1.0
@schematics/angular               9.1.0
@schematics/update                0.901.0
rxjs                              6.5.5
typescript                        3.8.3
webpack                           4.42.0

Anything else relevant?
In my scenario, the user language preference is stored in the cookies, and a shell app serves the corresponding folder. I can't add the locale to the URL.
Therefore in my angular.json I set baseHref="/" for all locales.

I am aware of this issue and the PR mentioned there, however the issue is still not resolved.

Duplicate of #16526

For our projects we do path based routing in our ingress, so we opted to do localization with other methods (ie cookies). Without different hashing on the localized files, the browser cache will return the same locale even though the user switched locales, and a different index file is being served.

We dont need to flatten anything. Our server-side picks up the cookie and serves the right files.

That sounds like the Cache-Control header is wrong then? If the response is contingent on a cookie which the browser doesn't key its cache by, then I think it's not safe to let the browser cache that URL.

I can see that marking those URLs as uncacheable could reduce performance for users who don't change languages, but I think that's a consequence of not putting the locale in the URL. Using a different hash is just another form of making the URL locale-dependent so the locale is taken into account in the browser cache key. Whether that's done via a locale-dependent hash, or an explicit locale path / query parameter is an implementation detail to the browser cache.

I don't think this is related to caching at all. You can reproduce it like explained above with just a regular "ng build --translate" - you end with 2 folders, one for each locale - so far so good!

But if you deep-dive it you notice that your main.js bundles (you have one of those for each language) is messed up. You end up with different bundles because there are differences due to language. But the HASH of all these main.ts bundles is the same.

Now depending on your solution you might end up with rewrites that (for the users browser) appear to be from the same path - that would be no problem, but since your ENGLISH-main.ts and your GERMAN-main.ts have the same hash you end up with issues when you switch the language.

Given, there may be ways to host it differently - but still: how could that NOT be a bug? The Hash of my german main.ts file is obviously based on the content of my english main.ts file. That's just wrong, it should be based on the content of my german main.ts, right?

@clydin

Given, there may be ways to host it differently - but still: how could that NOT be a bug? The Hash of my german main.ts file is obviously based on the content of my english main.ts file. That's just wrong, it should be based on the content of my german main.ts, right?

As I understand it, I believe we use Webpack's [hash] replacement here, not [contenthash]. The hash value is not based on file content, it's only guaranteed to be unique within a build. This SO answer seems to have a good breakdown of the differences. @clydin is that understanding correct?

Given that generated resources shouldn't be loaded into the same page (you shouldn't be loading a French chunk into a German application), then I don't think it's necessarily wrong to see duplicate hashes across different locales.

Thanks for explaining and the SO link! I think I understand, but it still feels wrong and there might be some bugs with that implementation - if I build my project while only changing the german translation file the hash would still be unchanged, right? That would lead to people not getting the newest version of my german main.ts file! So basically only changing translations without touching a source file is not supported with the current build?

The question is: is there a good reason NOT to use [contenthash] or maybe add an optional flag for that? This puts a strong opinion/restriction on how files are served. For me it also seems a bit also unintuitive (because of what I believe a file-hash should do)

Example

will try to explain my approach while hosting my mini-homepage https://budisoft.at that lead to this problem:

on my server there is a folder de_DE and another folder en_US, containing the two localized applications
I set some rules on my reverse proxy (based on "Accept-Language" and a Custom Cookie) to rewrite all URLs to /de or /en_US accordingly
the basehref is just set to "/" - my rules ensure that the correct bundles are loaded

This was a naive - but I think reasonable - approach and would work perfectly with [contenthash], but fails with [hash] (wrong language bundle get's loaded from cache)

Solution

Thanks to your response I think I finally understand what I'd have to do instead of dirty workaround:

rewrite requests just for the index.html.
keep the baseHrefs like default, e.g. "en_US" and "de"
now my english index.html would request a different file location (e.g. https://budisoft/**en_US**/main.03b15c0004fb5dfc.js) than my german version, even when the hash is the same.

I will implement it that way, but still hope that there can be a solution that enables some flexibility while serving the bundles sometime.

Looking over the comments quickly, it sounds like most of the motivation here comes down to wanting to put multiple locale outputs into a single flat directory, is that the ask here? Why exactly do folks want to do this, does it make some server configuration or URL pathing easier? I think a better understanding of the motivation here would help us identify the right solution.

Thanks for following up! I'll try to elaborate the use case in our application, please let me know if my explanation is not clear.

Our application is typically gated by a user login. To determine the language of the application, it will look at a user's language preferences, saved in their user account. This ensures that they are served the correct language wherever they log in. If the user is not yet logged in, or the user's language is not yet set, there is logic to determine the most preferable fallback language based on the environment in which the application is deployed, as well as the user's Accept-Language list.

Once the correct language is served, we store it in the browser's cookies, ensuring that these checks and redirects do not happen again until a different user - with a different preferred language - signs in.

Because of this process, we do not use the browser URL to determine which language should be served.

This does not mean that we can't have the language in the URL - we could determine the user's preferred language and then redirect to the correct URL once we know, this is true.

However, there are other features of the application that complicate this:

Users can bookmark direct links within the application - or, more relevant in this case, administrators might provide their users with these bookmarks. These links should not contain language info in the URL, because they could cause the application to be served in the wrong language. We would require additional steps to detect that the user's preferred locale does not match the URL, and redirect accordingly. Unlike our current process, this is something that would have to be done every time the user accesses the bookmark.
Users can also share file links with one another, even with anonymous users. Similar to bookmarks, having a language code in the URL could cause the file to be served in the wrong language. Admittedly, this could probably be resolved by stripping languages codes from share links.

While none of these individual problems might be insurmountable, it makes much more sense for our use case to simply not use the browser URL for the language. It is there that we run into this issue - switching languages is unreliable when the file hashes for each language are the same, when these files are served with the same URL. We currently use the workaround I mentioned in an earlier comment of adding language codes to file hashes, but this is somewhat slow and likely to break in the future.

Hey Bart, I think you are a bit in the same "mental trap" as I was. Yes, URLs to a specific content-page should not contain a language url, to ensure deep-linkability across languages.
Example: https://your-page/order/123456 <= this is a link to one specific order and should work, regardless of language.

But this is only important for your own pages/routes! This doesn't matter for JavaScript bundles though. Let me explain:

If you give the link above to a german user your reverse-proxy will probably use your cookie (+ fallback) to determine the correct index.html and retrieve it from a file system like this:
/opt/data/your-page/de/index.html for german OR
/opt/data/your-page/en-US/index.html for english

In each index.html you have <script> tags, linking to your bundles. And these links could absolutely contain the language, because once the user has the correct index.html does it really matter how he loads the bundle?
https://your-page/main.03b15c0004fb5dfc.js VS
https://your-page/en_US/main.03b15c0004fb5dfc.js

Sadly it seems as of now you NEED to use the second variant though. You end up with two main.03b15c0004fb5dfc.js files (same hash, but different content), but since you load them from different URLs you don't have caching issues.

Hope this is helpful

This issue causes problems when we upload source maps to Sentry. We have to create a release for each Language in Sentry. This then breaks resolved in the next release working correctly.

@bartcorremans

Hey, I just rewoved the script-workaround (thank you for that @bartcorremans !) from my homepage build https://budisoft.at and now everything is working fine. What I did was completely removing "baseHref": "/" in my angular.json - so the baseHrefs stay at default. I end up with the following after build:

german index.xml contains base href="https://github.com/de/"
english index.html contains base href="https://github.com/en_US/"

I changed my rewrite-rules to only rewrite for my index.html - all other urls/files can be loaded as-is

Now my bundles (e.g. main.1234567.js) are loaded from different URLs and it doesn't matter any longer that they have the same filename.

One small issue I had with that, was that routing did include the base-href at first. (e.g. routing to /articles would bring me to https://budisoft.at/en_US/articles). After consulting the documentation I added a provider in my app-module to bring the context back to /:
{provide: APP_BASE_HREF, useValue: ''}

With that everything seems to work as expected and I am happy

@alan-agius4

Thanks for explaining and the SO link! I think I understand, but it still feels wrong and there might be some bugs with that implementation - if I build my project while only changing the german translation file the hash would still be unchanged, right? That would lead to people not getting the newest version of my german main.ts file! So basically only changing translations without touching a source file is not supported with the current build?

The hash probably should be dependent on the translation files, as I think it's reasonable to expect these hashes to update on any text content changes. If that's not the case, then that sounds like a separate issue we can look at.

Since all the hashes are currently tied together across locales, that would mean that your French build is actually dependent on all the other translations, which is a bit weird and bad for build caching. It would make more sense that the French build only depends on French translations. @alan-agius4 would there be any build improvements from better isolating languages this way? In theory I think it would be better, but in practice everything is done via a Webpack compilation which probably preprocesses everything for this hash, so I'm not sure this would actually have a meaningful effect?

Thanks to your response I think I finally understand what I'd have to do instead of dirty workaround:

rewrite requests just for the index.html.

keep the baseHrefs like default, e.g. "en_US" and "de"

now my english index.html would request a different file location (e.g. https://budisoft/**en_US**/main.03b15c0004fb5dfc.js) than my german version, even when the hash is the same.

Yes, I think that is the right approach. If you want "language-agnostic" URLs, those should be limited to the URLs actually visible to users (such as HTML pages). JS bundles aren't typically user-visible or stable (hashes change on each deployment), so putting them in a locale-specific URL doesn't negatively impact the user.

This feels like it also addresses @bartcorremans' issue.

This issue causes problems when we upload source maps to Sentry. We have to create a release for each Language in Sentry. This then breaks resolved in the next release working correctly.

I'm not very familiar with Sentry, but it sounds like they're keying sourcemaps by file name? If so, that makes it a subset of the flattening locale resources use case. Given that Sentry has direct Angular support I wonder if that might be better addressed on their end? Seems to me like a sourcemapping utility should be able to handle multiple files with the same name in different directories, but I'm not an expert in that area and I'm sure there's more nuance to it than that.

The question is: is there a good reason NOT to use [contenthash] or maybe add an optional flag for that? This puts a strong opinion/restriction on how files are served. For me it also seems a bit also unintuitive (because of what I believe a file-hash should do)

I agree that this can be unintuitive. Devs have a strong bias to interpret a hash in a file name as a hash of that file's contents. That's not always true (as in this case), but it is often the initial expectation. One other advantage to consider about a [contenthash]-based approach is that files can be cached across deployments for any chunk that didn't change. This can result in some performance improvements, but it's hard to measure the exact impact of that.

I think the main reason not to change the current behavior is that Webpack chunking happens before localization as I understand it. So just changing to [contenthash] doesn't solve the problem because subsequent edits to these files to merge translations would alter the file and change its hash. The Angular CLI then needs to manually update all the files (and their chunk dependencies on each other) to use the new hashes. I think this also doesn't solve the problem of file conflicts, since a JS file without translations will be duplicated across locales, though in that case it is at least the same for each locale. That basically means we'd be implementing our own hashing system on top of Webpack's.

Hey Bart, I think you are a bit in the same "mental trap" as I was. Yes, URLs to a specific content-page should not contain a language url, to ensure deep-linkability across languages. Example: https://your-page/order/123456 <= this is a link to one specific order and should work, regardless of language.

But this is only important for your own pages/routes! This doesn't matter for JavaScript bundles though. Let me explain:

If you give the link above to a german user your reverse-proxy will probably use your cookie (+ fallback) to determine the correct index.html and retrieve it from a file system like this: /opt/data/your-page/de/index.html for german OR /opt/data/your-page/en-US/index.html for english

In each index.html you have <script> tags, linking to your bundles. And these links could absolutely contain the language, because once the user has the correct index.html does it really matter how he loads the bundle? https://your-page/main.03b15c0004fb5dfc.js VS https://your-page/en_US/main.03b15c0004fb5dfc.js

Sadly it seems as of now you NEED to use the second variant though. You end up with two main.03b15c0004fb5dfc.js files (same hash, but different content), but since you load them from different URLs you don't have caching issues.

Hope this is helpful

Hi matrium0,

Thank you for the thoughtful reply! After some tinkering I was able to get our setup to work this way. The main struggle came from angular-router redirecting us to the baseHref, as you mentioned in your later comment - which I wrongly thought was being caused by our server configuration. After figuring that out it was pretty straightforward!

Thank you again! This sees our use case covered without the need for unique file hashes per locale.

Following up with this conversation, really appreciate everyone's feedback here. It looks like we've mostly figured out how to meet these needs with the CLI's current behavior. I think the recommendation for anyone finding this issue in the future:

Hashes are not keyed by the locale. That means if you want to cache the same file across different locales, that locale must appear in the URL.
This only applies to JS files generated by the CLI, not user-visible links like HTML files. If you don't want user-visible links to have locales (like /my_home_page.html), you can automatically infer the user locale via cookies or backend state, and then link to /${locale}/main.abc123.js to load your application. That way user-facing links are locale-independent, but still load localized and cacheable JS sources.

I agree this behavior can be confusing, and in a perfect world asset hashes would probably be locale dependent. However, there is a lot of complexity and nuance to make that work and maintain it over time. Given that this can be worked around, I don't think changing this is the best use of our limited time and energy. So I'll close this issue for now, and we can reevaluate if it becomes a bigger challenge in the future.

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

Navigation Menu

Repository navigation

localized components containing i18n tags will produce identical chunk hashes when building with --localize #17416

🐞 Bug report

Command (mark with an `x`)

Description

🔬 Minimal Reproduction

🌍 Your Environment

33 remaining items

Example

Solution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

localized components containing i18n tags will produce identical chunk hashes when building with --localize #17416

Description

🐞 Bug report

Command (mark with an x)

Description

🔬 Minimal Reproduction

🌍 Your Environment

Activity

alan-agius4 commented on Apr 7, 2020

33 remaining items

dgp1130 commented on May 26, 2022

matrium0 commented on May 26, 2022

dgp1130 commented on May 26, 2022

matrium0 commented on May 27, 2022

Example

Solution

bartcorremans commented on May 27, 2022

matrium0 commented on May 27, 2022

thduttonuk commented on May 31, 2022

matrium0 commented on Jun 1, 2022

dgp1130 commented on Jun 1, 2022

bartcorremans commented on Jun 2, 2022

dgp1130 commented on Jun 16, 2022

angular-automatic-lock-bot commented on Jul 17, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Command (mark with an `x`)