Repository navigation
fix: honour preserve_https_for_internal_links for prefetch and <base href> - #2436
Open
workflowsdiy wants to merge 3 commits into
Open
workflowsdiy wants to merge 3 commits into
workflowsdiy wants to merge 3 commits into
Conversation
…inks Prefetch returns before the scraping strategy runs, and quick_extract_links normalized links without the HTTPS-preservation arguments. So with preserve_https_for_internal_links=True, the same page gave different internal links in prefetch and in the normal path. Pass the option and the requested scheme through, as the scraping strategy already does. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…e href> The normal scraping path replaced the page URL with <base href> before normalizing links, so with preserve_https_for_internal_links=True a <base href="http://other.com/"> on an https page turned its links into https://other.com/... Resolve against the base, but check same-host against the page URL, as the prefetch path now does. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
A <base href> spelling the page's host in capitals (http://EXAMPLE.COM/) no longer counted as same-host, so its links were not upgraded to https. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
preserve_https_for_internal_links=True, the same HTTPS page produced different links in prefetch and normal mode, and<base href>handling in normal mode upgraded links on other hosts.Examples, for page
https://example.com/:<a href="http://example.com/docs">: normal mode gavehttps://example.com/docs, prefetch gavehttp://example.com/docs. Prefetch returns before the scraping strategy runs, andquick_extract_linksnever received the option.<base href="http://other.com/">with<a href="chapter">: normal mode gavehttps://other.com/chapter. The same-host check compared against the<base>host, so a link to another host was rewritten.<base href="http://EXAMPLE.COM/">with<a href="chapter">: a case-sensitive host comparison stopped a same-host link being upgraded.Fix:
quick_extract_linksacceptspreserve_httpsandoriginal_scheme, andAsyncWebCrawlerpasses them from the config and the requested scheme.quick_extract_linksandcontent_scraping_strategy, links resolve against<base href>, but the same-host check uses the page URL.normalize_urlandnormalize_url_for_deep_crawlcompare hosts case-insensitively.Tests:
tests/test_prefetch_mode.pycovers prefetch and normal mode for same-host upgrade, option off, cross-host<base>, mixed-case<base>, and an https-to-http redirect. Existing prefetch, deep-crawl and scraping-strategy tests pass. Six failures intest_normalize_url.pyandtest_link_extractor.pyalso occur ondevelopand are unrelated.🤖 Generated with Claude Code