Skip to content

fix: match complete URL pattern suffixes - #2413

Open
ningyixi wants to merge 1 commit into
unclecode:developfrom
ningyixi:fix/complete-url-pattern-suffix
Open

ningyixi wants to merge 1 commit into
unclecode:developfrom
ningyixi:fix/complete-url-pattern-suffix

Conversation

@ningyixi

Copy link
Copy Markdown

Summary

URLPatternFilter("*.tar.gz") rejects https://example.com/archive.tar.gz, while URLPatternFilter("*.html") accepts https://example.com/html. The suffix fast path keeps only the final extension segment of the URL and removes the pattern's literal dot.

Keep the dot in the stored suffix and match the complete URL path suffix. Query strings and fragments remain excluded, and reverse filters and lists of suffix patterns retain their behavior.

List of files changed and why

  • crawl4ai/deep_crawling/filters.py: preserve the literal suffix and use str.endswith for matching (two runtime lines changed).
  • tests/unit/test_url_pattern_suffix.py: cover multi-part extensions, extensionless filenames, reverse filters, query/fragment handling, case sensitivity, and combined suffix patterns.

How Has This Been Tested?

Python 3.12.14, Linux; editable install with pytest, pytest-asyncio, Black, the PDF extra, and Playwright Chromium installed locally.

  • Before the fix, the new regression module has 7 failed, 14 passed. After the fix, all 21 pass.
  • Regression, domain-filter, and configuration-provenance tests: 54 passed.
  • python -m pytest tests/unit -q -k 'not test_seeder_client_has_no_proxy_by_default and not test_cache_validator_client_has_no_proxy_by_default': 224 passed, 6 skipped, 2 deselected. This includes the existing local HTTPS/browser tests. The two excluded tests assume no ambient HTTP proxy; both also fail on unchanged base 2efcccc in this proxy-configured environment. One existing Pydantic serialization warning remains.
  • Black check on the new test module and git diff --check pass.

The full repository suite and external-service/LLM integrations were not run. No model API requests were made.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas (the existing suffix-path comment still applies)
  • I have made corresponding changes to the documentation (no API or setup changes; restores existing wildcard semantics)
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes (bounded suite and baseline environment exceptions detailed above)

AI assistance: this change and its tests were prepared with Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant