Fix: BFS deep crawl refetches its start page and stream mode drops the last page - #2433

Two bugs in BFSDeepCrawlStrategy, both present on main (0.9.5) and develop.

1. Batch mode fetches the start page twice

_arun_batch never adds the URLs of the level it is fetching to visited (_arun_stream does). A page that links to itself, such as a home page's logo link, is queued again at the next depth, fetched a second time, and uses up a max_pages slot.

Repro with 0.9.5:

cfg = CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1, max_pages=5))
results = await crawler.arun("https://docs.crawl4ai.com/", config=cfg)
[r.url for r in results]
# ['https://docs.crawl4ai.com/', 'https://docs.crawl4ai.com/', 'https://docs.crawl4ai.com/core/ask-ai/', ...]

Fix: both modes add the level's URLs to visited, normalized with normalize_url_for_deep_crawl like discovered links, so a start URL of https://Example.com also matches a link to https://example.com/.

2. Stream mode drops the page that reaches max_pages

_arun_stream counts the page that reaches max_pages and then breaks before yielding it. With max_pages=5 it fetches 5 pages and returns 4. It now yields the page, then stops (the same order BestFirstCrawlingStrategy already uses).

Tests

Added to tests/deep_crawling/test_bfs_max_pages_next_level.py:

• test_stream_crawl_yields_the_page_that_reaches_max_pages
• test_crawl_does_not_refetch_start_url (batch and stream), on a site where every page links back to the home page

All three fail before the change and pass after. pytest tests/deep_crawling: 90 passed. Live check against docs.crawl4ai.com with max_depth=1, max_pages=5: 5 unique pages in both batch and stream mode (0.9.5 returns the start page twice in batch mode and 4 pages in stream mode).

🤖 Generated with Claude Code