Skip to content

fix(nodes): stop FetchNodeLevelK from skipping the page after an empty one - #1177

Open
RaphaelFakhri wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
RaphaelFakhri:fix/fetch-level-k-skipped-page
Open

RaphaelFakhri wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
RaphaelFakhri:fix/fetch-level-k-skipped-page

Conversation

@RaphaelFakhri

Copy link
Copy Markdown

Summary

FetchNodeLevelK.obtain_content removes documents with empty content from documents while it iterates over that same list. Removing an item during iteration shifts the list, so the loop skips the next document. On the last depth level, the skipped page is never fetched and DepthSearchGraph returns without it, with no warning.

This change iterates over a copy of the list, so removing a failed document no longer affects the loop.

Fixes #1176

Changes

  • scrapegraphai/nodes/fetch_node_level_k.py: iterate over list(documents) in obtain_content.
  • tests/test_fetch_node_level_k.py: new test. With depth: 2, the root page links to an empty page and then to a page with content. The test checks that the second page is fetched and returned.

Testing

  • uv run pytest tests/test_fetch_node_level_k.py: fails before the fix (/full is never fetched), passes after.
  • The unit suites from .github/workflows/test-suite.yml plus the new file: 82 passed.
  • ruff check, black --check and isort --check-only pass on the changed files.

The new test file is not in the explicit file list in .github/workflows/test-suite.yml. You can add it there if you want CI to run it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant