Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: apify/crawlee-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: apify/crawlee-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: claude/crawlee-python-pr-1999-bthig3
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 12 files changed
  • 3 contributors

Commits on Jun 30, 2026

  1. feat: provide Request instances in skipped request callbacks

    @vdusek
    vdusek committed Jun 30, 2026
    Configuration menu
    Copy the full SHA
    3bfba40 View commit details
    Browse the repository at this point in the history

  2. docs: remove stray blank line in robots.txt skipped request example

    @vdusek
    vdusek committed Jun 30, 2026
    Configuration menu
    Copy the full SHA
    af58657 View commit details
    Browse the repository at this point in the history

  3. test: add failing tests reproducing PR #1999 review findings 

    These tests encode concrete user scenarios that the PR mishandles. They are
expected to fail against the PR as written and document the intended behavior.

Finding #1 - annotation-driven dispatch silently degrades to `str`:
- A hook annotated `Request` in a module using `from __future__ import
  annotations` + a TYPE_CHECKING import (the idiomatic style Crawlee itself
  uses) makes `get_type_hints` raise NameError, so the blanket
  `except Exception: return False` silently hands the user a `str`.
- `Request | None` is not matched by the exact-identity `is Request` check.

Finding #2 - `transform_request_function` is now applied to robots.txt-skipped
links, so a transform returning 'skip' removes a robots-blocked URL from the
`on_skipped_request` callback entirely.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01God28J38pkTpE81Tkij6op
    @Pijukatel @claude
    Pijukatel and claude committed Jun 30, 2026
    Configuration menu
    Copy the full SHA
    faaa827 View commit details
    Browse the repository at this point in the history
Loading