Skip to content

fix: zero out declarative cursor on concurrent streams#723

Merged
maxi297 merged 1 commit intomainfrom
maxi297/fix-cursor-datetime-format-issue
Aug 25, 2025
Merged

fix: zero out declarative cursor on concurrent streams#723
maxi297 merged 1 commit intomainfrom
maxi297/fix-cursor-datetime-format-issue

Conversation

@maxi297
Copy link
Contributor

@maxi297 maxi297 commented Aug 24, 2025

What

The v6.60.16 of the CDK brought back by accident some of the processing of the declarative cursor which is less resilient than the Concurrent CDK one. In order to fix this, we need to add this logic as part of the ModelToComponentFactory.

For more information, see https://airbytehq-team.slack.com/archives/C09BWEEGWTE

Summary by CodeRabbit

  • Bug Fixes
    • Improved stability for declarative sources using simple retrievers by resolving a state/partition interaction that could cause setup issues.
    • Prevents incorrect partition generation in certain configurations, reducing the likelihood of sync failures.
    • Enhances reliability of incremental syncs by ensuring state is handled consistently during stream initialization.

@github-actions github-actions bot added bug Something isn't working security labels Aug 24, 2025
@github-actions
Copy link

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@maxi297/fix-cursor-datetime-format-issue#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch maxi297/fix-cursor-datetime-format-issue

Helpful Resources

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment

📝 Edit this welcome message.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Aug 24, 2025

📝 Walkthrough

Walkthrough

Implements a temporary workaround in create_declarative_stream: if the retriever is a SimpleRetriever, its cursor is cleared (set to None) before building the partition generator. Other flow and public interfaces remain unchanged.

Changes

Cohort / File(s) Summary
Declarative stream factory
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
Adds a guarded workaround to set retriever.cursor = None when retriever is SimpleRetriever prior to partition generator creation; includes FIXME for later removal; no public API changes.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant C as create_declarative_stream
  participant R as SimpleRetriever
  participant P as PartitionGenerator

  C->>R: Instantiate retriever
  alt retriever is SimpleRetriever
    Note over C,R: TEMP workaround (FIXME)
    C->>R: Set cursor = None
  end
  C->>P: Construct partition generator (uses retriever)
  Note over C,P: Remaining flow unchanged
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested labels

bug

Suggested reviewers

  • brianjlai
  • darynaishchenko
  • tolik0

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch maxi297/fix-cursor-datetime-format-issue

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (3)

2093-2098: Scope the zero-out to concurrent mode and add a debug trace for observability, wdyt?

The intent is clear and aligns with restoring the Concurrent CDK behavior. To make this safer and self-documenting, could we:

  • Only zero out when a concurrent cursor is actually being used; and
  • Emit a debug log so operators understand why SimpleRetriever.cursor is being nulled?
-            # FIXME to be removed once we migrate everything to DefaultStream
-            if isinstance(retriever, SimpleRetriever):
-                # We zero it out here, but since this is a cursor reference, the state is still properly
-                # instantiated for the other components that reference it
-                retriever.cursor = None
+            # FIXME to be removed once we migrate everything to DefaultStream
+            # When running in concurrent mode, the concurrent cursor owns state progression.
+            # Ensure the SimpleRetriever does not attempt to process a declarative cursor.
+            if isinstance(retriever, SimpleRetriever) and isinstance(concurrent_cursor, ConcurrentCursor):
+                logging.getLogger(f"airbyte.{stream_name}").debug(
+                    "Zeroing SimpleRetriever.cursor; ConcurrentCursor manages state for stream '%s'.",
+                    stream_name,
+                )
+                # We zero it out here, but since this is a cursor reference, the state instance remains
+                # available to other components that have their own references.
+                retriever.cursor = None

2093-2098: Add a regression test to lock in the behavior, wdyt?

A small test would catch future regressions where SimpleRetriever starts observing/closing slices in concurrent flows. For example:

  • Build a declarative stream with a DatetimeBasedCursor that resolves to a ConcurrentCursor.
  • Assert that in the produced DefaultStream, the underlying SimpleRetriever has cursor is None while DefaultStream.cursor is a ConcurrentCursor.

If helpful, I can draft a test skeleton targeting create_declarative_stream and asserting the invariants above. Want me to push that?


1956-1961: Double-check: partition_field_start set from partition_field_end looks unintended, wdyt?

This block configures DatetimeBasedRequestOptionsProvider for DatetimeBasedCursorModel, but both partition fields are sourced from cursor_model.partition_field_end. Should partition_field_start come from cursor_model.partition_field_start instead?

             request_options_provider = DatetimeBasedRequestOptionsProvider(
                 start_time_option=start_time_option,
                 end_time_option=end_time_option,
-                partition_field_start=cursor_model.partition_field_end,
-                partition_field_end=cursor_model.partition_field_end,
+                partition_field_start=cursor_model.partition_field_start,
+                partition_field_end=cursor_model.partition_field_end,
                 config=config,
                 parameters=model.parameters or {},
             )

If this is intentional (e.g., a specific API quirk), could we add a short comment to prevent future “fixes”?

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 2b07f93 and da7d22c.

📒 Files selected for processing (1)
  • airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1)
airbyte_cdk/sources/declarative/retrievers/simple_retriever.py (1)
  • SimpleRetriever (53-602)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
  • GitHub Check: Check: source-pokeapi
  • GitHub Check: Check: source-intercom
  • GitHub Check: Check: destination-motherduck
  • GitHub Check: Check: source-shopify
  • GitHub Check: Check: source-hardcoded-records
  • GitHub Check: Pytest (Fast)
  • GitHub Check: Pytest (All, Python 3.10, Ubuntu)
  • GitHub Check: Pytest (All, Python 3.11, Ubuntu)
  • GitHub Check: SDM Docker Image Build

@github-actions
Copy link

PyTest Results (Fast)

3 692 tests  ±0   3 681 ✅ ±0   7m 5s ⏱️ +44s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit da7d22c. ± Comparison against base commit 2b07f93.

@github-actions
Copy link

PyTest Results (Full)

3 695 tests  ±0   3 684 ✅ ±0   9m 22s ⏱️ -17s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit da7d22c. ± Comparison against base commit 2b07f93.

@maxi297 maxi297 merged commit 48c6836 into main Aug 25, 2025
27 checks passed
@maxi297 maxi297 deleted the maxi297/fix-cursor-datetime-format-issue branch August 25, 2025 17:21
@coderabbitai coderabbitai bot mentioned this pull request Oct 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants