feat(components): add MrScraper integration - #14516
Open
ai-mrscraper wants to merge 15 commits into
Open
Conversation
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughAdded optional MrScraper SDK support, eight LFX components, component-index registrations, frontend icon and bundle mappings, and unit tests for SDK calls, validation, defaults, and missing-dependency errors. ChangesMrScraper package and registry
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant LangflowComponent
participant MrScraperSDK
participant Data
LangflowComponent->>MrScraperSDK: Invoke scraping, crawling, retrieval, or rerun API
MrScraperSDK-->>LangflowComponent: Return SDK response
LangflowComponent->>Data: Wrap response as Data
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (7 passed)
✨ Finishing Touches 💡 2⚔️ Resolve merge conflicts 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/backend/tests/unit/components/bundles/mrscraper/test_mrscraper_components.py (1)
22-350: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy liftUse the required component test base and fixtures.
These component test classes do not inherit
ComponentTestBaseWithoutClient. They also do not define the requiredcomponent_class,default_kwargs, andfile_names_mappingfixtures.Use
ComponentTestBaseWithoutClientbecause these tests use mocked SDK clients and do not require API access. As per coding guidelines, “Component tests must use eitherComponentTestBaseWithClientfor components needing API access orComponentTestBaseWithoutClientfor pure logic components, and must includecomponent_class,default_kwargs, andfile_names_mappingfixtures.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/backend/tests/unit/components/bundles/mrscraper/test_mrscraper_components.py` around lines 22 - 350, Update the Mrscraper component test classes to inherit from ComponentTestBaseWithoutClient and add the required component_class, default_kwargs, and file_names_mapping fixtures for each component. Preserve the existing mocked SDK behavior and parameterized missing-SDK coverage while conforming to the shared component test base contract.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@src/backend/tests/unit/components/bundles/mrscraper/test_mrscraper_components.py`:
- Around line 22-350: Update the Mrscraper component test classes to inherit
from ComponentTestBaseWithoutClient and add the required component_class,
default_kwargs, and file_names_mapping fixtures for each component. Preserve the
existing mocked SDK behavior and parameterized missing-SDK coverage while
conforming to the shared component test base contract.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 351a81a9-1a13-425f-9641-faea24767b43
⛔ Files ignored due to path filters (2)
src/frontend/src/icons/Mrscraper/mrscraper.svgis excluded by!**/*.svguv.lockis excluded by!**/*.lock
📒 Files selected for processing (23)
src/backend/base/pyproject.tomlsrc/backend/tests/unit/components/bundles/mrscraper/__init__.pysrc/backend/tests/unit/components/bundles/mrscraper/conftest.pysrc/backend/tests/unit/components/bundles/mrscraper/test_mrscraper_components.pysrc/frontend/src/icons/Mrscraper/MrscraperIcon.jsxsrc/frontend/src/icons/Mrscraper/index.tsxsrc/frontend/src/icons/eagerIconImports.tssrc/frontend/src/icons/lazyIconImports.tssrc/frontend/src/utils/styleUtils.tssrc/lfx/src/lfx/_assets/component_index.jsonsrc/lfx/src/lfx/components/__init__.pysrc/lfx/src/lfx/components/mrscraper/__init__.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_ai_scraper.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_batch_scrape.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_crawl_website.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_fetch_html.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_get_result.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_get_results.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_run_ai_scraper.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_run_manual_scraper.pysrc/lfx/tests/unit/components/mrscraper/__init__.pysrc/lfx/tests/unit/components/mrscraper/conftest.pysrc/lfx/tests/unit/components/mrscraper/test_mrscraper_components.py
ai-mrscraper
changed the title
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds MrScraper as an official
lfx-bundlesprovider with eight components for AI scraping, website crawling, rendered HTML fetching, batch reruns, and result retrieval.Components
Implementation
src/bundles/lfx-bundles/src/lfx_bundles/mrscraper/src/bundles/lfx-bundles/tests/test_mrscraper/lfx-bundles[mrscraper]usingmrscraper-sdk>=0.1.2,<1.0.0DataValidation
lfx-bundlestests passLinks