Skip to content

fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler#1496

Merged
Pijukatel merged 5 commits intoapify:masterfrom
Mantisus:sql-deepcopy
Oct 23, 2025
Merged

fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler#1496
Pijukatel merged 5 commits intoapify:masterfrom
Mantisus:sql-deepcopy

Conversation

@Mantisus
Copy link
Collaborator

@Mantisus Mantisus commented Oct 17, 2025 •
edited
Loading

Description

  • Remove the use of deepcopy for kwargs in AdaptivePlaywrightCrawler for compatibility with SqlStorageClient

Issues

@Mantisus Mantisus self-assigned this Oct 17, 2025
@Mantisus Mantisus requested review from janbuchar and vdusek October 17, 2025 00:24
@vdusek vdusek changed the title fix: Add special deepcopy logic for SqlStorageClient for compatibility with AdaptivePlaywrightCrawler. fix: Resolve compatibility issue between SqlStorageClient and AdaptivePlaywrightCrawler Oct 17, 2025
vdusek
vdusek reviewed Oct 18, 2025
View reviewed changes
@Mantisus Mantisus requested a review from vdusek October 19, 2025 01:46
vdusek
vdusek approved these changes Oct 19, 2025
View reviewed changes
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, lgtm

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 19, 2025
@janbuchar
Copy link
Collaborator

janbuchar commented Oct 22, 2025

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

janbuchar
janbuchar requested changes Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment above

@Mantisus
Copy link
Collaborator Author

Mantisus commented Oct 22, 2025

Huh. Huh. While this does fix the issue, I think the root cause is the unfortunate way deepcopy is used when constructing the subcrawlers in adaptive crawler. Could we fix that instead?

Updated.

@Pijukatel could you take a look at this as well? You are the most familiar with the AdaptivePlaywrightCrawler. I just want to make sure I'm not missing any important cases where deepcopy was used.

janbuchar
janbuchar reviewed Oct 22, 2025
View reviewed changes
@Mantisus Mantisus requested a review from janbuchar October 22, 2025 15:54
janbuchar
janbuchar approved these changes Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@janbuchar
Copy link
Collaborator

janbuchar commented Oct 22, 2025

Perhaps we could add a test that shows that adaptive crawler can be instantiated with sql storage client, that wouldn't be too hard 🙂

@Pijukatel Pijukatel merged commit ce172c4 into apify:master Oct 23, 2025
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek approved these changes

@janbuchar janbuchar janbuchar approved these changes

@Pijukatel Pijukatel Pijukatel approved these changes

Assignees

@Mantisus Mantisus

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Crawler's storage_client errors when using SqlStorageClient

4 participants

@Mantisus @janbuchar @vdusek @Pijukatel