Skip to content

feat: Dynamic memory snapshots #1715

Draft
Pijukatel wants to merge 1 commit intomasterfrom
dynamic-snapshotter
Draft

feat: Dynamic memory snapshots #1715
Pijukatel wants to merge 1 commit intomasterfrom
dynamic-snapshotter

Conversation

@Pijukatel
Copy link
Collaborator

@Pijukatel Pijukatel commented Feb 5, 2026 •
edited
Loading

Description

  • Add Ratio type to represent the maximum relative available memory of the system.
  • Allow to initialize the Snapshotter.max_memory_size and MemorySnapshot.max_memory_size with either Ratio (dynamic memory) or ByteSize (fixed memory)
  • When Ratio is used, the MemorySnapshot.is_overloaded will take into account the current available memory. (Previously, it would take into account only the initial available memory.)

Top level usage in Crawlers:
Fixed memory

BasicCrawler(configuration=Configuration(memory_mbytes=1024))

Dynamic memory

BasicCrawler(configuration=Configuration(available_memory_ratio=0.5))

Issues

Testing

  • Unit test

Checklist

  • CI passed

@github-actions github-actions bot added this to the 133rd sprint - Tooling team milestone Feb 5, 2026
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Feb 5, 2026
@codecov
Copy link

codecov bot commented Feb 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.52%. Comparing base (bf55e3d) to head (23c803d).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1715      +/-   ##
==========================================
+ Coverage   92.46%   92.52%   +0.05%     
==========================================
  Files         156      156              
  Lines       10460    10474      +14     
==========================================
+ Hits         9672     9691      +19     
+ Misses        788      783       -5
Flag Coverage Δ
unit 92.52% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Pijukatel Pijukatel mentioned this pull request Feb 5, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Pijukatel Pijukatel

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Milestone

133rd sprint - Tooling team

Development

Successfully merging this pull request may close these issues.

Snapshotter does not account for dynamic memory scaling (e.g., K8s burstable QoS)

1 participant

@Pijukatel