Skip to content

Improve ML model loading#22073

Open
stelfrag wants to merge 5 commits intonetdata:masterfrom
stelfrag:fix_ml_model_load
Open

Improve ML model loading#22073
stelfrag wants to merge 5 commits intonetdata:masterfrom
stelfrag:fix_ml_model_load

Conversation

@stelfrag
Copy link
Copy Markdown
Collaborator

@stelfrag stelfrag commented Mar 27, 2026

Summary
  • Load the most recent models (up to the configured required models)

Summary by cubic

Load only the most recent ML models up to the configured limit to speed up initialization and reduce DB work.

  • Refactors
    • Rewrote db_models_load to select explicit columns, fetch the N newest via subquery (ORDER BY after DESC LIMIT @n), then return them sorted by after ASC.
    • Bind Cfg.num_models_to_use as @n and tidy ml_dimension_stream_kmeans indentation.

Written for commit 17c8239. Summary will update on new commits.

- Refactor `db_models_load` to retrieve the most recent models first with a limit and sort them ascending by `after`.
@stelfrag stelfrag requested a review from thiagoftsm March 27, 2026 22:39
@github-actions github-actions bot added the area/ml Machine Learning Related Issues label Mar 27, 2026
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

@stelfrag stelfrag marked this pull request as ready for review March 27, 2026 22:46
@stelfrag stelfrag requested a review from vkalintiris as a code owner March 27, 2026 22:46
Copilot AI review requested due to automatic review settings March 27, 2026 22:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates ML model initialization to load only a limited number of recent models from the SQLite models table, aiming to reduce startup DB work while keeping models in chronological order for processing.

Changes:

  • Refactors the db_models_load SQL to select the latest N models (via a DESC + LIMIT subquery) and then order them ASC for processing.
  • Binds Cfg.num_models_to_use into the load query as @n.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…estamp for improved model ordering consistency.
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/ml/ml.cc">

<violation number="1" location="src/ml/ml.cc:176">
P2: Ordering by `before` in the inner top-N query is not supported by the existing index and can force a full sort before `LIMIT`, causing unnecessary DB overhead.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

@thiagoftsm thiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After changes, PR is working as expected. LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ml Machine Learning Related Issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants