feat(adaptive): calculate lexical statistics, saturation, and metrics in EmbeddingStrategy - #2438

Description

In AdaptiveCrawler with strategy='embedding', lexical and term statistics (unique terms, total terms, vocabulary richness, crawl efficiency) and saturation were not calculated or populated in CrawlState, resulting in 0 or missing values in print_stats() and coverage_stats. Furthermore, learning_score was omitted from state.metrics in calculate_confidence(), causing get_quality_confidence() to default to 0.00%.

This PR brings feature parity between EmbeddingStrategy and StatisticalStrategy regarding statistics tracking and reporting, without modifying StatisticalStrategy.

Changes

• EmbeddingStrategy._update_text_statistics(): Tracks term_frequencies, document_frequencies, documents_with_terms, total_documents, new_terms_history, and crawl_order on crawled documents.
• EmbeddingStrategy.calculate_confidence():
  • Sets state.metrics['learning_score'] and state.metrics['coverage'].
  • Calculates and stores state.metrics['saturation'] and state.metrics['consistency'].
  • Vectorially computes distance metrics: avg_min_distance, avg_close_neighbors (<0.3 distance), avg_very_close_neighbors (<0.2 distance), and total_kb_docs.
• AdaptiveCrawler.print_stats(): Added "Saturation" row to the summary table for EmbeddingStrategy.
• Tests: Added tests/adaptive/test_embedding_statistics.py verifying all statistics, confidence scoring, distance metrics, and table formatting.

Checklist

• Based on and targeting develop
• Tests pass (pytest tests/adaptive/test_embedding_statistics.py)
• No breaking changes
• Follows code style guidelines

Comparison

Using Embedding Strategy (old verson)

Using Embedding Strategy (new verson with statistics, same arguments)

Using Statistical Strategy for reference (same confidence_threshold and min_gain_threshold)