Feat: Add Hybrid Search (Vector + FTS) - #1475

Purpose of the change

This PR introduces Hybrid Search capability to MemMachine, combining vector similarity search with keyword-based Full-Text Search (FTS) to significantly improve retrieval accuracy. The implementation addresses the fundamental limitation of pure vector search where exact keyword matches (error messages, model names, technical terms, identifiers) are often missed due to semantic embedding dilution.

This PR enables clients to opt-in to hybrid retrieval via a simple use_fts flag without changing query patterns.

Description

1. API Extension - Added use_fts: bool parameter across the API stack to enable hybrid search mode. Default false maintains backward compatibility.

2. FTS Index Setup - FTS indexes are auto-created during ingest. The index name follows the pattern fts_{sanitized_derivative_collection}_content and is indexed on the Derivative node's content field. No manual index creation is required; the index is ready for use when hybrid search is enabled.

3. FTS Search Implementation - New _search_fts() method (packages/server/src/memmachine_server/episodic_memory/long_term_memory/long_term_memory.py:363-460) executes Neo4j Full-Text Search queries on Derivative nodes using db.index.fulltext.queryNodes(). It follows the DERIVED_FROM relation to retrieve Episode nodes, returning FTS scores and Episode objects with metadata extracted from Neo4j.

4. Hybrid Search Orchestration - New _search_scored_hybrid() method (packages/server/src/memmachine_server/episodic_memory/long_term_memory/long_term_memory.py:295-349) runs Vector Search and FTS in parallel via asyncio.gather() (no latency overhead). Results are merged with UID-based deduplication: Vector results (top-k, RRF-reranked) are preserved in order, and unique FTS results (top-10) are appended. Returns up to top-k + 10 episodes.

5. Query Escaping Utility - Static method _escape_lucene_query() (packages/server/src/memmachine_server/episodic_memory/long_term_memory/long_term_memory.py:351-361) escapes Lucene special characters (+ - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /) to prevent query syntax errors on user input containing special characters.

Motivation: Vector search excels at semantic similarity but struggles with exact keyword matches (error messages like CUDA_ERROR_OUT_OF_MEMORY, model names like RTX-4090, acronyms like API/HTTP, function/class names). FTS complements vector search by providing precise term matching.

Dependencies: No new external dependencies. Uses existing Neo4j FTS indexes (auto-created on ingest), asyncio, and vector search infrastructure.

Summary of Changes

This PR adds Hybrid Search functionality by integrating Neo4j Full-Text Search (FTS) indexes with existing vector search infrastructure. The key changes include:

Fixes/Closes

N/A (New feature)

Type of change

[Please delete options that are not relevant.]

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (does not change functionality, e.g., code style improvements, linting)
• Documentation update
• Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
• Security (improves security without changing functionality)

How Has This Been Tested?

Test Environment:

• Benchmark: LongMemEval_M (500 questions, 896 gold turns)
• Dataset: longmemeval_m_cleaned.json
• Configuration: Qwen3-Embedding-4B + Qwen3.5-397B-A17B-NVFP4

[Please delete options that are not relevant.]

• Unit Test
• Integration Test
• End-to-end Test
• Test Script (please provide)
• Manual verification (list step-by-step instructions)

Usage Example:

When data is ingested, FTS indexes are automatically created. To enable hybrid search, pass use_fts=True in the QueryParam when calling the search function:

from memmachine_server.retrieval_agent.common.agent_api import QueryParam

# Hybrid search enabled
results = await do_query(
    QueryParam(
        query="CUDA_ERROR_OUT_OF_MEMORY",
        limit=50,
        memory=memory,
        use_fts=True,  # Enable Hybrid Search (Vector + FTS)
    )
)

Test Results: [Attach logs, screenshots, or relevant output]

Key Findings:

• Hybrid Search achieves +2.97% improvement over Vector-only (top-k=50)
• Outperforms Vector top-k=80 (0.9010) while fetching fewer total results
• Single-session-user queries: +7.8% improvement (largest gain)
• Temporal-reasoning queries: +3.9% improvement

Checklist

[Please delete options that are not relevant.]

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A (Backend feature - no UI changes)

Further comments

Known Limitations:

1. Hybrid search currently only supported on declarative backend (Neo4j - VectorGraphStore), not event backend (VectorStore)
2. FTS returns limited metadata (timestamp/producer_id set to defaults)
3. FTS uses BM25 scores while vector uses cosine/euclidean; threshold semantics differ