Skip to content

Evaluate the existing BERT/MiniLM embedding model (feature/bert-dsl-embeddings): quality + on-device numbers, and a merge path #273

Description

@michalharakal

Context

feature/bert-dsl-embeddings already carries a substantial embedding stack that
develop doesn't have:

  • BertEmbeddingModel one-call factory (fromSafeTensors + fromHuggingFace)
    in llm-providers, with tests
  • the DSL BERT in llm-inference/bert (legacy eager BERT stack removed on the
    branch)
  • MiniLmExportHarness (jvmTest) — fixed-seq-bucket StableHLO export for the
    on-device embedder cartridge, plus eager reference probes
  • a kbert-cli app in llm-apps

So "do we have a usable embedding model?" is really "how good is the one we
already have, and what does it take to land it?"

Ask — an evaluation pass with recorded numbers

  1. Correctness / parity: cosine similarity of branch embeddings vs the
    reference implementation (sentence-transformers all-MiniLM-L6-v2) over a
    fixed sentence set — eager path and StableHLO-exported path both, per
    fixed-seq bucket.
  2. Retrieval quality: a small STS/retrieval benchmark (e.g. STS-B subset)
    scored against the reference model's numbers, so regressions are visible in
    review rather than discovered by users.
  3. Performance: embeddings/sec + resident memory on JVM, Android and iOS —
    embedding models are the most credible near-term on-device story (they're
    small and latency-tolerant), and the same mobile constraints we documented
    for generative models (no native kernels, ART heap cap) apply here and
    should be quantified.
  4. Merge path: what remains between the branch and develop (the branch also
    removes the legacy eager BERT stack — that's a breaking change worth its own
    review), and whether the export harness belongs in commonTest rather than
    jvmTest so the parity probes run cross-target.

We can run the on-device measurements (physical Android + iPhone) and contribute
the parity/benchmark test harness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions