Firehose phase 1: validated profiles hose (kind 0) + extensible hose seam - #8
Merged
Conversation
Introduce a composable 'hose' seam for the firehose indexer and ship the first concrete hose: profiles (kind 0). - src/hoses/profiles.js: declares its kinds, owns its Mongo indexes (pubkey, created_at, and the content_text search index so /api/search works on a fresh deploy), and verifies each event's schnorr signature (NIP-01 id recompute + @noble/curves) before a latest-wins upsert. A relay can no longer inject a forged did:nostr profile. - src/indexer.js: generic hose registry — subscribe to the union of registered hoses' kinds and dispatch by kind. Kind 0 routes through the profiles hose; kinds 3 and 10002 keep the legacy upsert path until their own phases (no behaviour change). - deps: @noble/curves, @noble/hashes (keeps the no-nostr-tools approach). - test/profiles-hose.test.js: sign real kind-0 events and assert accept/reject (valid, empty content, tampered content/id, forged sig, non-hex pubkey, wrong kind, non-JSON content, malformed input). Closes #7
Make the firehose a switch so a deploy can run a single hose without double-writing against the legacy firehose/followshose processes during the phased migration. - planIngest(allHoses, env): pure, testable selector. HOSES env (comma list of hose names; default: all registered) picks active hoses; INDEX_LEGACY_KINDS (default on) gates the raw-upsert fallback for kinds not yet migrated to a hose (3, 10002). Subscription kinds = union of enabled hoses' kinds + legacy kinds, with a hose-owned kind removed from the legacy set so nothing is double-subscribed. - runIndexer: use the plan; warn on unknown hose names / empty kind set. - .env.example: document HOSES + INDEX_LEGACY_KINDS. - test/indexer-plan.test.js: 8 cases (defaults, legacy off, unknown names, single-hose deploy, hose-owned legacy kind removal). Default behaviour unchanged: profiles hose + legacy 3/10002.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7. First step of the phased firehose rework — profiles only, but it lays down the seam later phases reuse.
What
src/hoses/): a hose is a small module{ name, kinds, ensureIndexes(db), ingest(event, db) }.src/indexer.jsis now a generic runner — it subscribes to the union of registered hoses' kinds and dispatches each event to the matching hose. Adding follows / relay-lists / relay-health later = drop in a sibling module.src/hoses/profiles.js— the first concrete hose (kind 0):@noble/curves. Rejects wrong kind, malformed fields, non-JSON content, tamperedid, and forged sigs. A malicious relay can no longer inject a forgeddid:nostrprofile.pubkey,created_at, and thecontent_texttext index — so/api/searchworks on a fresh deploy (today that index only exists because it was created by hand).beacon(unchanged doc shape).indexer.js: kind 0 → profiles hose; kinds 3 and 10002 keep the existingupsertEventpath (migrated in their own phases — no behaviour change).@noble/curves,@noble/hashes(keeps the repo's "no nostr-tools" approach).Out of scope (next phases)
Follows hose (2) · relay-lists hose (3) · active relay-health prober that unfreezes
/relays(4) · deploy + retire the legacy server hoses (5).Verification
npm test— 17/17 pass (8 existing + 9 new profiles-hose tests: valid, empty content, tampered content/id, forged sig, non-hex pubkey, wrong kind, non-JSON content, malformed input).ensureIndexescreatespubkey_1 / created_at_-1 / content_text; latest-wins keeps the newest event; stale + forged events rejected.Note on deployment
The live
beacon-newon nostr.social runs an old copied (non-git)serve.jsfrom before the/relayspage, and the indexing there is still the legacyfirehose/followshoseprocesses — so this hose isn't exercised in prod yet. Wiring the server to a real git checkout + running our indexer is the phase-5 consolidation.