Repository navigation
storage: ranked NIP-50 search over LMDB — inverted index, ts_rank-equivalent ordering, and a rebuild path #1110
Description
NIP-50 search over the LMDB store currently refuses (NotImplemented { module: "storage::lmdb::nip50_search" }, crates/nostream-storage/src/lmdb/events.rs:1028-1048). #1109 makes it answer, with set parity only: a scan applying the token rule and returning the fixture's rows, in an implementation-defined order.
This is the other half, and it is not a port — the LMDB store has none of what it needs:
An inverted index maintained on write. Postgres has a GIN index on to_tsvector('simple', event_content) and this port already inlines the configuration so the indexed expression is the one used (postgres/sql.rs:443-452, and the capture misses its own index by binding ?::regconfig). The LMDB backend has no text index at all.
Frequency ranking that reproduces ts_rank. The fixture's expected order for the search rows is proximity-based, not frequency-based: the four rows in script entry 21 tie on term frequency, which is why #1109 cannot reach it. Ranking is therefore a design decision to be recorded, not an inherited contract — either an ADR specifying LMDB's own ranking and a fixture re-recorded against it, or a deliberate divergence documented where clients read it.
A rebuild path the store does not have. The index must be derivable from the events already on disk, and nothing today can do that: Databases::create fills an empty database on open, and there is a per-value ROW_FORMAT_VERSION, but no migrate, reindex or backfill command exists anywhere in the crate. An index added after events are stored is a latent bug without one, so the rebuild is part of this issue, not a follow-up to it.
Acceptance: a ranked query returns the fixture's order, a store written before the index existed can be rebuilt in one command whose output is comparable to a freshly built store, and the rebuild is exercised by a test rather than described.
Relation: blocks nothing in #1109; it is the ranked-search half of the storage follow-up whose other slices are #864's migration tool and benchmark rig.
Filed from job 028's recon, which found both the missing index and the absent rebuild path while scoping the scan-time version.
Activity
chappie-daemon commentedon Nov 11, 2026
Superseded in substance by #1109, which now builds the index, the marker and the open-time rebuild in job 028. I had scoped that job to a scan-time filter and filed this issue so the index work was not lost; 028's spec came back validated over two adversarial passes, so the work moved back rather than being done twice in the same files.
What still belongs here when #1109 merges: the ranked half. 028 will reproduce the fixture's set and rank by (span, id) where Postgres keeps
created_at DESCon a limited search; making the two orderings agree, and deciding whether LMDB's own ranking should be recorded as an ADR instead, is not in 028's criteria. The rebuild and the index are 028's.Kept open for that, and for the residual 028 names itself: an index that also carries tag keys answers every criterion correctly and costs only space.
chappie-daemon commentedon Nov 11, 2026
A measurement that decides the index's key shape, from the scan-time attempt: the two candidate rules do not agree on the fixture, so a term index cannot simply index substrings.
On the fixture corpus (483 entries,
search: "bitcoin lightning"):f8ef979358ca…whose content containsbitcoininveniiceand tag#LightningNetworkcontentscript[21]expects:5454369c…,9ae1fef4…,cb41289d…,c0040000…So the recorded set is only reachable by tokenising, and two consequences follow for this issue:
‖event_idgives the right set; a substring index would need every n-gram and still match the fifth row.contentonly, never tags. The fifth row matches on a tag, and the expected four do not come from tags; that is the boundary the fixture draws, and the index must draw the same one or the set diverges in the other direction.Also worth carrying: the substring rule is what the capture's in-memory matcher does, so Postgres's two paths disagree with each other on this corpus. Ranked parity later has to decide which one it is being compared against, and the fixture only recorded the SQL one.
chappie-daemon commentedon Nov 11, 2026
The index design is not lost — it is archived, with its spec and the branch that implements it.
Job 028 was scoped to a scan-time filter, so the index build it had already produced was set aside rather than deleted:
factory/028b-nip50-indexat8916b38incameri/nostream— seven paths, +1956/−121, gates green (1266 passed / 0 failed / 4 ignored), red half real (op 21 … nip50_searchat base,replayed=114 excluded=48,50,112,115at head), ADR 0011 D11–D17 andnip-parity.mdin the commit. Not merged, not on 028's branch.docs/factory/028b-nip50-index/in the private docs repo — the 22-criterion spec (digestc26b9d62), both adversarial rounds, recon, and the build report.What the spec's adversarial passes bought, which is why it is worth keeping: the first round named six implementations that pass undetected — stale-key index never cleaned on delete, lazy build-on-first-search, marker-written-first rebuild, verdict fed truncated terms, unconditional rebuild, tf+id ranking — and the second named three more, including a
page-only router thatcount's limited path bypasses atevents.rs:803, which would have shipped a count that disagreed with its own page.Two corrections this issue should carry from the scan-time job:
(min cover span, event_id)reproduces entry 21's order and set — spans 2/2/2/4, ids0x54 < 0xc0 < 0xcb. So this issue's ranking question changes from "decide a ranking" to "implement one known to fit four rows and prove it beyond them", which is a smaller and sharper task.contentper candidate on every search. That is the argument to make in the ADR, and it is a better one than parity.Also carried:
EXCLUDED_MARKERfeeds only theprintln!, so un-excluding entry 21 is two assertions plus a printed line; a stale marker leaves the test green and only the printed output catches it.chappie-daemon commentedon Nov 11, 2026
Scope narrowed once more: 028 builds the index after all, at
8916b38, with ADR 0012 and the(min cover span, event_id)ranking. So the index, the marker and the rebuild are not this issue's — they ship in the job.What stays here, and is now precise:
(min cover span, event_id)was fitted to entry 21's four expected rows (spans 2/2/2/4, ids0x54 < 0xc0 < 0xcb). Fitting four rows is a hypothesis, not a specification. This issue is where that hypothesis gets a corpus large enough to falsify it, and where the answer is either "recorded as LMDB's rank" or "replaced".EXCLUDED_MARKERfeeds only theprintln!, so un-excluding entry 21 is two assertions plus a printed line — a stale marker leaves the test green and only the printed output catches it.Two builds exist for this area and both are green: the index at
8916b38(shipping) and the scan-time filter at6b50424onfactory/028a-nip50-scan(preserved). If the index is later reverted, the scan is a working fallback rather than a rewrite.