Skip to content

lmdb: fulltext query engine#1202

Open
v0l wants to merge 2 commits intorust-nostr:masterfrom
v0l:search-query
Open

lmdb: fulltext query engine#1202
v0l wants to merge 2 commits intorust-nostr:masterfrom
v0l:search-query

Conversation

@v0l
Copy link
Contributor

@v0l v0l commented Jan 2, 2026

Description

  • Implement full-text search query parsing using tantivy-query-grammar
  • Support AND/NOT operators and phrase queries
  • Add field-based tag searching (e.g., title:bitcoin)
  • Search all multi-character tags by default (instead of whitelist)

Details

Adds a proper query parser for the search filter field in nostr-lmdb, replacing the previous simple substring matching with a full query language:

Query Syntax:

  • bitcoin nostr - AND: both terms must match (word-boundary matching)
  • +bitcoin -scam - explicit Must/MustNot operators
  • "hello world" - phrase query (exact match)
  • title:introduction - field query (searches specific tag)

Matching Behavior:

  • Partial word matching with simple contains(phrase)
  • Case-insensitive
  • Searches event content and all tags with keys > 1 character (skips single-letter tags like e, p, t which are typically references)
  • Field queries can target any tag, including single-letter tags

Implementation:

  • New query.rs module with match_query() function
  • Uses tantivy-query-grammar for parsing (same syntax as Tantivy search engine)
  • Evaluates AST recursively with proper boolean logic

Checklist

v0l added 2 commits January 2, 2026 22:21
lmdb: match literals partially with simple contains(phrase)


/// Match a query against an event
#[cfg(test)]
Copy link
Member

@TheAwiteb TheAwiteb Jan 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is #[cfg(test)] here?

Comment on lines -171 to -249
#[cfg(test)]
mod tests {
use nostr::{Event, EventBuilder, Keys, Tag};

use super::*;

fn create_test_event(content: &str) -> Event {
let keys = Keys::generate();
EventBuilder::text_note(content)
.sign_with_keys(&keys)
.unwrap()
}

#[test]
fn test_search_match_in_content() {
let event = create_test_event("Hello World");
let event: EventBorrow = (&event).into();

let mut filter = DatabaseFilter::from(Filter::new());

// Case insensitive match
filter.search = Some("hello".to_string());
assert!(filter.match_event(&event));

filter.search = Some("world".to_string());
assert!(filter.match_event(&event));

// No match
filter.search = Some("rust".to_string());
assert!(!filter.match_event(&event));
}

#[test]
fn test_search_match_in_tags() {
let keys = Keys::generate();
let event = EventBuilder::text_note("content")
.tag(Tag::parse(["title", "Search userfacing tags"]).unwrap())
.sign_with_keys(&keys)
.unwrap();
let event: EventBorrow = (&event).into();

let mut filter = DatabaseFilter::from(Filter::new());

filter.search = Some("userfacing".to_string());
assert!(filter.match_event(&event));

filter.search = Some("bitcoin".to_string());
assert!(!filter.match_event(&event));
}

#[test]
fn test_search_empty_query() {
let event = create_test_event("test");
let event: EventBorrow = (&event).into();
let mut filter = DatabaseFilter::from(Filter::new());

filter.search = Some("".to_string());
assert!(!filter.match_event(&event));
}

#[test]
fn test_search_no_query() {
let event = create_test_event("test");
let event: EventBorrow = (&event).into();
let filter = DatabaseFilter::from(Filter::new());

assert!(filter.match_event(&event));
}

#[test]
fn test_search_partial_match() {
let event = create_test_event("nostr protocol");
let event: EventBorrow = (&event).into();
let mut filter = DatabaseFilter::from(Filter::new());

filter.search = Some("proto".to_string());
assert!(filter.match_event(&event));
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't remove old tests

Comment on lines +270 to +286
// Match specific tag by field name
assert!(match_query("title:introduction", &event));

// Match 't' tag
assert!(match_query("t:bitcoin", &event));

// Field name is case-insensitive
assert!(match_query("TITLE:nostr", &event));

// Non-matching field
assert!(!match_query("title:ethereum", &event));

// Field that doesn't exist
assert!(!match_query("author:someone", &event));

// Match summary tag (not in allowed tags, but accessible via field syntax)
assert!(match_query("summary:guide", &event));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a thing in NIP-50.

A query string may contain key:value pairs (two words separated by colon), these are extensions, relays SHOULD ignore extensions they don't support.

key:value should be an extension, not a tag search

Comment on lines +223 to +230
// Must have bitcoin, must not have ethereum
assert!(match_query("+bitcoin -ethereum", &event));

// Must not have bitcoin - should not match
assert!(!match_query("-bitcoin", &event));

// Must have bitcoin, must not have decentralized - should not match
assert!(!match_query("+bitcoin -decentralized", &event));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

search field is a string describing a query in a human-readable form, i.e. "best nostr apps". Relays SHOULD interpret the query to the best of their ability and return events that match it.

- and + is not a human-readable form, right? Also they are not in the NIP-50

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the - and + are defined in the tantivy_query_grammar package as a way of including and excluding results for a search term. However, for anybody who will use the API to perform a search, this is not obvious to them and should not be.
But since the NIP does state:

Clients may include kinds, ids and other filter field to restrict the search results to particular event kinds.

is it viable to add the keys: Include, exclude to DatabaseFilter where clients can pass their preferences. This would then need to be translated to - and +

@v0l
Copy link
Contributor Author

v0l commented Jan 5, 2026

nostr-protocol/nips#2182

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants