Skip to content

Add incremental search index and boolean/phrase query support#16

Open
Intina47 wants to merge 1 commit intomainfrom
mamba/add-local-full-text-indexing-for-notes
Open

Add incremental search index and boolean/phrase query support#16
Intina47 wants to merge 1 commit intomainfrom
mamba/add-local-full-text-indexing-for-notes

Conversation

@Intina47
Copy link
Owner

Motivation

  • Provide fast, local full-text search over the single-note journal with incremental updates to avoid full reindexes.
  • Support expressive queries including boolean operators and exact phrase matching for accurate retrieval.
  • Expose search from the CLI so users can run jot search <query> against the local journal.

Description

  • Implemented an incremental on-disk index stored at ~/.jot/index.json with search_index.go that hashes lines, reuses unchanged entries, and builds a term->entry map.
  • Added query parsing and boolean/phrase support in search_query.go with implicit AND, operator precedence, and conversion to RPN for evaluation.
  • Added jot search wiring in main.go which updates the index via UpdateIndex and prints matching entries returned by SearchIndex.
  • Added tests and benchmarks in search_index_test.go covering incremental update behavior and boolean/phrase queries, plus BenchmarkIndexing1k and BenchmarkSearch1k for latency measurements.

Testing

  • Ran unit tests with go test ./... and they passed (ok for the package).

Codex Task

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a805c1e9f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +38 to +42
if prev != nil && needsImplicitAnd(*prev, tok) {
tokens = append(tokens, token{kind: tokenAnd})
}
tokens = append(tokens, tok)
prev = &tok

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fix implicit AND insertion using range variable pointer

Because prev is set to &tok from a Go range loop, it always points to the loop variable that gets reused on each iteration. That means needsImplicitAnd(*prev, tok) is comparing the current token to itself instead of the previous token, so an implicit AND is inserted before every term/phrase regardless of the actual prior token. Queries with explicit operators or parentheses (e.g., a OR b, a (b)) will be rewritten incorrectly (e.g., a OR AND b) and can fail parsing or change meaning. Use a separate prevToken value or index-based loop so the previous token is preserved.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant