External index configuration?

[cmahnke]

Maybe I wasn't looking hard enough, but currently there seem to be no way to have an external index configuration?

There is the data attribute way, but that isn't very reusable especially since it couples templates and index definition - it's as flexible and reusable as inline CSS. I know it's possible to do some preprocessing before using the JS or Python indexing APIs, but that in turn increases the required implementation effort.

How about a method that would get rid of inline changes by using a mapping like (greatly simplified) CSS Selector => contents of data attributes.
Those can be applied in a generic preprocessor step that would be done before feeding a content file into the next step of the indexer pipeline...

[bglw]

👋 You're correct that the two methods for configuring Pagefind further are data attributes, or the indexing APIs.

Out of interest, what attributes are you looking to configure this way?

It is a feature that can be built, so it's mostly about finding a nice configuration for it that doesn't feel too cumbersome. (It's also slightly tricky to implement in the parsing setup Pagefind uses, but that's an internal issue, with a few solutions)

[cmahnke]

Just out of curiosity I've created an prototype. Have a look here:
https://gist.github.com/cmahnke/5049d42cd6dabc19cfd6c85161947fa2

You'll find the config file at the bottom, I've decided to just use all available attributes, I'm also tying to publish a blog article on this. First impression, I should have done this using the JS API not Python ;)

@bglw Have look here: https://christianmahnke.de/en/post/configure-pagefind/

[bglw]

Thanks @cmahnke ! That prototype helps picture things 🙂

I'm keen to get this (or, something reasonably close to this) built into Pagefind at some point — though I would be unwise to try put any timeframe on it. I'll have somewhat limited time to invest in Pagefind in the near future, though that will ramp back up soon enough. Within that, my main focus is a larger body of work that improves Pagefind's ranking quality when searching.

When I do have a chance to move onto this, there are some things that will make it non-trivial compared to the Python prototype. Primarily, Pagefind uses a streaming HTML parser, which means we have to catch any specified selectors as they whizz past. Doing this while maintaining the indexing performance for large sites might need a few revisions.

All that to say, I like the idea! And I'm keen to have it exist! But hopefully it isn't a blocker for Pagefind usage from you, since it won't exist imminently 😄

[cmahnke]

@bglw : Thanks for having a look!

First I need to say, that Pagefind provides great solution for doing scalable static site search, great work! Not having this as part of the distributed package isn't a blocker, that's the reason for creating this prototype.

There one thing not finished in the prototype:

• Multiple index documents for one input document
  Didn't needed it yet, but the idea was be able to generate multiple documents for indexing from one HTML file, this would allow addressing document fragments directly. This was also planned as a way to emulate multi value fields.

I terms of implementing this I would advise not to add this to the core yet. Background is my lack of understanding of Rust: I suspect it to be harder then a JS based (initial) solution. Adding it on the JS level has several benefits from my point of view:

• Better hackable by the community:
  JS has a broader community, so contributions are more likely. Same goes for the proposed preprocessing pipeline plugins (as mentioned at the end of the blog article).
• Second stage prototype:
  By broadening the possible base of contributers it might be possible to identify common patterns before moving those to the core. This is especially true for indexing related issues (see below).
• No core changes needed, this would only affect the JS wrapper
  Having this completely optional wouldn't impose any performance penalties

Actually it would also be possible to develop something like this completely externally (as in: not as part of this project), there might be some interest since the prototype also addresses several existing issues:

• Exclude one or more directories and/or files #772
• Skip page aliases #787

By the way, there are some other things that might be considered:

• PagefindUI might be based on the ModularUI that would keep the options and features more closely together an reducing the maintenance overhead.
• Better reference documentation, like a list of all attributes, they are currently scattered across those sections covering them. Same goes for query operators and syntax.
• Multi value fields for meta would be nice, maybe implicit by collecting all variations given in a document

All that said, I won't have resources to contribute any of those things for the next months, but might comment on any developments in such a direction.

[bglw]

Adding this is something I would be doing in core Pagefind rather than in one of the wrapper packages. (Obviously, I don't mind anything being developed completely externally! But if it is to become a Pagefind feature it has to be in core).

I'll keep you posted as and when there's any movement here 🙂