Excluding folders from index

[ndeville]

Is there a way to exclude (sub)folders or specific files from the index?

I'm getting duplicate results because the same article will show up in different folders, eg tags folder, author folder, etc..

Using Pelican as SSG and I can't figure out how to use data-pagefind-body or data-pagefind-ignore in the article's HTML in a way that they would be targeted depending on folder the article resides in.

For clarity, here is my output/source folder indicating where folders should be removed:

.
├── 2019  /// REMOVE
├── 2020  /// REMOVE
├── 2021  /// REMOVE
├── 2022  /// REMOVE
├── ABOUT  /// REMOVE
├── BingSiteAuth.xml
├── CNAME
├── ai
├── apps
├── archives.html  /// REMOVE
├── author  /// REMOVE
├── authors.html  /// REMOVE
├── b2b-sales
├── books
├── careerplaybook
├── categories
├── csvs  /// REMOVE
├── extra  /// REMOVE
├── favicon.ico
├── googlexxxxxxxxxxx.html  /// REMOVE
├── helpers
├── home-office
├── images
├── index.html
├── interests
├── leadership
├── learning
├── movies
├── pagefind  /// my Pagefind index folder
├── pdfs
├── projects
├── python
├── random
├── robots.txt
├── sitemap.xml
├── startups
├── tag  /// REMOVE
├── tags.html  /// REMOVE
└── theme  /// REMOVE

[bglw]

Hello! 👋

Hmm, Pagefind could probably do with a better way of excluding things if there are this many folders and files to exclude, but there is a way now that might just take a few experiments.

Using the glob option on the Pagefind CLI, you can control what files are ingested (instead of the default **/*.{html} which captures everything. The glob will be slightly ugly when it gets this complicated, since you'll need to opt-in to files for now — I'd definitely recommend configuring it in a pagefind config file rather than over the CLI.

For example, if I have the file tree:

index.html
other.html # <- I want to exclude this file
about/
  index.html
  nested/
    index.html

I could use the pagefind.yml config file:

glob: "{index.html,about/**/*.html}"

To only index the root index.html file, as well as all .html files anywhere under the about folder. (Thus skipping other.html by omission)

Looking at your file tree, this will unfortunately get to be quite a long glob string, so I'll also open an issue for better configuration support here (The glob option could take an array, and there could be a matching exclude_glob option)

(I've just added a test for complex globs to make sure it works as expected: Scenario: Complex exclusionary file glob can be configured, so you can see a real implementation there)

[ndeville]

Yes indeed, sorry I didn't revert to confirm.
The glob approach worked well. Thank you!

Remaining issues I need to address:

• pagefind indexes category index pages (need to figure out how to carve these out)
• categories get displayed in a weird way where I need to click on each, and it shows empty ones despite having showEmptyFilters: false in pagefind.yml

See https://notes.nicolasdeville.com

But I need to spend some time on these and will post another discussion thread if stuck.

Thanks again for your help - and your hard work on Pagefind. Really love it!

[bglw]

Nice.

categories get displayed in a weird way

You'll want to adjust your filter tagging — the value in the attribute is used as the filter name, and the value of the element is used as the filter value. So where you currently have something like:

<span data-pagefind-filter="books">books</span>

You'll instead want:

<span data-pagefind-filter="category">books</span>

Where everything across the site is tagged with the category filter key. The former example above is equivalent to { books: "books" }, and the latter is { category: "books" }

With that tagging, you would only see one filter group of category on the left of your search, and each option would be a checkbox in that group :)

pagefind indexes category index pages (need to figure out how to carve these out)

Would adding an exclude_glob option (#128) help with this one?

[ndeville]

The filter change worked. Thanks!
Any way I can have it so that Category is expanded by default instead of collapsed? (see Live implementation)

For the category index pages what I don't understand though is that your glob suggestion has the format "{{xxx,yyy}/**/*.html}" and thus should not index them in the first place as these are like "{{xxx,yyy}/index*.html}" 🤔

This is about removing from Pagefind the index*.html files at the root of each category folder.

Perhaps an exclude_glob with "{{xxx,yyy}/index*.html}" would work indeed. I'll look into it.

Thanks again!

[bglw]

Any way I can have it so that Category is expanded by default

I can add an option for this. (Currently it collapses by default if there are many filter options). Tracking as #148

glob suggestion

A {{xxx,yyy}/**/*.html} glob will match {{xxx,yyy}/index*.html} — the /**/ globstar matches any amount of directories (including zero). To exclusively go another directory deep you could roll with something like {{xxx,yyy}/*/**/*.html}, a single star to match any directory, and then a globstar for any further amount.

Definitely sounds like an exclude_glob will be the best option, though. I'll look at getting this implemented soon. Tracking as #128

[ndeville]

You're great! 🏆

Learned something about glob and the new syntax solved my issue! 😁

Expand by default would be a nice to have but not an issue.

Thanks again for all your work & support on Pagefind! 🙏🏼

Let me know if I can help with anything.