Preserve patterns#5

Merged

cpetersen merged 2 commits intomainfrom

preserve-patterns

Sep 29, 2025

Commits on Sep 28, 2025

Implementing preserving patterns for letter, lowercase and whitespace tokenizer, plus limited support for char group tokenizer
cpetersen
committed
Implementing preserve_patterns for url/email and sentence tokenizer. Partially implemented for path hierarchy tokenizer
cpetersen
committed