Silently fails for new language

[kafan1986]

I tried training the model with some "hindi" data but the model silently fails

https://go.dev/play/p/58RVSno27ot

[mish15]

interesting, have not looked at this for a long time. Will ping some others. Might be worth adding a test in a pull request, even if it is failing, will help to look at it

[kafan1986]

@mish15 One more thing I noticed for auto completion was not working when set with any threshold apart from 0. Which means same model can not be used for autocompletion and suggestions

[mish15]

a threshold greater than zero requires more training to determine the validity of text.