(feat): add text normalization for Thai and emoji

[triphopp]

What does this changes

• Emoji replacement with sentiment-aware tokens (<<EMO_POS>>, <<EMO_NEG>>)
• Normalization of elongated/repeated characters (e.g., "มากกกกกก" → "มากก")

What was wrong

• Sentiment classification of emojis

How this fixes it

Applies sentiment-aware emoji replacement based on a defined dictionary

Your checklist for this pull request

• [ /] Passed code styles and structures
• [/ ] Passed code linting checks and unit test

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bact]

Thank you.

Currently our text normalization functions are in
https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/util/normalize.py

If you like, you can try to fit the new functions in that file structure.

[coveralls]

coverage: 52.88%. remained the same
when pulling 74ed33d on triphop-mahithi:text-cleaning
into a069230 on PyThaiNLP:dev.

[triphopp]

Thank you.

Currently our text normalization functions are in https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/util/normalize.py

If you like, you can try to fit the new functions in that file structure.

Thank you for pointing that out.
I apologize — I didn’t look carefully enough and missed the function in the file. I’ll take another look and try to fit the new functions accordingly.

[bact]

You can also try to use the list defined here:

pythainlp/pythainlp/__init__.py

Lines 20 to 22 in a069230

	# Paiyannoi, Maiyamok, Phinthu, Thanthakhat, Nikhahit, Yamakkan:
	# These signs can be part of a word
	thai_signs = "\u0e2f\u0e3a\u0e46\u0e4c\u0e4d\u0e4e" # 6 chars

(it doesn't include ฯลฯ though)

[bact]

Suggested change

	emoji_sentiment = {
	"positive": [
	"😊", "😁", "😂", "🤣", "😄", "😍", "😘", "😻", "👍", "👏", "💕", "❤️", "😇", "😎", "🥰", "😃", "☺️"
	],
	"negative": [
	"😢", "😭", "😠", "😡", "😤", "👎", "💔", "😞", "😖", "😩", "😣", "😫", "😓", "😰", "😱", "😿"
	],
	"neutral": [
	"😐", "😶", "🤔", "😑", "😬", "😴", "😕", "😒", "🙄", "😮", "🤨", "😲"
	]
	}
	def replace_emoji_with_sentiment(sentence: str, emoji_dict: dict) -> str:
	for emo in emoji_dict["positive"]:
	sentence = sentence.replace(emo, " <<EMO_POS>> ")
	for emo in emoji_dict["negative"]:
	sentence = sentence.replace(emo, " <<EMO_NEG>> ")
	for emo in emoji_dict["neutral"]:
	sentence = sentence.replace(emo, " <<EMO_NEU>> ")
	return sentence
	emoji_sentiment = {
	"POS": [
	"😊", "😁", "😂", "🤣", "😄", "😍", "😘", "😻", "👍", "👏", "💕", "❤️", "😇", "😎", "🥰", "😃", "☺️"
	],
	"NEG": [
	"😢", "😭", "😠", "😡", "😤", "👎", "💔", "😞", "😖", "😩", "😣", "😫", "😓", "😰", "😱", "😿"
	],
	"NEU": [
	"😐", "😶", "🤔", "😑", "😬", "😴", "😕", "😒", "🙄", "😮", "🤨", "😲"
	]
	}
	# Create an emoji-sentiment map from `emoji_sentiment`
	emoji_to_tag = {}
	for sentiment, emojis in emoji_sentiment.items():
	tag = f" <<EMO_{sentiment}>> "
	for emo in emojis:
	emoji_to_tag[emo] = tag
	# Alternatively, we can have the static map predefined.
	def replace_emoji_with_sentiment(sentence: str) -> str:
	for emo, tag in emoji_to_tag.items():
	sentence = sentence.replace(emo, tag)
	return sentence

We can also try to reduce the number of text passes from three to one by having an emoji-sentiment map.

[triphopp]

Thanks! would it be okay if I add this code to the text normalization function?

[bact]

please take it and use it in any way you want :)

[triphopp]

Okay, I think this is the same as emojiconv.py

[wannaphong]

@triphopMahithi Hello! Any update?

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[bact]

please rebase as we have updated a test.