Templated SMTP Output: Enforce SMTPUTF8 policy

[Psych0meter]

The bot encountered this kind of error when processing emails containing non-ASCII characters (e.g., accents or special symbols):

UnicodeEncodeError: 'ascii' codec can't encode character '\xe0' in position 4: ordinal not in range(128)

Dump example :

{
    "__type": "Event",
    "extra.doc_count": 380,
    "extra.subject": "Merci \u00e0 tous pour cette ann\u00e9e GiGA exceptionnelle \u2013 cap sur 2025 ! (raw: Merci =?utf-8?Q?=C3=A0?= tous pour cette =?utf-8?Q?ann=C3=A9e?= GiGA exceptionnelle =?utf-8?Q?=E2=80)",
    "feed.accuracy": 100.0,
    "feed.name": "xxx",
    "feed.provider": "xxx",
    "raw": "xxx",
    "time.observation": "2024-12-23T20:53:35.319135+00:00"
}

[sebix]

Thank you for the fix! I'll look into it, hopefully this weekend.

If the CI tests fail, it's likely not because of your changes.

[sebix]

I'm struggling to reproduce the error you are getting with \u00e0.

For this purpose I added the character to the tests: sebix@40cbc90
It works on all Python versions: https://github.com/sebix/intelmq/actions/runs/13356080154

The switch to the UT8 Policy also changes the line separator
https://docs.python.org/3/library/email.policy.html#email.policy.SMTPUTF8
It further states:

Should only be used for SMTP transmission if the sender or recipient addresses have non-ASCII characters (the smtplib.SMTP.send_message() method handles this automatically).

In your example, the non-ASCII characters are in the subject, not in from or to.

https://docs.python.org/3/library/email.policy.html#email.policy.EmailPolicy.utf8
makes me wonder even more about the issue you experienced. It says

If False, follow RFC 5322, supporting non-ASCII characters in headers by encoding them as “encoded words”. If True, follow RFC 6532 and use utf-8 encoding for headers.

So the switch doesn't enable/disable UTF-8, it just activates the conversion. Both settings should work.

[sebix]

I now tested sending an e-mail with a recipient-address containing a non-ASCII character (user+à@example.com) to a server not supporting SMTPUTF8 (postfix-2.11.3) and I got:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/smtplib.py", line 969, in send_message
    ''.join([from_addr, *to_addrs]).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xe0' in position 46: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/intelmqctl", line 33, in <module>
    sys.exit(load_entry_point('intelmq', 'console_scripts', 'intelmqctl')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastianw/dev/intelmq/intelmq/bin/intelmqctl.py", line 1302, in main
    return x.run()
           ^^^^^^^
  File "/home/sebastianw/dev/intelmq/intelmq/bin/intelmqctl.py", line 430, in run
    retval, results = args.func(**args_dict)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastianw/dev/intelmq/intelmq/bin/intelmqctl.py", line 441, in bot_run
    retval, results = self._processmanager.bot_run(**kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sebastianw/dev/intelmq/intelmq/lib/processmanager.py", line 132, in bot_run
    output = bd.run()
             ^^^^^^^^
  File "/home/sebastianw/dev/intelmq/intelmq/lib/bot_debugger.py", line 82, in run
    self._process(self.dryrun, self.msg, self.show)
  File "/home/sebastianw/dev/intelmq/intelmq/lib/bot_debugger.py", line 166, in _process
    self.instance.process()
  File "/home/sebastianw/dev/intelmq/intelmq/bots/outputs/templated_smtp/output.py", line 212, in process
    smtp.send_message(msg, from_addr=msg['From'],
  File "/usr/lib64/python3.11/smtplib.py", line 972, in send_message
    raise SMTPNotSupportedError(
smtplib.SMTPNotSupportedError: One or more source or delivery addresses require internationalized email support, but the server does not advertise the required SMTPUTF8 capability

But the result is the same, independent of using policy.SMTPUTF8 or not.

So the issue is not the subject but the sender/recipient address and that your server doesn't support SMTPUTF8 and therefore requires a conversion for compatibility.

[aaronkaplan]

So what to do with this PR now? Merge or leave?
@sebix ?

[sebix]

IMO postpone until the situation (bug report) is clarified.