Improve Logging Flexibility and Systemd Integration in Graylog Sidecar

[nesterenkovy]

I would like to propose several improvements for better control over logging in Graylog Sidecar, including enhanced configurability, reduced disk writes, and better integration with systemd.

1. Configurable Logging Levels

Currently, Graylog Sidecar does not provide flexible control over log levels. It would be beneficial to allow users to specify log levels in /etc/graylog/sidecar/sidecar.yml, following the available levels in the logrus library (Debug, Info, Warn, Error, Fatal, Panic). This would help fine-tune logging verbosity based on user requirements.

2. Ability to Disable Logging to a Separate File

To minimize write operations on SSDs, there should be an option to disable logging to a separate file (sidecar.log). Right now, the log file is created in the directory specified by log_path in sidecar.yml. As a temporary workaround, I am using log_path: "/run/log/graylog-sidecar" instead of the default /var/log/graylog-sidecar, but a built-in option to fully disable file logging would be much better.

3. Avoid Duplicate Timestamps in systemd Journal

Currently, when logs are redirected to journald via systemd, the logrus library formats messages as if they were being written to a file, resulting in duplicated timestamps in the system journal. Example:

Mar 05 15:44:28 rocky-01 systemd[1]: Started Wrapper service for Graylog controlled collector.
Mar 05 15:44:28 rocky-01 graylog-sidecar[2764]: time="2025-03-05T15:44:28+02:00" level=info msg="Using node-id: d837ad1e-b5a8-469c-ac09-49b6d6172bae"
Mar 05 15:44:28 rocky-01 graylog-sidecar[2764]: time="2025-03-05T15:44:28+02:00" level=info msg="No node name was configured, falling back to hostname"
Mar 05 15:44:28 rocky-01 graylog-sidecar[2764]: time="2025-03-05T15:44:28+02:00" level=info msg="Starting signal distributor"
Mar 05 15:44:28 rocky-01 graylog-sidecar[2764]: time="2025-03-05T15:44:28+02:00" level=info msg="Adding process runner for: filebeat-67c2d2136aab1b202b11994f"
Mar 05 15:44:28 rocky-01 graylog-sidecar[2764]: time="2025-03-05T15:44:28+02:00" level=info msg="Adding process runner for: auditbeat-67c2d2136aab1b202b119956"

This redundancy makes log analysis harder and clutters the journal. Ideally, Sidecar should recognize when logs are being forwarded to journald and adjust its formatting accordingly.

4. Native Support for sd_journal_send

For modern Linux distributions, it would be great to have an option to use sd_journal_send() instead of standard console output logging. This would improve compatibility with systemd-based logging and make logs more structured in journald.

Would it be possible to consider these improvements in future releases? I believe they would enhance the usability and efficiency of Graylog Sidecar, especially in systemd-based environments.

Thank you!

[tellistone]

Thanks for these suggestions, we'll revisit these when we get to next planning improvements to Graylog Sidecar