[Kitchen] Problems with testing CentOS

[myii]

Made some attempts to get the CentOS testing working again but ran into a number of hurdles.

centos-6

This was actually working as-is a few months back for version: '4.4:

• https://travis-ci.com/github/saltstack-formulas/zabbix-formula/jobs/250901749
  • This is the last run that worked on centos-6.

The zabbix-server-mysql package is no longer available in the repo:

• https://repo.zabbix.com/zabbix/4.4/rhel/6/x86_64/

The highest version that still has it available is 2.4 (this isn't really useful, though):

• https://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/

centos-7 & centos-8

Essentially, these are failing due to issues between docker and systemd. The main error that reoccurs:

New main PID does not belong to service, and PID file is not owned by root. Refusing.

Research:

• Cgroupfs and cgroup ns handling in Docker breaks non-root services using latest systemd docker/for-linux#835 (comment)
  • Suggestion here is to bypass the service and just run the binary directly when in a container.
• Test for CentOS 7 current failing due to bug in docker voxpupuli/puppet-zabbix#645
• cgroup delegation issue moby/moby#38749
• Stricter PIDfile handling breaks all non-root services running in docker systemd/systemd#11752 (comment)
  • Get the same duplication as well for systemd.

Potential solutions

1. Check if we're in a container and then run the service manually instead, using a check similar to:
   1. refactor(ng/init.sls): use grains.get to detect containers ntp-formula#45
2. Use Vagrant boxes instead:
   1. Biggest issue here is that this will only be local testing, which is always difficult to work with.

[an0nz]

For my own kitchen tests I got around this issue thanks to this issue apache/bigtop#640
If you mount cgroups in as a volume to your test container services can run as non-root users.
eg: /sys/fs/cgroup:/sys/fs/cgroup:ro

For Kitchen I used the following as I only test for OS's where this is needed but the same driver_config can be set in the platforms definition if you need it on specific images.

suites:
  - name: build
    driver_config:
      volume: 
        - /sys/fs/cgroup:/sys/fs/cgroup:ro

I do not use this formula so haven't tested to see if it fixes the problem just thought it might be relevant as I came across this bug while hunting for something to fix the same issue in my own salt repos.