rrdcached slow start with hundreds of thousands rrd files

[oreggin]

I have two LibreNMS instance, each produce and updates 500-800k rrd files. When I restart any of them, rrdcached starts very slow, it takes ~2 hours.
command line options:
/usr/bin/rrdcached -B -F -R -w 1800 -z 1800 -l 0:42217 -b /opt/librenms/rrd/ -j /var/lib/rrdcached/journal/ -G librenms -U librenms -p /run/rrdcached.pid -l unix:/run/rrdcached.sock
RRD files is on 1TByte XFS volume, blkid:
/dev/vdb: LABEL="RRD_XFS" UUID="78752a7b-bd48-4aa3-bbc2-01928c1695e8" BLOCK_SIZE="4096" TYPE="xfs"
When rrdcached starting up, I see 50-70 reading IOPs with 300-400kByte/sec I/O BW. After rrdcached started everything works fine.
First of all I need to find out what causes this very slow startup. Any opinion would be appreciated. Maybe I need to turn off the journal in rrdcached? It would be help if I move from XFS to ZFS and using persistent L2ARC?
Thanks!
Best Regards,
Tibor

[oreggin]

I moved one of them onto ZFS and checked with zpool iostat -r -n 10 what is going on at rrdcached startup, and I see it is 4k synchronous reads. I don't know what it doing at startup, but it is possible to optimize that? Maybe it is reading with single thread? After startup rrdcached performs well and producing ten times more 4k sync read IOPS then under startup phase.
Best Regards,
Tibor

[oreggin]

I tried ZFS and BTRFS, and the consequences is:

1. CoW filesystems is not suitable for RRDs. ZFS fragmentation hits 70% after 10 days, BTRFS also fragmented itself, 300GB of RRDs used 450GB after a week. IO and CPU Load is at least 2 times highers then on any of EXT4/XFS filesystems.

2. If you have hundreds of thousands of RRDs and using journal in rrdcached then you must use SSD especially on network storage otherwise rrdcached starts might takes hours because of the lot of synchronous random 4k reads.

[TheWitness]

No surprises here.

[hvisage]

On FreeBSD 14.3 using UFS on a nocache iothreads=1 virtio-scsi-single on PVE using LVM-thin, we also see like 20+ minute startups with >513 thousand RRDs. Yes, ZFS + compression is my next stop, but doesn't help to fight the following design to be understood:

using truss(1) is is reading each and every rrd file one after the other - so you WILL be disk RANDOM READ bound as doesn't matter what FS, those would be sequentially placed

so, the issue here is massive amounts of smallish files read in a single thread.
It is also something relevant to remember later, that if (in cases like LibreNMS/sflow/etc.) all the files are updated at each time interval, yes you can spread it, but still, it will be "massive"