Ubuntu 22.04, Linux kernels and me

This is a summary of interesting performance issues I have had with the HWE kernels (6.5.x) on Ubuntu 22.04, mostly this year. This is mostly a note to my future self.

• Use ext4 instead of XFS
• I have used XFS since pre-2010 because it avoided a few perf problems that show up in the ext family. But I finally found a reason to switch to ext4. The problem is that with 6.5 kernels (when HWE is used) there are perf problems and many error messages (WQ_UNBOUND, CPU hogging). Some details are here. I am curious if XFS needs a few updates to match whatever changed in the 6.5 kernels.
• Switch from schedutil to performance CPU frequency governor
• This issue hurt MyRocks a lot more than Postgres or InnoDB. I don't know the root cause but MyRocks perf was lousy with the schedutil CPU frequency governor. See here for more details from me, and this LWN post might be relevant.
• readahead requests are dropped
• RocksDB changed how readahead is done when O_DIRECT is not used and the change uses the readahead call. It looks like readahead requests are ignored when the request size is larger than the value of max_sectors_kb. Alas, to figure out the value of max_sectors_kb you must read /sys/block/$device/queue/max_sectors_kb and I prefer to avoid that (whether I have to do it, or RocksDB code does it). Surely there is a better way to handle this.