so i noticed some weird performance issues with my zfs array on my homeserver/nas a couple days ago... none of this is top tier hardware (5400rpm disks, a micro office pc, etc.) so it was never going to be blazing fast, but i got a lot more out of these disks when they were in my desktop before. i did a lot of investigating and what i could figure out was:
- copying/moving between datasets on the pool was very slow (like 10MB/s tops)
- copying from the pool to SSD was fast (like >100MB/s)
- copying from SSD to the pool was slow (like 20MB/s ish)
- well-seeded torrents ramped up quickly to around 100MB/s (i have gigabit) but then dropped down to around 20MB/s or lower fairly quickly (obviously a lot of moving parts here)
- zfs scrub was fast (issuing at well over 100MB/s)
- cpu was mostly idle so it wasn't bottlenecking on the FS processing or something
- SMART data was fine
- no error messages from zfs or dmesg or anything
this suggested to me that writes were the bottleneck somehow, and it was probably something about the zpool setup, the enclosure, or the server
tried a bunch of stuff including rebooting, changing recordsize, twiddling a couple other options (compression, atime)... no dice. then i realized that a reboot doesn't power cycle the usb3 storage enclosure (QNAP TR-004U), which has its own power adapter.
so i shut down the server and turned off the enclosure for good measure, then booted everything from cold, and... it seems to have improved things a bunch?
- copying/moving between datasets is still slow but it's more like 30MB/s now, so still an improvement
- copying from SSD to the pool is like 70MB/s or so
- well-seeded torrents seem able to run at like 50MB/s for extended periods
i'm kinda mad that i spent hours on this and didn't find any sort of root cause, but shit, i'll take it tbh. although the atime (off) and recordsize (128k β 1M) changes probably helped too.
