I just spent several weeks troubleshooting my setup and the conclusion is that the Dell Optiplex 7060 is either accidentally or deliberately handicapped by design flaws. The one I'm about to talk about wouldn't be the first such design flaw I've found; I think I posted about this previously, but installing most kinds of PCI Express cards, notably disk controllers and faster-than-gigabit networking, is not compatible with having memory in slot 4, and you have to tape over some of the pins on your PCIe cards to work around it.
Any USB 3.0 adapter in the system, whether the built-in controller or a PCIe card, cannot safely run multiple external drives in USB 3.0 drive enclosures without causing data corruption. Nothing I do will get this setup to stop generating CKSUM errors while scrubbing a ZFS raidz1.
Connecting the exact same enclosure to the exact same machine except over eSATA makes the data corruption go away, but it runs dramatically slower. Same with using USB 2.0. I even rigged the drives to the internal SATA ports and powered them with an external power supply and that worked perfectly. (And it was dramatically faster!)
There is nothing else wrong with the machine that I have been able to find with all manner of stress testing. The USB enclosure works perfectly in other machines and I can scrub the zpool with no errors there. It's just the Dell Optiplex using multiple drives over USB 3.0 that causes it.
It doesn't make any sense. Why does a PCIe USB 3.0 card have the same problem as the onboard USB 3.0? Why does the same USB 3.0 card in another machine also work perfectly? What the hell could be causing the problem? Noise on the PCIe lanes? But why does it only affect USB 3.0 controllers?
I got sick of troubleshooting and bought some SATA SSDs I can mount inside the Optiplex and use the SATA ports that I've already confirmed work perfectly. That's kinda what I wanted to do anyway, but I was waiting for an impossibly good deal on SAS SSDs. If I didn't just Pick Something to Solve The Problem and go for it, I was going to hesitate and pass on pretty-good deals forever.
.png)