Phantom writes are one reason ZFS stores checksums separately from the blocks being written. If you store checksums with the block itself, then failure modes like phantom writes can result in reading data that you think is valid (because the checksum matches) but it's the wrong data.
The worst part of this is that often when disks fail, they just become extremely slow (100s of milliseconds) rather than explicitly failing. That can be significantly worse than just failing explicitly and having the OS read from another disk.
Nit: Remember the old fallacy: if there's a 50% chance of rain on each of Saturday and Sunday, that doesn't mean there's 100% chance of rain this weekend. The quoted number of nonrecoverable errors per bits read is presumably an expected value on a uniform distribution (which is ludicrous, since errors are not even remotely independent, but I don't know what else they could be expecting the reader to assume). In that case, the expected probability of an error on each bit read is 1 / 1E14. The probability of zero errors after reading 3E13 bits is ((1E14-1)/1E14)^3E13, which is about 75%.
> The worst part of this is that often when disks fail, they just become extremely slow (100s of milliseconds) rather than explicitly failing.
This is the main difference between consumer-grade and enterprise-grade drives firmware. Take the HGST Deskstar and Ultrastar, or Seagate Constellation ES and Barracuda 2 TB; they are physically exactly the same, but behave very differently when encountering an error. Basically the enterprise drive will fail shortly BUT recover (because it's probably RAID-backed), while the desktop drive will try to read the data when asked at all prices (because it's probably NOT RAID-backed), therefore suffering from extremely long time-outs.
> . The probability of zero errors after reading 3E13 bits is ((1E14-1)/1E14)^3E13, which is about 75%.
Still not the sort of statistics you want betting your precious data against...
>This is the main difference between consumer-grade and enterprise-grade drives firmware. Take the HGST Deskstar and Ultrastar, or Seagate Constellation ES and Barracuda 2 TB; they are physically exactly the same, but behave very differently when encountering an error. Basically the enterprise drive will fail shortly BUT recover (because it's probably RAID-backed), while the desktop drive will try to read the data when asked at all prices (because it's probably NOT RAID-backed), therefore suffering from extremely long time-outs.
That is, in fact, why I pay the huge premium for 'enterprise' sata.
And yes, Usually 'enterprise' drives fail rather than just getting really slow.
But not always. 'enterprise' drives sometimes get shitty rather than just failing outright, too. I mean, it's usually like 1/3rd to 1/5th expected performance rather than 1/100th, but this is disk. It's already on the edge of unacceptably slow. just cutting performance in half means I'm getting complaints.
Overall? reducing this problem from twice a month to twice a year is worth the premium, but all spinning rust is shit. The 'enterprise' stuff is only slightly less shit.
It bothers me that RAID controllers don't handle this more intelligently.
The worst part of this is that often when disks fail, they just become extremely slow (100s of milliseconds) rather than explicitly failing. That can be significantly worse than just failing explicitly and having the OS read from another disk.
Nit: Remember the old fallacy: if there's a 50% chance of rain on each of Saturday and Sunday, that doesn't mean there's 100% chance of rain this weekend. The quoted number of nonrecoverable errors per bits read is presumably an expected value on a uniform distribution (which is ludicrous, since errors are not even remotely independent, but I don't know what else they could be expecting the reader to assume). In that case, the expected probability of an error on each bit read is 1 / 1E14. The probability of zero errors after reading 3E13 bits is ((1E14-1)/1E14)^3E13, which is about 75%.