I got woken up at 4:00 this morning by a page from Big Brother notifying me that one of my Seagate ST3160827AS SATA drives died. I’ve got two of them in a RAID-1 configuration so there was no data loss. Here’s the relavent log entries:

Nov 19 04:04:06 notanut kernel: ata1: status=0x51 { DriveReady SeekComplete Error }
Nov 19 04:04:06 notanut kernel: ata1: error=0x40 { UncorrectableError }
Nov 19 04:04:10 notanut kernel: ata1: status=0x51 { DriveReady SeekComplete Error }
Nov 19 04:04:10 notanut kernel: ata1: error=0x40 { UncorrectableError }
Nov 19 04:04:13 notanut kernel: ata1: status=0x51 { DriveReady SeekComplete Error }
Nov 19 04:04:13 notanut kernel: ata1: error=0x40 { UncorrectableError }
Nov 19 04:04:17 notanut kernel: ata1: status=0x51 { DriveReady SeekComplete Error }
Nov 19 04:04:17 notanut kernel: ata1: error=0x40 { UncorrectableError }
Nov 19 04:04:21 notanut kernel: ata1: status=0x51 { DriveReady SeekComplete Error }
Nov 19 04:04:21 notanut kernel: ata1: error=0x40 { UncorrectableError }
Nov 19 04:04:21 notanut kernel: sd 0:0:0:0: SCSI error: return code = 0x8000002
Nov 19 04:04:21 notanut kernel: sda: Current: sense key: Medium Error
Nov 19 04:04:21 notanut kernel: Additional sense: Unrecovered read error – auto reallocate failed
Nov 19 04:04:21 notanut kernel: end_request: I/O error, dev sda, sector 53104600

So I got out to Fry’s and picked up another one and set about replacing it. I first tried to “warm swap” it using some SCSI commands. That didn’t work at all, so on to plan B. Boot off the rescue CD, rebuild the partition table on the new drive, resync the RAID, reinstall the grub boot block, and bring the system back up. Thankfully everything went fine, just a little trouble with paths in the grub-install script that was easily fixed with vi. Two hours later the machine’s back up and running