zpool mirror what ?
Recently I came home to find my OpenSolaris server not answering on ssh and when I went to investigate, I found it off and a nasty black scorch mark where some component or other used to occupy the motherboard.
So I ordered a new motherboard, processor (core i3 35w), and 8GB ram, and while I was at it got a new case that has more fans and enough space for the 4 HDDs I have to all be in bays (I had one just sitting in the bottom of the case before) and plugged the drives back in and it booted right up.
Anyway, I had my friend Brian over after work and looking at my zpools. I have 2 pools, one is the rpool and is a 2 disk mirror and the other is for my home file share use and is similarly a 2 disk mirror. So he noticed that I had created my non root pool with partitions on the disks instead of handing zpool the whole drive which precludes zfs from being able to use the disk’s cache. So I set about to fix this. The solution seemed to be to remove one device from the storage mirror, and they add it back as a whole disk and let the mirror resilver, then swap the other drive out and back in as a whole disk.
It sounded simple. When I went to do it, I found that I could remove a device from either pool just fine, but when I went to add it back, zpool kept telling me that the device was part of a pool already. This didn’t match the config it had shown me and the disk address that I had just removed from the mirror, so I tried seeing what it would say about each of the drives in turn to find out which pools it thought things were in. I found one that it said was not part of any pool, but it wasn’t at the same scsi controller and target that I had set free.
Ok… well It let me add it, but it was at this point that I realized that this was probably because my new motherboard had a different sata controller and thus the SCSI emulation layer was detecting things differently, even though zpool status showed the old setup that it had been originally built with, underneath the disks were all at different places. Was I hosed? After adding that disk back in, the mirror resilvered, and I was able to remove the other disk from that same pool and add it back in as a whole disk, but at this point zpool hesitated. I was afraid it was going to spew bits, but then it figured out what all the new addresses were supposed to be and started showing all the disks with the correct new addressing.
:~ $ zpool status pool: longstor state: ONLINE scrub: resilver completed after 1h2m with 0 errors on Wed Sep 21 19:50:29 2011 config: NAME STATE READ WRITE CKSUM longstor ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c5d0 ONLINE 0 0 0 c6d0 ONLINE 0 0 0 244G resilvered
errors: No known data errors
pool: rpool state: ONLINE scrub: resilver completed after 0h3m with 0 errors on Wed Sep 21 18:49:20 2011 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c4d0s0 ONLINE 0 0 0 c7d0s0 ONLINE 0 0 0 5.67G resilvered errors: No known data errors
ZFS, pretty nifty stuff