http://blog.gmane.org/gmane.os.solaris.opensolaris.zfs hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49232 <pre>Sorry I can't comment on MPxIO, except that I thought zfs could by itself discern two paths to the same drive, if only to protect against double-importing the disk into pool. 2012-05-25 21:07, Sašo Kiselkov wrote: &gt; I'd like to create a 5x 9-drive raidz's on the JBOD, but I am not sure it is a good idea to use such low protection (raidz1) with large drives. At least, I was led to believe that after 2Tb in size raidz2 is preferable, and raidz3 is optimal due to long scrub/resilver times leading to large timeframes that a pool with an error is exposed to possible fatal errors (due to double-failures with single-protection). //Jim </pre> Jim Klimov 2012-05-25T17:35:51 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49231 <pre>I'm currently trying to get a SuperMicro JBOD with dual SAS expander chips running in MPxIO, but I'm a total amateur to this and would like to ask about how to detect whether MPxIO is working (or not). My SAS topology is: *) One LSI SAS2008-equipped HBA (running the latest IT firmware from LSI) with two external ports. *) Two SAS cables running from the HBA to the SuperMicro JBOD, where they enter the JBOD's rear backplane (which is equipped with two LSI SAS expander chips). *) From the rear backplane, via two internal SAS cables to the front backplane (also with two SAS expanders on it) *) The JBOD is populated with 45 2TB Toshiba SAS 7200rpm drives The machine also has a PERC H700 for the boot media, configured into a hardware RAID-1 (on which rpool resides). Here is the relevant part from cfgadm -al for the MPxIO bits: c5 scsi-sas connected configured unknown c5::dsk/c5t50000393D8CB4452d0 disk connected configured unknown c5::dsk/c5t50000393E8C90CF2d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2A6d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2AAd0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2BEd0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2C6d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2E2d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF2F2d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF5C6d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF28Ad0 disk connected configured unknown c5::dsk/c5t50000393E8CAF32Ed0 disk connected configured unknown c5::dsk/c5t50000393E8CAF35Ad0 disk connected configured unknown c5::dsk/c5t50000393E8CAF35Ed0 disk connected configured unknown c5::dsk/c5t50000393E8CAF36Ad0 disk connected configured unknown c5::dsk/c5t50000393E8CAF36Ed0 disk connected configured unknown c5::dsk/c5t50000393E8CAF52Ed0 disk connected configured unknown c5::dsk/c5t50000393E8CAF53Ad0 disk connected configured unknown c5::dsk/c5t50000393E8CAF53Ed0 disk connected configured unknown c5::dsk/c5t50000393E8CAF312d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF316d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF506d0 disk connected configured unknown c5::dsk/c5t50000393E8CAF546d0 disk connected configured unknown c5::dsk/c5t50000393F8C84F5Ed0 disk connected configured unknown c5::dsk/c5t50000393F8C84FBAd0 disk connected configured unknown c5::dsk/c5t50000393F8C851EEd0 disk connected configured unknown c5::dsk/c5t50000393F8C852A6d0 disk connected configured unknown c5::dsk/c5t50000393F8C852C2d0 disk connected configured unknown c5::dsk/c5t50000393F8C852CAd0 disk connected configured unknown c5::dsk/c5t50000393F8C852EAd0 disk connected configured unknown c5::dsk/c5t50000393F8C854BAd0 disk connected configured unknown c5::dsk/c5t50000393F8C854E2d0 disk connected configured unknown c5::dsk/c5t50000393F8C855AAd0 disk connected configured unknown c5::dsk/c5t50000393F8C8509Ad0 disk connected configured unknown c5::dsk/c5t50000393F8C8520Ad0 disk connected configured unknown c5::dsk/c5t50000393F8C8528Ad0 disk connected configured unknown c5::dsk/c5t50000393F8C8530Ed0 disk connected configured unknown c5::dsk/c5t50000393F8C8531Ed0 disk connected configured unknown c5::dsk/c5t50000393F8C8557Ed0 disk connected configured unknown c5::dsk/c5t50000393F8C8558Ed0 disk connected configured unknown c5::dsk/c5t50000393F8C8560Ad0 disk connected configured unknown c5::dsk/c5t50000393F8C85106d0 disk connected configured unknown c5::dsk/c5t50000393F8C85222d0 disk connected configured unknown c5::dsk/c5t50000393F8C85246d0 disk connected configured unknown c5::dsk/c5t50000393F8C85366d0 disk connected configured unknown c5::dsk/c5t50000393F8C85636d0 disk connected configured unknown c5::es/ses0 ESI connected configured unknown c5::es/ses1 ESI connected configured unknown c5::smp/expd0 smp connected configured unknown c5::smp/expd1 smp connected configured unknown c6 scsi-sas connected configured unknown c6::dsk/c6t50000393D8CB4453d0 disk connected configured unknown c6::dsk/c6t50000393E8C90CF3d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2A7d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2ABd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2BFd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2C7d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2E3d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF2F3d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF5C7d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF28Bd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF32Fd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF35Bd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF35Fd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF36Bd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF36Fd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF52Fd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF53Bd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF53Fd0 disk connected configured unknown c6::dsk/c6t50000393E8CAF313d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF317d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF507d0 disk connected configured unknown c6::dsk/c6t50000393E8CAF547d0 disk connected configured unknown c6::dsk/c6t50000393F8C84F5Fd0 disk connected configured unknown c6::dsk/c6t50000393F8C84FBBd0 disk connected configured unknown c6::dsk/c6t50000393F8C851EFd0 disk connected configured unknown c6::dsk/c6t50000393F8C852A7d0 disk connected configured unknown c6::dsk/c6t50000393F8C852C3d0 disk connected configured unknown c6::dsk/c6t50000393F8C852CBd0 disk connected configured unknown c6::dsk/c6t50000393F8C852EBd0 disk connected configured unknown c6::dsk/c6t50000393F8C854BBd0 disk connected configured unknown c6::dsk/c6t50000393F8C854E3d0 disk connected configured unknown c6::dsk/c6t50000393F8C855ABd0 disk connected configured unknown c6::dsk/c6t50000393F8C8509Bd0 disk connected configured unknown c6::dsk/c6t50000393F8C8520Bd0 disk connected configured unknown c6::dsk/c6t50000393F8C8528Bd0 disk connected configured unknown c6::dsk/c6t50000393F8C8530Fd0 disk connected configured unknown c6::dsk/c6t50000393F8C8531Fd0 disk connected configured unknown c6::dsk/c6t50000393F8C8557Fd0 disk connected configured unknown c6::dsk/c6t50000393F8C8558Fd0 disk connected configured unknown c6::dsk/c6t50000393F8C8560Bd0 disk connected configured unknown c6::dsk/c6t50000393F8C85107d0 disk connected configured unknown c6::dsk/c6t50000393F8C85223d0 disk connected configured unknown c6::dsk/c6t50000393F8C85247d0 disk connected configured unknown c6::dsk/c6t50000393F8C85367d0 disk connected configured unknown c6::dsk/c6t50000393F8C85637d0 disk connected configured unknown c6::es/ses2 ESI connected configured unknown c6::es/ses3 ESI connected configured unknown c6::smp/expd2 smp connected configured unknown c6::smp/expd3 smp connected configured unknown I can see all drives in format(1M), but "mpathadm list lu" doesn't show anything, even after I did "stmsboot -e" (and rebooted). Can anybody please help me find out whether MPxIO is properly enabled and how I can start using it? My understanding is that MPxIO should mask each drive under some common name (e.g. c5t20d0) and underneath handle all of the multipathing internally, but maybe I'm wrong and it behaves completely differently. I'd like to create a 5x 9-drive raidz's on the JBOD, but I'm not sure how to do it now that I can see each drive twice... All help greatly appreciated! Cheers, -- Saso </pre> Sašo Kiselkov 2012-05-25T17:07:15 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49230 <pre>I have a chance to pick up a system at a reasonable price built with an AMD FX8120 8 core 3.1 GHz on a Gigabyte motherboard. Is anybody runing with this combo? Looking for info from an actual user. Thanks. </pre> Nomen Nescio 2012-05-25T16:01:30 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49229 <pre> Indeed it is, and I've covered this in the thread earlier - the bulk copying phase ("DD-phase") should monitor its real progress, and if it detects lags in comparison to the average or expected speeds (expected = some tuning variable i.e. 50Mb/s), the process should skip over some (arbitrary) range of sectors and go on from another location (such skipped sectors are in danger indeed, until the scrub-phase detects and reconstructs them) or fall back to the original resilver method completely. That was already described in some detail I thought of at the time of the posting, and I can't add much to that yet. From what I've seen with faulty sectors is that they are usually either single errors or a "scratched" range which can be worked around with i.e. partitioning for legacy FSes (if the SMART relocation doesn't deal with them properly for any reason), while most of the rest of the disk is okay. Retries may be lengthy, ranging from several seconds up to a minute, but they are often constrained in a few locations and *may* add little delay in the overall scheme of things. If the delay is more than acceptable and/or we can't find a "working location" on the source disk, we just fall back to the old method - either original resilver, or if much data has been copied to the new disk - to the new selective scrub (it being much like the resilver, but taking into account those sectors on the target disk which may have been copied over correctly). A somewhat worse case is intermittent errors in random times and logical disk locations due to who knows what - overheating, firmware overflow errors, bus resets, or whatever. It's rather them being the reason for scrub-validation of data after mass migration, perhaps (as well as a reason for preventive regular scrubs)... //Jim </pre> Jim Klimov 2012-05-24T15:53:11 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49228 <pre>big assumption below... On May 24, 2012, at 6:06 AM, Jim Klimov wrote: This is a big assumption -- that the disk will operate normally, even for data it cannot read. In my experience, this assumption is not valid for the majority of HDD failure modes. Also, in the case of consumer-grade disks, a single sector media error could take a very long time to retry/fail. np -- richard -- ZFS Performance and Training Richard.Elling&lt; at &gt;RichardElling.com +1-760-896-4422 _______________________________________________ zfs-discuss mailing list zfs-discuss&lt; at &gt;opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss </pre> Richard Elling 2012-05-24T14:55:35 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49227 <pre>Let me try to formulate my idea again... You called a similar process "pushing the rope" some time ago, I think. I feel like I'm passing some exam and am trying to pick answers for a discipline like philosophy and I have no idea about the examinator's preferences - is he an ex-Communism teacher or an eager new religion fanatic? The same answer can lead to an A or to an F on a state exam. Ah, that was some fun experience :) Well, what we know is what remains after we forget everything that we were taught, while the exams are our last chance to learn something at all =) 2012-05-24 10:28, Richard Elling wrote: Bigger-better-faster? ;) The original proposal in this thread was about understanding how resilvers and scrubs work, why they are so dog slow on HDDs in comparison to sequential reads, and thinking aloud what can be improved in this area. One of the later posts was about improving the disk replacement (where the original is still responsive, but may be imperfect) for filled-up fragmented pools by including a stage of fast data transfer and a different IO pattern for verification and updating of the new disk image, in comparison with current resilver's IO patterns. This may or may not have some benefits in certain (corner?) cases which are of practical interest to some users on this list, and if this discussion leads to a POC made by a competent ZFS programmer, which can be tested on a variety of ZFS pools (without risking one's only pool on a homeNAS) - so much the better. Then we would see if this scenario is viable or utterly useless and bad in every tested case. The practical numbers I have from the same box and disks are: * Copy from a 250Gb raidz1 (9*(4+1)) pool to a single-disk 3Tb test pool took 24 hours to fill the new disk - including the ZFS overheads. * Copying of one raw 250(232)Gb partition takes under 2 hours (if it can sustain about 70Mb/s reads from the source without distractions like other pool IO - then 1 hour). * Proper resilvering (reading all BP-tree from the original pool, reading all blocks from the TLVDEV, writing reconstructed(?) sectors to the target disk) from one partition to another took 17 hours. * Full scrubbing (reading all blocks from the pool, fixing checksum mismatches) takes 25-27 hours. * Selective scrubbing - unimplemented, timeframe unknown (reading all BP-tree from the original pool, reading all blocks from the TLVDEV including the target disk and the original disk, fixing checksum mismatches without panicky messages and/or hotspares kicking in). I *guess* it would have similar speed to a resilver, but less bound to random write IO patterns, which may be better for latencies of other tasks on the system. So, in case of original resilver, I replace the not-yet-dead disk with a hotspare, and after 17 hours of waiting I see if it was successfully resilvered or not. During this time the disk can die for example, leaving my pool with lowered protection (or lack thereof in case of raidz1 or two-way mirrors). In case of the new method proposed for a POC implementation, after 1 hour I'd already have a somewhat reliable copy of that vdev (a few blocks may have mismatches, but if the source disk dies or is taken away now - not the whole TLVDEV or pool is degraded and has compromised protection). Then after the same +17 hours for scrubs I'd be certain that this copy is good. If the new writes incoming to this TLVDEV between start of DD and end of scrub are directed to be written on both the source disk and its copy, then there are less (down to zero) checksum discrepancies that the scrub phase would find. First it was a theoretical speculation, but a couple of days later the incomplete resilver made me a practical experiment of the idea. Ok, then my made-up and dismissed argument does not stand ;) Thanks for the discussion, //Jim Klimov </pre> Jim Klimov 2012-05-24T13:06:48 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49226 <pre> On May 23, 2012, at 2:56 PM, Jim Klimov wrote: The FMA message is consistent with the man page. Efficiency allows use of denominators other than time. Speed is restricted to a denominator of time. There is no flame war here, look elsewhere. Operationally, your method loses every time. You have not made a case for why this hybrid and failure-prone procedure is required. What problem are you trying to solve? Why not follow the well-designed existing procedure? The failure data does not support your hypothesis. -- richard </pre> Richard Elling 2012-05-24T06:28:14 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49225 <pre> There were some pretty major fixes and new features added between snv_117 and snv_134 (the last OpenSolaris release). It might be worth updating to snv_134 at the very least. -B </pre> Brandon High 2012-05-24T06:12:24 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49224 <pre>Thanks again, 2012-05-24 1:01, Richard Elling wrote: Indeed, even in snv_117 the zpool man page says that. But the console/dmesg message was also quite clear, so go figure whom to trust (or fear) more ;) fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-GH, TYPE: Fault, VER: 1, SEVERITY: Major EVENT-TIME: Wed May 16 03:27:31 MSK 2012 PLATFORM: Sun Fire X4500, CSN: 0804AMT023 , HOSTNAME: thumper SOURCE: zfs-diagnosis, REV: 1.0 EVENT-ID: cc25a316-4018-4f13-c675-d1d84c6325c3 DESC: The number of checksum errors associated with a ZFS device exceeded acceptable levels. Refer to http://sun.com/msg/ZFS-8000-GH for more information. AUTO-RESPONSE: The device has been marked as degraded. An attempt will be made to activate a hot spare if available. IMPACT: Fault tolerance of the pool may be compromised. REC-ACTION: Run 'zpool status -x' and replace the bad device. Ummm... this is likely to start a flame war with other posters, and you did not say what efficiency is to you? How can we compare apples to meat, not even knowing whether the latter is a steak or a pork knee? I, for now, choose to stand by a statement that reduction of the timeframe that the old disk needs to be in the system is a good thing, as well as that changing the IO pattern from random writes into (mostly) sequential writes and after that random reads may be also somewhat more efficient, especially under other loads (interfering less with them). Even though the whole replacement process may take more wallclock time, there are cases when I'd likely trust it to do a better job than original resilvering. I think, someone with equipment could stage an experiment and compare the two procedures (existing and proposed) on a nearly full and somewhat fragmented pool. Maybe you can disenchant me (not with vague phrases but either theory or practice) and I would then see that my trust is blind, misdirected and without basement. =) That's why I proposed to tuck this scenario under the zfs hood (DD + selective scrub + ditto writes during the process, as an optional alternative to current resilver), or explain coherently why this should not be done - not for any situation. Implementing it as a standard supported command would be KISS ;) Especially if it is known that with some quirks this procedure works, and may be beneficial to some cases, i.e. by reducing the timeframe that a pool with a flaky disk in place is exposed to potential loss of redundancy and large amounts of data, and in the worst case the loss is constrained to those sectors which couldn't be (correctly) read by DD from the source disk and couldn't be reconstructed by raidz/mirror redundancies due to whatever overlaying problems (i.e. a sector from same block died on another disk too). In this case, mostly, the fright of having the device kicked out of the pool automatically instead of getting it "synced" ("resilvered" is an improper term here, I guess) to proper state. In general - since this is a part of some migration procedure which is, again, expected to have errors, we don't really care for signalling them. Why doesn't the original resilver signal several million CKSUM errors per new empty disk when it does reconstruction of sectors onto it? I'd say this is functionally identical. (At least, would be - if it were part of a supported procedure as I suggest). Thanks, //Jim Klimov PS: I pondered for a while if I should make up an argument that on a dying disk mechanics, lots of random IO (resilver) instead of sequential IO (DD) would cause it to die faster, but that's just a FUD not backed by any scientific data or statistics - which you likely have, and perhaps opposing this argument indeed. </pre> Jim Klimov 2012-05-23T21:56:16 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49223 <pre> no The man page is clear on this topic, IMHO DEGRADED One or more top-level vdevs is in the degraded state because one or more component devices are offline. Sufficient replicas exist to continue functioning. One or more component devices is in the degraded or faulted state, but sufficient replicas exist to continue functioning. The underlying condi- tions are as follows: o The number of checksum errors exceeds acceptable levels and the device is degraded as an indication that some- thing may be wrong. ZFS continues to use the device as necessary. speed != efficiency IMHO, this is too operationally complex for most folks. KISS wins. What is it about error counters that frightens you enough to want to clear them often? -- richard -- ZFS Performance and Training Richard.Elling&lt; at &gt;RichardElling.com +1-760-896-4422 _______________________________________________ zfs-discuss mailing list zfs-discuss&lt; at &gt;opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss </pre> Richard Elling 2012-05-23T21:01:29 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49222 <pre> Thank you Richard for taking notice of this thread and the definitive answers I needed not quote below, for further questions ;) Doesn't this status preclude the device with many CKSUM errors from participating in the pool (TLVDEV) and the remainder of the scrub in particular? At least the textual error message infers that if a hotspare were available for the pool, it would kick in and invalidate the device I am scrubbing to update into the pool after the DD-phase (well, it was not DD but a hung-up resilver in this case, but that is not substantial). Such automatic replacement is definitely not what I needed in this particular case, so if it were to happen - it would be a problem indeed, in and of itself. &gt; dd, or simular dumb block copiers, should work fine. &gt; However, they are inefficient... Define efficient? In terms of transferring the 900Gb payload of a 1Tb HDD used for ZFS for a year - DD would beat resilver anytime, in terms of getting most or (less likely) all of the valid bits with data onto the new device. It is the next phase (getting the rest of the bits into valid state) that needs some attention, manual or automated. Again, DD is not a good usecase indeed for pools with little data on big disks, and while I see why these could be used (i.e. to never face fragmentation), I haven't seen them in practice around here. &gt;... and operationally difficult to manage Actually, that's why I asked whether it makes sense to automate such a scenario as another legal variant of disk replacement, complete with fast data transfer and verification and simultaneous work of the new and old devices until the data migration is marked complete. In particular that would take care of accepting the scrub errors as an expected part of the disk replacement and not a fatal fault/degradation, and/or allowing new writes to propagate onto the new disk while the replacement is going on and minimize discrepancies right on the run. In visible effect this would be similar to current resilver during replacement of a live disk with a hotspare, but the prcess would follow a different scenario I suggested earlier in the thread. ... Understood, point taken, I won't try to promote such a "solution", and I agree that certainly it is not a good general idea indeed. It should be noted however (or I want to be corrected, please, if I am wrong), that: 1) Errors are expected on this run since the DD'ed copy is expected to deviate from current pool state; if the "degradation" mark of new disk would force it to be kicked out of the pool just because there are many CKSUM errors - which we know should be there due to manual DD-phase - then the reason is good IMHO (in this one case); 2) The progress is tracked by logging the error counts into a text file. If the admin fired up the script (manually in his terminal or a vnc/screen session), he can also look into the log file or even tail it. 3) The individual CKSUM errors are summed up in fmstat output, and this script does not zero them out, so even system-side tracking is not disturbed here. Anyhow, if there is a device with just a few CKSUM errors, then the next scrub clears its error counts anyway (if no new problems are found).... Thanks, //Jim Klimov </pre> Jim Klimov 2012-05-23T20:32:35 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49221 <pre>comments far below... On May 22, 2012, at 1:42 AM, Jim Klimov wrote: dd, or simular dumb block copiers, should work fine. However, they are inefficient and operationally difficult to manage, which is why they tend to fall in the prefer-to-use-something-else catagory. It shouldn't unless you did something to confuse it, such as having both the original and the dd copy online at the same time. In that case, you will have two different copies of the same identified device that are independent. This is an operational mistake, hence my comment above. Not needed. DEGRADED is the status. You clear degraded states by fixing the problem and running zpool clear. DEGRADED, in and of itself, is not a problem. I would never allow such scripts in my site. It is important to track the progress and state changes. This script resets those counters for no good reason. I post this comment in the hope that future searches will not encourage people to try such things. -- richard -- ZFS Performance and Training Richard.Elling&lt; at &gt;RichardElling.com +1-760-896-4422 _______________________________________________ zfs-discuss mailing list zfs-discuss&lt; at &gt;opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss </pre> Richard Elling 2012-05-23T16:54:14 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49220 <pre> I think this is at least in part an issue with older code. There have been various fixes for hangs/restarts/incomplete replaces and sparings over the time since. I've done it a couple of times at least: * a failed disk in a raidz1, where i didn't trust that the other disks didn't also have errors. Basically did a ddrescue from one disk to the new. I think these days, a 'replace' where the original disk is still online will use that content, like a hotspare replace, rather than assume it has gone away and must be recreated, but that wasn't the case at the time. * Where I had an iscsi mirror of a laptop hard disk, but it was out of date and had been detached when the laptop iscsi initiator refused to start. Later, the disk developed a few bad sectors. I made a new submirror, let it sync (with the error still), then blatted bits of the old image over the new in the areas where the bad sectors where being reported. Scrub again, and they were fixed (as well as some blocks on the new submirror repaired coming back up to date again). Pretty much as you did: decline to panic and restart scrubs. -- Dan._______________________________________________ zfs-discuss mailing list zfs-discuss&lt; at &gt;opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss </pre> Daniel Carosone 2012-05-23T00:00:31 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49219 <pre> I got into similar situation last night on that Thumper - it is now migrating a flaky source disk in the array from an original old 250Gb disk into a same-sized partition on the new 3Tb drive (as I outlined as IDEA7 in another thread). The source disk itself had about 300 CKSUM errors during the process, and for reasons beyond my current understanding, the resilver never completed. In zpool status it said that the process was done several hours before the time I looked at it, but the TLVDEV still had a "spare" component device comprised of the old disk and new partition, and the (same) hotspare device in the pool was "INUSE". After a while we just detached the old disk from the pool and ran scrub, which first found some 178 CKSUM errors on the new partition right away, and degraded the TLVDEV and pool. We cleared the errors, and ran the script below to log the detected errors and clear them, so the disk is fixed and not kicked out of the pool due to mismatches. Overall 1277 errors were logged and apparently fixed, and the pool is now on its second full scrub run - no bugs so far (knocking wood; certainly none this early in the scrub as we had last time). So in effect, this methodology works for two of us :) Since you did similar stuff already, I have a few questions: 1) How/what did you DD? The whole slice with the zfs vdev? Did the system complain (much) about the renaming of the device compared to paths embedded in pool/vdev headers? Did you do anything manually to remedy that (forcing import, DDing some handcrafted uberblocks, anything?) 2) How did you "treat errors as expected" during scrub? As I've discovered, there were hoops to jump through. Is there a switch to disable "degrading" of pools and TLVDEVs based on only the CKSUM counts? My raw hoop-jumping script: ----- #!/bin/bash # /root/scrubwatch.sh # Watches 'pond' scrub and resets errors to avoid auto-degrading # the device, but logs the detected error counts however. # See also "fmstat|grep zfs-diag" for precise counts. # See also https://blogs.oracle.com/bobn/entry/zfs_and_fma_two_great # for details on FMA and fmstat with zfs hotspares while true; do zpool status pond | gegrep -A4 -B3 'resilv|error|c1t2d|c5t6d|%' date echo "" C1="`zpool status pond | grep c1t2d`" C2="`echo "$C1" | grep 'c1t2d0s1 ONLINE 0 0 0'`" if [ x"$C2" = x ]; then echo "`date`: $C1" &gt;&gt; /var/tmp/zpool-clear_pond.log zpool clear pond zpool status pond | gegrep -A4 -B3 'resilv|error|c1t2d|c5t6d|%' date fi echo "" sleep 60 done HTH, //Jim Klimov </pre> Jim Klimov 2012-05-22T08:42:02 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49218 <pre>Thank you for reading and replying :-) 2012-05-22 6:18, Bob Friesenhahn wrote: For a not-full not-fragmented pool it is likely that a run of the original resilvering would be faster and known to be correct ;) Well, the point in this case was for a "selective scrub" as I called it in the text, which would indeed read all blocks and dittos, if their DVA lies on that VDEV where we replaced the disk and expect discrepancies. In this limitation it is like a resilver, but is indeed a scrub in effect. Besides, from what I've seen, a resilver just rewrites the given range of TXGs (like [0-current]) on the target disk based on reads from other disks in the TLVDEV and on the assumption that up to some TXG point (zero or above) that disk had valid data matching other disks in the array. Here, after the DD-stage, we have no guarantee that the source disk was fully valid (just a hope, augmented by lack of read errors and timeouts during the DD-phase), and to some degree we don't know if the writes that came in during the replacement process were properly written onto the target disk as well as its counterpart being replaced and still a (more) valid part of the pool. If we do that write-cloning and if the source disk was okay, the scrub shouldn't find any errors I think. //Jim Klimov </pre> Jim Klimov 2012-05-22T08:17:20 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49217 <pre> I've done basically this kind of thing before: dd a disk and then scrub rather than replace, treating errors as expected. zdb will show you usage per metaslab, you could use that and effectively select offset ranges to skip any empty ones. After a while, and once the pool has seen usage fill past low %'ages, I'd say most metaslabs would have some usage, so you might not save much time. Going to finer detail within a metaslab is not worthwhile - much more involved and involves the seeks you're trying to avoid. -- Dan._______________________________________________ zfs-discuss mailing list zfs-discuss&lt; at &gt;opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss </pre> Daniel Carosone 2012-05-22T03:30:02 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49216 <pre> I did read all of your text. :-) This is an interesting idea and could be of some use but it would be wise to test it first a few times before suggesting it as a general course. Zfs is still totally not foolproof. I still see postings from time to time regarding pools which panic/crash the system (probably due to memory corruption). Zfs will try to keep the data compacted at the beginning of the partition so if you have a way to know how far out it extends, then the initial 'dd' could be much faster when the pool is not close to full. Zfs scrub does need to do many more reads than a resilver since it reads all data and metadata copies. Triggering a resilver operation for the specific disk would likely hasten progress. Bob </pre> Bob Friesenhahn 2012-05-22T02:18:03 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49215 <pre>I hope there is some good outcome of this thread after all, below... I wonder if anyone else thinks the following proposal is reasonable? ;) 2012-05-18 10:18, Daniel Carosone wrote: As I detail below, i see Replace happening when a device is going to be gone - but is still available and is being proactively replaced. Well, I've gone to a swimming pool today to swim the halfmile and clear my head (metaphorically at least), and from the depths I emerged with another idea: From what I do see with the pool I'm upgrading (in another thread), there is also a "Replace" mode for hotspare devices, namely: * I attached the hotspare to the pool zpool add poolname spare c1t2d0 * I asked the pool to migrate a flaky disk's data to the new disk: zpool replace poolname c5t6d0 c1t2d0 * I asked the pool to forget the old disk so it can be removed: zpool detach poolname c5t6d0 (cfgadm, removal, pluck in the new disk, cfgadm, etc) From iostat I see that all existing TLVDEV's drives, including the one being replaced, are actively thrashed by reads for many hours, with some writes pouring onto the new disk. SO THE IDEA IS as follows: the disk being explicitly replaced, as in upgrades of the pool to larger drives, should first be copied onto new media "DD-style", which would be sequential IO for both devices, bandwidth-bound and rather fast. Then there should be a selective scrub, reading and checking allocated blocks from this TLVDEV only - like resilver does today - and repairing possible discrepancies (since the pool was likely live during the "DD stage", as well as errors were possible on the source drive as well as any other), and after this selective scrub the process is complete. BENEFITS: * The pool quickly gets a more-or-less good copy of the original disk, if it has not died completely and is able to serve reads for DD-style copying. This decreases the window of exposure of the TLVDEV to complete failure due to decreased redundancy, and can already help to salvage much of the data in case of partly bad source disk. That is, after the DD-style copy the new disk may be able to serve much of the valid data, and discrepancies might be easy to repair using normal checksum-mismatch modes - if the old disk kicks the bucket and/or is removed before the selective scrub is complete to gracefully finish the replacement procedure. The standard scrubbing approach after the DD-copy takes care of ensuring that by the end of the procedure the new disk's data is fully valid. This also allows to not bother about the problems of the source disk being updated in locations ahead or behind the point where we're reading now - some corrections to be made by the selective scrub are expected anyway. However, arguably, incoming writes may be placed on the source disk and its syncing-up spare replacement (into correct sector locations right from the start). * Instead of scheduling many random writes, which may be slower due to sync requirements, caching priorities, etc., we lean towards many random reads - which would still be used if we were using the original replace/resilver mode. Arguably, the reads can be optimized better by ZFS pipeline and HDD NCQ/TCQ, and in a safer manner than (random) write optimizations. * This method should be beneficial to raidz as well as mirrors, although the latter may have more options to cheaply recover bad sectors detected (as HDD IO errors) on source media of the one disk being replaced, on the fly - during DD-phase. CAVEATS: * This mode is of benefit for users whose pools are rather fragmented and full, so that sequential copy is noticeably faster than BP-tree-walk based resilvering. It is about 30x quicker on the utilized servers and homeNAS'es that I see. For example, on a Thumper in my other thread, resilvering of a 250Gb disk (partition) takes 15-17 hours while writing files and zfs-sends into a single-disk ZFS pool located on the same 3Tb drive fills it up in 24 hours. A full scrub of the original pool (45*250Gb) takes 24-27 hours. Time matters. The ZFS "marketing" states that it is quicker to repair because it only tests and copies the allocated data and not the whole disk as other RAID systems - well, this is only good as long as the pools are kept relatively empty. While I can agree with benefits of limiting the disk seeks via partitioning, i.e. by buying a 100Tb array and using only 10Tb by allocating smaller disk slices, I don't see a good reason to allocate the 100Tb array and consistently keep it used at 10Tb, sorry. Perhaps this mode with DD-style preamble should be triggered by a separate command-line request (by admin's discretion) or if it ever becomes a default option - it should be used instead of the original resilver-only method after some watermark value of disk utilization and/or known fragmentation. * If the original disk (being replaced) is a piece of faulty hardware, it can cause problems during the DD stage, such as: ** Lags - HDD retries on bad sectors can take a considerate amount of time based on firmware settings/capabilities. ** Loss of device visibility from the controller, reset storms, etc.; physical failure of original disk during the copy - these would lead to inability to continue reading the disk. ** Erroneous reads - returned garbage will be compensated by the following scrub after the DDing phase. If the DDing process detects that the average read speed has dropped to some unacceptable level or has stalled completely, it can try to seek from another original-disk location and/or fall back to original resilvering from other vdevs and abandon the DD phase. This does not mean that retries should be avoided, or that the first encountered error (even a connection error) should be the cause for aborting or restarting the DD-phase. It is not yet critical if some sectors were skipped during the DDing phase - the following selective scrub will (should) recover them, possibly by retrying the original disk as well, and maybe it has got to recover and relocate the bad sectors in the background by this time. Even if the DD-phase was aborted after a non-trivial amount of copying, the scrub/resilver should, IMHO, also read from the partially filled new disk and only rewrite those sectors that require rewriting (especially important for media that is sensitive to write-wearing). * Overall (wallclock) length of this replacement is likely to be higher than of the original method - since about the same amount of time will be needed for the scrub as for resilver, and some time will be added for DDing, and maybe plagued with retries etc. when hitting faulty sectors. However, a milestone of relative data safety will be reached a lot faster (if the source disk is substantially readable). * Errors (on the target disk) are expected during the selective scrub stage, and should be fixed quietly and not cause CKSUM error counter bumps nor other panicky clutter. This is so far a relatively raw idea and I've probably missed something. Do you think it is worth pursuing and asking some zfs developers to make a POC? ;) Thanks, //Jim Klimov </pre> Jim Klimov 2012-05-20T22:55:24 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49214 <pre>Yup, that's exactly what I did last night... zoned=off mountpoint=/some/place mount unmount mountpoint=legacy zoned=on Thanks! On May 20, 2012, at 3:09 AM, Jim Klimov wrote: </pre> Anil Jangity 2012-05-20T13:31:22 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49213 <pre> I've had such problems a few times on snv_117 and/or older, blocking recursive snapshots and the zfs-auto-snap service, and liveupgrade (OpenSolaris SXCE) among other things, but I thought they were fixed in later releases before the last ones like yours. Essentially the workaround was to mount the "offending" dataset and unmount it again if it is not needed now - this clears some internal ZFS flags and allows snapshots to proceed. This could be tricky with "zoned=on" datasets and to an extent those with a legacy mountpoint, but those problems can be worked around (i.e. disable "zoned", mount, umount, reenable "zoned"). Hope this helps, //Jim Klimov </pre> Jim Klimov 2012-05-20T10:09:30 http://permalink.gmane.org/gmane.os.solaris.opensolaris.zfs/49212 <pre>What causes these messages? cannot create snapshot 'zones/rani/ROOT/zbe-2&lt; at &gt;migration': dataset is busy There are zones living in zones pool, but none of the zones are running or mounted. root&lt; at &gt;:~# zfs get -r mounted,zoned,mountpoint zones/rani NAME PROPERTY VALUE SOURCE zones/rani mounted yes - zones/rani zoned off default zones/rani mountpoint /zones/rani default zones/rani/ROOT mounted no - zones/rani/ROOT zoned on local zones/rani/ROOT mountpoint legacy local zones/rani/ROOT&lt; at &gt;original mounted - - zones/rani/ROOT&lt; at &gt;original zoned - - zones/rani/ROOT&lt; at &gt;original mountpoint - - zones/rani/ROOT/zbe-2 mounted no - zones/rani/ROOT/zbe-2 zoned on local zones/rani/ROOT/zbe-2 mountpoint legacy inherited from zones/rani/ROOT root&lt; at &gt;:~# This is on a very old OS now snv_134b… I am working to upgrade them soon! </pre> Anil Jangity 2012-05-20T04:18:06 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.os.solaris.opensolaris.zfs