gmane.os.solaris.opensolaris.zfs

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-23T21:56:16

Thanks again,

2012-05-24 1:01, Richard Elling wrote:

Indeed, even in snv_117 the zpool man page says that. But the
console/dmesg message was also quite clear, so go figure whom
to trust (or fear) more ;)

fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-GH, TYPE: Fault, 
VER: 1, SEVERITY: Major
EVENT-TIME: Wed May 16 03:27:31 MSK 2012
PLATFORM: Sun Fire X4500, CSN: 0804AMT023            , HOSTNAME: thumper
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: cc25a316-4018-4f13-c675-d1d84c6325c3
DESC: The number of checksum errors associated with a ZFS device
exceeded acceptable levels.  Refer to http://sun.com/msg/ZFS-8000-GH for 
more information.
AUTO-RESPONSE: The device has been marked as degraded.  An attempt
will be made to activate a hot spare if available.
IMPACT: Fault tolerance of the pool may be compromised.
REC-ACTION: Run 'zpool status -x' and replace the bad device.




Ummm... this is likely to start a flame war with other posters,
and you did not say what efficiency is to you? How can we compare
ap

Re: How does resilver/scrub work?

Richard Elling — 2012-05-23T21:01:29


no


The man page is clear on this topic, IMHO

     DEGRADED
                 One or more top-level vdevs is in  the  degraded
                 state  because one or more component devices are
                 offline. Sufficient replicas exist  to  continue
                 functioning.

                 One or more component devices is in the degraded
                 or  faulted state, but sufficient replicas exist
                 to continue functioning. The  underlying  condi-
                 tions are as follows:

                     o    The number of checksum  errors  exceeds
                          acceptable  levels  and  the  device is
                          degraded as an  indication  that  some-
                          thing  may  be  wrong. ZFS continues to
                          use the device as necessary.



speed != efficiency


IMHO, this is too operationally complex for most folks. KISS wins.



What is it about error counters that frightens you enough to want to clear 
th

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-23T20:32:35

Thank you Richard for taking notice of this thread and the
definitive answers I needed not quote below, for further
questions ;)


Doesn't this status preclude the device with many CKSUM errors
from participating in the pool (TLVDEV) and the remainder of
the scrub in particular?

At least the textual error message infers that if a hotspare
were available for the pool, it would kick in and invalidate
the device I am scrubbing to update into the pool after the
DD-phase (well, it was not DD but a hung-up resilver in this
case, but that is not substantial).

Such automatic replacement is definitely not what I needed
in this particular case, so if it were to happen - it would
be a problem indeed, in and of itself.

 > dd, or simular dumb block copiers, should work fine.
 > However, they are inefficient...

Define efficient? In terms of transferring the 900Gb payload
of a 1Tb HDD used for ZFS for a year - DD would beat resilver
anytime, in terms of getting most or (less likely) all of the
valid bits with data ont

Re: How does resilver/scrub work?

Richard Elling — 2012-05-23T16:54:14

comments far below...

On May 22, 2012, at 1:42 AM, Jim Klimov wrote:


dd, or simular dumb block copiers, should work fine. However, they 
are inefficient and operationally difficult to manage, which is why they
tend to fall in the prefer-to-use-something-else catagory.


It shouldn't unless you did something to confuse it, such as having both
the original and the dd copy online at the same time. In that case, you
will have two different copies of the same identified device that are
independent. This is an operational mistake, hence my comment above.


Not needed.


DEGRADED is the status. You clear degraded states by fixing the problem
and running zpool clear. DEGRADED, in and of itself, is not a problem.


I would never allow such scripts in my site. It is important to track the 
progress and state changes. This script resets those counters for no
good reason.

I post this comment in the hope that future searches will not encourage 
people to try such things.
 -- richard

--
ZFS Performance and Training
R

Re: How does resilver/scrub work?

Daniel Carosone — 2012-05-23T00:00:31

I think this is at least in part an issue with older code.  There have
been various fixes for hangs/restarts/incomplete replaces and sparings
over the time since.  


I've done it a couple of times at least:

 * a failed disk in a raidz1, where i didn't trust that the other
   disks didn't also have errors.  Basically did a ddrescue from one
   disk to the new. I think these days, a 'replace' where the
   original disk is still online will use that content, like a
   hotspare replace, rather than assume it has gone away and must be
   recreated, but that wasn't the case at the time.

 * Where I had an iscsi mirror of a laptop hard disk, but it was out
   of date and had been detached when the laptop iscsi initiator
   refused to start.  Later, the disk developed a few bad sectors.  I
   made a new submirror, let it sync (with the error still), then
   blatted bits of the old image over the new in the areas where the
   bad sectors where being reported.  Scrub again, and they were fixed
   (as well as some b

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-22T08:42:02

I got into similar situation last night on that Thumper -
it is now migrating a flaky source disk in the array from
an original old 250Gb disk into a same-sized partition on
the new 3Tb drive (as I outlined as IDEA7 in another thread).
The source disk itself had about 300 CKSUM errors during
the process, and for reasons beyond my current understanding,
the resilver never completed.

In zpool status it said that the process was done several
hours before the time I looked at it, but the TLVDEV still
had a "spare" component device comprised of the old disk
and new partition, and the (same) hotspare device in the
pool was "INUSE".

After a while we just detached the old disk from the pool
and ran scrub, which first found some 178 CKSUM errors on
the new partition right away, and degraded the TLVDEV and
pool.

We cleared the errors, and ran the script below to log
the detected errors and clear them, so the disk is fixed
and not kicked out of the pool due to mismatches.
Overall 1277 errors were logged and apparen

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-22T08:17:20

Thank you for reading and replying :-)

2012-05-22 6:18, Bob Friesenhahn wrote:

For a not-full not-fragmented pool it is likely that a run
of the original resilvering would be faster and known to
be correct ;)


Well, the point in this case was for a "selective scrub" as
I called it in the text, which would indeed read all blocks
and dittos, if their DVA lies on that VDEV where we replaced
the disk and expect discrepancies. In this limitation it is
like a resilver, but is indeed a scrub in effect.

Besides, from what I've seen, a resilver just rewrites the
given range of TXGs (like [0-current]) on the target disk
based on reads from other disks in the TLVDEV and on the
assumption that up to some TXG point (zero or above) that
disk had valid data matching other disks in the array.

Here, after the DD-stage, we have no guarantee that the
source disk was fully valid (just a hope, augmented by
lack of read errors and timeouts during the DD-phase),
and to some degree we don't know if the writes that came
in duri

Re: How does resilver/scrub work?

Daniel Carosone — 2012-05-22T03:30:02

I've done basically this kind of thing before: dd a disk and then
scrub rather than replace, treating errors as expected. 


zdb will show you usage per metaslab, you could use that and
effectively select offset ranges to skip any empty ones.  After a
while, and once the pool has seen usage fill past low %'ages, I'd say
most metaslabs would have some usage, so you might not save much
time.  Going to finer detail within a metaslab is not worthwhile -
much more involved and involves the seeks you're trying to avoid.

--
Dan._______________________________________________
zfs-discuss mailing list
zfs-discuss< at >opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: How does resilver/scrub work?

Bob Friesenhahn — 2012-05-22T02:18:03

I did read all of your text. :-)

This is an interesting idea and could be of some use but it would be 
wise to test it first a few times before suggesting it as a general 
course.  Zfs is still totally not foolproof.  I still see postings 
from time to time regarding pools which panic/crash the system 
(probably due to memory corruption).

Zfs will try to keep the data compacted at the beginning of the 
partition so if you have a way to know how far out it extends, then 
the initial 'dd' could be much faster when the pool is not close to 
full.

Zfs scrub does need to do many more reads than a resilver since it 
reads all data and metadata copies.  Triggering a resilver operation 
for the specific disk would likely hasten progress.

Bob

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-20T22:55:24

I hope there is some good outcome of this thread after all, below...
I wonder if anyone else thinks the following proposal is reasonable? ;)

2012-05-18 10:18, Daniel Carosone wrote:

As I detail below, i see Replace happening when a device
is going to be gone - but is still available and is being
proactively replaced.


Well, I've gone to a swimming pool today to swim the halfmile
and clear my head (metaphorically at least), and from the
depths I emerged with another idea:

 From what I do see with the pool I'm upgrading (in another
thread), there is also a "Replace" mode for hotspare devices,
namely:
* I attached the hotspare to the pool
   zpool add poolname spare c1t2d0
* I asked the pool to migrate a flaky disk's data to the new disk:
   zpool replace poolname c5t6d0 c1t2d0
* I asked the pool to forget the old disk so it can be removed:
   zpool detach poolname c5t6d0
   (cfgadm, removal, pluck in the new disk, cfgadm, etc)

 From iostat I see that all existing TLVDEV's drives, including
the one being r

Re: dataset is busy when doing snapshot

Anil Jangity — 2012-05-20T13:31:22

Yup, that's exactly what I did last night...

zoned=off
mountpoint=/some/place
mount
unmount
mountpoint=legacy
zoned=on

Thanks!

On May 20, 2012, at 3:09 AM, Jim Klimov wrote:

Re: dataset is busy when doing snapshot

Jim Klimov — 2012-05-20T10:09:30

I've had such problems a few times on snv_117 and/or older,
blocking recursive snapshots and the zfs-auto-snap service,
and liveupgrade (OpenSolaris SXCE) among other things, but
I thought they were fixed in later releases before the last
ones like yours.

Essentially the workaround was to mount the "offending"
dataset and unmount it again if it is not needed now -
this clears some internal ZFS flags and allows snapshots
to proceed. This could be tricky with "zoned=on" datasets
and to an extent those with a legacy mountpoint, but
those problems can be worked around (i.e. disable "zoned",
mount, umount, reenable "zoned").

Hope this helps,
//Jim Klimov

dataset is busy when doing snapshot

Anil Jangity — 2012-05-20T04:18:06

What causes these messages?

cannot create snapshot 'zones/rani/ROOT/zbe-2< at >migration': dataset is busy


There are zones living in zones pool, but none of the zones are running or mounted.


root< at >:~# zfs get -r mounted,zoned,mountpoint zones/rani
NAME                      PROPERTY    VALUE        SOURCE
zones/rani                mounted     yes          -
zones/rani                zoned       off          default
zones/rani                mountpoint  /zones/rani  default
zones/rani/ROOT           mounted     no           -
zones/rani/ROOT           zoned       on           local
zones/rani/ROOT           mountpoint  legacy       local
zones/rani/ROOT< at >original  mounted     -            -
zones/rani/ROOT< at >original  zoned       -            -
zones/rani/ROOT< at >original  mountpoint  -            -
zones/rani/ROOT/zbe-2     mounted     no           -
zones/rani/ROOT/zbe-2     zoned       on           local
zones/rani/ROOT/zbe-2     mountpoint  legacy       inherited from zones/rani/ROOT
root< at >:~# 

This is on a very

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-18T20:10:41

My memory fails me here for a precise answer... I think that
the on-disk data within a raidzN top-level VDEV (mirrors are
trivial) is laid out as follows, for an arbitrary 6-disk set
of raidz2 TLVDEV:

D1   D2   D3   D4   D5   D6
Ar1  Ar2  Ad1  Ad2  Ad3  Ad4
Br1  Br2  Bd1  Cr1  Cr2  Cd1
Cd2  Cd3  Cd4  Cr3  Cr4  Cd5
Cd6  Dr1  Dr2  Dd1  ...

In these examples above, several blocks are laid out in sectors
of different disks, including the redundancy blocks. Sequential
accesses on one disk progress in a column from top to bottom.
Accesses in a row are parallelized between many disks.

The "A" block userdata is 4 sectors long, with 2 redundancy blocks.
The "B" block has just one userdata sector, and the "C" block has
6 userdata sectors with a redundancy started for each 4 sectors.

AFAIK each ZFS block fully resides within one TLVDEV (and ditto
copies have their own separate life in another TLVDEV if available),
and striping over several TLVDEVs occurs at a whole-block level.
This, in particular, allows disbalan

Re: How does resilver/scrub work?

Edward Ned Harvey — 2012-05-18T15:08:58

I'm reading the ZFS on-disk spec, and I get the idea that there's an
uberblock pointing to a self-balancing tree (some say b-tree, some say
avl-tree, some say nv-tree), where data is only contained in the nodes.  But
I haven't found one particular important detail yet:

On which values does the balancing tree balance?  Is it balancing on the
logical block address?  This would make sense, as an application requests to
read/write some logical block, making it easy and fast to find the
corresponding physical blocks...

If that is the case, wouldn't scrub/resilver need to work according to
logical block order?  (Which would also be random-ish, but decidedly NOT the
same as TXG temporal order.)

Re: How does resilver/scrub work?

Jim Klimov — 2012-05-18T15:04:37

First of all, thank you Daniel for taking the time to post a
lengthy reply! I do not get that kind of high-quality feedback
very often :)

I hope the community and googlers would benefit from that
conversation sometime. I did straighten out some thoughts
and (mis-)understandings, at least, more on that below :)

2012-05-18 15:30, Daniel Carosone wrote:
 >>    While waiting for that resilver to complete last week,
 >> I caught myself wondering how the resilvers (are supposed
 >> to) work in ZFS?
 > The devil finds work for idle hands... :-)

Or rather, brains ;)

 > Well, I'm not that - certainly not on the code.  It would probably be
 > best (for both of us) to spend idle time looking at the code, before
 > spending too much on speculation. Nonetheless, let's have at it! :)

Good idea any day, but rather lengthy in time. I have looked at the
code, at blogs, at mailing list archives, at the aged ZFS spec, for
about a year on-and-off now, and as you could see - understanding
remains imperfect ;)

Besides, tur

Re: How does resilver/scrub work?

Edward Ned Harvey — 2012-05-18T13:58:38

I'm reading the ZFS on-disk spec, and I get the idea that there's an uberblock pointing to a self-balancing tree (some say b-tree, some say avl-tree, some say nv-tree), where data is only contained in the nodes.  But I haven't found one particular important detail yet:

On which values does the balancing tree balance?  Is it balancing on the logical block address?  This would make sense, as an application requests to read/write some logical block, making it easy and fast to find the corresponding physical blocks...

If that is the case, wouldn't scrub/resilver need to work according to logical block order?  (Which would also be random-ish, but decidedly NOT the same as TXG temporal order.)

Zpool import FAULTED

Mohamed Magdy — 2012-05-18T02:28:43

Dears,

I need help to get my pool online again.


root< at >node1 # zdb -l  /dev/dsk/c2t1d15****

--------------------------------------------****

LABEL 0****

--------------------------------------------****

failed to unpack label 0****

--------------------------------------------****

LABEL 1****

--------------------------------------------****

failed to unpack label 1****

--------------------------------------------****

LABEL 2****

--------------------------------------------****

failed to unpack label 2****

--------------------------------------------****

LABEL 3****

--------------------------------------------****

failed to unpack label 3****

root< at >node1 # zdb -l  /dev/dsk/c4t2d19****

--------------------------------------------****

LABEL 0****

--------------------------------------------****

failed to unpack label 0****

--------------------------------------------****

LABEL 1****

--------------------------------------------****

failed to unpack label 1****

----------------------------

Re: How does resilver/scrub work?

Daniel Carosone — 2012-05-18T11:30:03

Given the latter point, I'm going to guess depth-first.  Yes, I should
look at the code instead of posting speculation. 

--
Dan.
_______________________________________________
zfs-discuss mailing list
zfs-discuss< at >opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: How does resilver/scrub work?

Daniel Carosone — 2012-05-18T06:18:12

The devil finds work for idle hands... :-)


Well, I'm not that - certainly not on the code.  It would probably be
best (for both of us) to spend idle time looking at the code, before
spending too much on speculation. Nonetheless, let's have at it! :)


The tradeoff will be code complexity and resulting fragility. Choose
wisely what you wish for.


Less likely, that's pretty much always going to have to go in txg
order.


No. Scrub (and any other repair, such as for errors found in the
course of normal reads) rewrite the reconstructed blocks in-place: to
the original DVA as referenced by its parents in the BP tree, even if
the device underneath that DVA is actually a new disk.

There is no COW. This is not a rewrite, and there is no original data
to preserve, this is a repair: making the disk sector contain what the
rest of the filesystem tree 'expects' it to contain. More specifically,
making it contain data that checksums to the value that block pointers
elsewhere say it should, via reconstruction using r

Re: Zpool import FAULTED

Richard Elling — 2012-05-18T04:32:15

On May 17, 2012, at 7:34 PM, Mohamed Magdy wrote:


In Solaris, /dev/dsk/c2t1d15 might not be the same as the slice
/dev/dsk/c2t1d15s0. Try with the slice.

If you cannot solve this problem, or find another path to the disk, then
you cannot import the pool.
 -- richard


--
ZFS Performance and Training
Richard.Elling< at >RichardElling.com
+1-760-896-4422







_______________________________________________
zfs-discuss mailing list
zfs-discuss< at >opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss