gmane.comp.linux.drbd

Re: re source Primary but inconsistent

envisionrx — 2012-05-25T15:53:08

I think I found the answer to my question in another forum post:
http://old.nabble.com/General-question-about-DRBD-and-HA-to33737923.html

Seems that drbd can handle this situation, impressive.. very impressive.. :)

Re: Rescue after reduce :(

Christian Völker — 2012-05-25T12:56:27

Hi,

I definitely performed the fsck on the drbd device, not on the lvm and
not on the physical one.

Meanwhile I was able to recover by extending the LV again to the old
size. Had to re-create the metadata on the secondary and it's syncing
now. fsck is running again,  Well, weekend is safe :D

But how do I resize then properly?

This is what I did:

-fsck -f /dev/drbd0
-resize2fs --new size 1400G
-drbdadm -- --new-size 1450G resize
-fsck -f /dev/drbd0
-lvreduce /dev/vg/drbd on primary
 -lvreduce /dev/vg/drbd on secondary
*BANG* :-(

Any hints?

Christian

Re: Rescue after reduce :(

Florian Haas — 2012-05-25T09:36:45

Definitely, but the question was whether fsck would throw errors just
from being run against a DRBD backing device. Which it wouldn't.

Florian

Re: Rescue after reduce :(

Felix Frank — 2012-05-25T09:08:04

Uhm, but then DRBD won't know about the fixes that fsck has applied,
right? Isn't this setting yourself up for inconsistent disks among your
nodes?

One would probably want to do a verify after such stunts anyway, I presume.

Cheers,
Felix

Re: Rescue after reduce :(

Florian Haas — 2012-05-25T09:03:39

Only if the filesystem had errors. :) You can always pretty much do
anything with DRBD's backing device if DRBD isn't using it.


"Pretty sure" doesn't help that much. Check your shell history.


Logs will tell. That's why I asked for them.

Cheers,
Florian

Re: Moving a DRBD cluster from physical machines toVMWaremachines

Pascal BERTON — 2012-05-25T08:30:14

Keith,

I'm not sure it's a good idea, however the first thing to avoid is to
present the replicated disk space to your virtualized peers as vDisks on a
VMFS. Use RDMs instead, in physical mode if possible, to grab some more
perfs.
Then for this RDM, use a vdisk node that will cause the instantiation of a
new vSCSI adapter (not the 0:1 default, but rather 1:0 for instance). It
will probably create an LSI Logic Parallel virtual adapter, change it to the
vmware paravirtual scsi driver, it will help a lot too.
Then, not sure it has a big impact on DRBD but VMware can generate IOs up to
32MB. It's a known perf issue for midrange storage arrays such as HP EVAs
(and even EMC Clariions I guess) causing big slowdowns due to IO
segmentation handling by the storage processors, we use to reduce it to 64k
or 128k instead. Again, I'm not sure it would have an impact on DRBD, may be
the gurus can give their advices about that ? Would it even make sense to
decrease it to 4K since DRBD chuncks are 4K large ? Or is it anothe

Re: Rescue after reduce :(

Christian Völker — 2012-05-25T07:37:51

Hi

Yes, on the /dev/drbd0. Otherwise fsck would have thrown errors, right?


Yes. Pretty sure about it. 
Even if not it shouldn't have done any harm as the filesystem was at this stage at 1400GB and the drbd0 device at 1450GB. Correct?


I will post them this evening. 

Well, then what could be a reason for both being diskless?


Thanls

Christian

Re: Rescue after reduce :(

Florian Haas — 2012-05-25T07:19:55

fsck of your /dev/drbdX device, I hope?


Again, did you run this on your /dev/drbdX and not on your backing device?


Kernel logs?


If you ran your fsck on your DRBD like you would be expected to, then
that shouldn't be a problem. The DRBD device would already have the
smaller size as per "drbdadm resize".

Cheers,
Florian

Re: leaked indexing text in users guide

Philipp Marek — 2012-05-25T07:16:02

Hello Paul,

... 
thanks for telling, I've fixed that.


Regards,

Phil

Re: Changing the name of a resource

Felix Frank — 2012-05-25T06:58:14

Hi,

On 05/23/2012 11:50 PM, John Anthony wrote:

I don't see how this is being problematic in any way. Resource names
only matter to the userland side of things. You need not adjust anything
at all - just update the drbd configs on both nodes. Use
drbd-overview.pl to check that the nodes are happy.

Why your node tears down the connection, I'm not sure. Do the peer's
logs reveal anything else?

Cheers,
Felix

Rescue after reduce :(

Christian Völker — 2012-05-25T06:32:02

Hi,

I've a drbd device (8.3) on both sides on aLVM volume.

I tried to reduce the device now. Steps I did:
- fsck -f (ext3)
- reduced filesystem to 1,400G
- drbdadm -- --new-size=1450G resize
- lvreduce drbdvol -L 1500G
- fsck -f 
- lvreduce on secondary

So far everything went fine. 

After mounting the device I got a corrupted filesystem  with i/o errors and both nodes are on "diskless".

I assume after the first lvreduce the primary was already on diskless but performed the fsck pn the secondarys disk (which was still on the larger old size)
Any clue how to recover?

And any hints how to perform the reduce properly?




Christian

leaked indexing text in users guide

Paul Theodoropoulos — 2012-05-24T20:16:27

http://www.drbd.org/users-guide/s-nested-lvm.html

    In order to enable this configuration, follow these steps:

      *

        Set an appropriate |filter| option in your |/etc/lvm/lvm.conf|:

        indexterm:[LVM]indexterm:[filter expression (LVM)]

    filter = ["a|sd.*|", "a|drbd.*|", "r|.*|"]


Just fyi.

Re: "PingAck not received" messages

Matthew Bloch — 2012-05-25T00:10:34

I might make that change if it happens again - sortof annoying to lose a
difficult bug as I'm pinning it down but I'll take the quieter life for
now :)

Re: Recovering from erroneous sync state

Zev Weiss — 2012-05-24T20:03:03

On May 23, 2012, at 4:48 PM, Lars Ellenberg wrote:


One would think, yes...but that's all it shows.  (No idea why.)


Ah, hadn't thought to reject outgoing packets as well -- and doing so seems to have finally had the intended effect (got both sides into StandAlone, then able to force one back to primary and reattach).  All back to normal now.

Thanks!

Zev

re source Primary but inconsistent

envisionrx — 2012-05-24T17:05:57

I was creating a new resource on our servers and ended up in an interesting
state and I'm wondering if DRBD (8.3.12) just handles this correctly, or if
this is bad.

I created a stacked resource on server1, using steps like this:
1) created an lvm to back the resource lvcreate -n store1 -L 2048g backingvg
2) added the resource configuration to /etc/drbd.d directory for both the
lower and upper resources and copied the config to the other nodes.
3) ran drbdadm create-md on the lower resource: drbdadm create-md
store1_lower
4) brought up the lower device: drbdadm up store1_lower
5) made the resource primary: drbdsetup /dev/drbd1 primary -o
6) ran drbdadm create-md on the upper resource: drbdadm --stacked create-md
store1
7) brought up the upper resource: drbdadm up --stacked store1
8) made the upper resource primary: drbdsetup /dev/drbd11 primary -o
9) formatted the upper resource: mkfs.ext4 /dev/drbd11

It should be noted that all my other resources are primary on the other node
(server2), I thought it would

harddisk problems

Marcel Kraan — 2012-05-24T16:22:59

i have so much problems with my (virtual) disks.
i see articles from virtual box

ext4 and kvm going error?
i need to set the Host IO ?

But i don't understand?
Is there a solution?

2 x 
quad core super micro.
32GB 
RAID5 6TB (4TB)



Buffer I/O error on device sdc1, logical block 65273728
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273729
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273730
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273731
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273732
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273733
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273734
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 65273735
lost page write due to I/O error on sdc1
Buffer I/O

Re: DRBD initial settings for two disks

Tero Mäntyvaara — 2012-05-24T13:32:19

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 24.5.2012 15:23, Florian Haas wrote:
in the example titled "Multi-volume DRBD resource configuration".
So if I don't want to or can't install DRBD 8.4 could I use two
resources to get around this multi-volume limitation in 8.3:

/etc/drbd.conf|:
resource r0 {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        }
        net {
                cram-hmac-alg sha1;
                shared-secret "secret";
        }
        on drbd01 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 192.168.0.1:7788;
                meta-disk internal;
        }
        on drbd02 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 192.168.0.2:7788;
                meta-disk internal;
        }
}

resource r1 {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        }
        ne

Re: "PingAck not received" messages

Florian Haas — 2012-05-24T13:32:46

I've seen the "I/O frozen for no reason" problem which seems to be an
upstream XFS issue, which Debian is hardly to blame for. The others I
personally haven't encountered. Just for clarification, what seemed
odd to me was not that you would update off the Debian stock squeeze
kernel, but that you'd consider pulling a CentOS kernel, of ostensibly
thejsame kernel version, into a Debian system. I'd just go to the
current Debian backports kernel. But we're going off topic. :)


Indeed.


It would still be interesting to find out whether you ever saw these
random disconnects with protocol C, or whether this appears to be a
B-only issue.

Cheers,
Florian

Re: "PingAck not received" messages

Matthew Bloch — 2012-05-24T13:09:34

No indeed, nothing external that we could detect after hours of layer 2 
tracing, and no messages that would indicate a malfunction on either of 
the hosts.  But this network problem was only visible via DRBD's 
messages, and if it's gone it's hard to reason about it any further (not 
that I miss it).  As I said I couldn't see any symptoms via ICMP or 
TCP-based tests between the hosts.


Then you've not hit the "scheduler divide by zero" bug or the "I/O 
frozen for 120s for no reason" bug or the "CPU#x stuck for 9999999s" 
bug?  These are all things that are filed vaguely on the Redhat bug 
trackers, as far as I know, and usually closed a few kernel versions 
later with "well I haven't seen it for a few kernel versions so it's 
probably OK"!

These are relatively rare bugs, except for some of our customers, when 
they're not at all rare and we haul them up to e.g. whatever wheezy has. 
  Except in this case they broke the briding code in 3.2.0 which is 
going to cause a virtualising customer some problems

Re: "PingAck not received" messages

Florian Haas — 2012-05-24T12:55:10

On Thu, May 24, 2012 at 1:53 PM, Matthew Bloch <matthew-WkKPyC9aYE69FHfhHBbuYA< at >public.gmane.org>
wrote:

Sure, if you have kernel-induced network problems on one of your
nodes, that would definitely explain the issues you're seeing. But you
insisted from the start that there were no network issues. :)


Might as well not be a DRBD bug at all, just DRBD trying to do the
right thing in the face of a flaky network stack.


That would seem like an odd thing to do. FWIW, we've been running
happily on squeeze kernels for months.


You can always to that from a device with metadata as well. kpartx is
your friend.


That's a fair point, but realistically, how long does it take you to
take the backup off your snapshot? And does this normally coincide
with the DRBD device getting hammered, which is pretty much the only
situation in which a downstream client would likely feel any
disruption?

Just my two cents. Or pence. :)

Cheers,
Florian

Re: DRBD initial settings for two disks

Florian Haas — 2012-05-24T12:23:32

That looks fine; take a look at
http://www.drbd.org/users-guide/s-configure-resource.html#s-drbdconf-example
in the example titled "Multi-volume DRBD resource configuration".

Do recall, however, that multiple volumes per resource are a DRBD 8.4
feature. So if you're actually trying to do this on Ubuntu lucid,
you'll have to make sure you install 8.4 before attempting this.

Cheers,
Florian