gmane.linux.redhat.cluster

Re: rgmanager is jamed

Fabio M. Di Nitto — 2012-05-26T07:05:37

This looks like a kernel dlm problem. I can see you found a workaround,
but that should not be necessary since upgrades between releases should
work.

can you please file a ticket with GSS and escalate it? Might be a good
idea to grab sosreports before those logs are flushed away in rotate.

Thanks
Fabio

Re: Where to get CLVM

Digimer — 2012-05-26T03:53:05

CentOS is a binary compatible, community released version of Red Hat. So
whatever is available in Red Hat is available in CentOS, including clvmd.

This tutorial covers, among other things, how to install and configure
Clustered LVM. Be sure to configure fencing as without it, clvmd will
hang (by design) the first time a node fails and can't be fenced.

https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial

Re: Where to get CLVM

Rajagopal Swaminathan — 2012-05-26T03:41:29

Greetings,

On Fri, May 25, 2012 at 9:57 PM, Chen, Ming Ming <ming-ming.chen< at >hp.com> wrote:

The cluster package should include CLVM.

Re: Where can I get clvm

Chen, Ming Ming — 2012-05-25T22:05:44

When I install the CentOS6.2, I picked the High Availability add-ons, and I cannot find the lvm2-cluster package? Please advice.

Thanks

Ming



From: linux-cluster-bounces< at >redhat.com [mailto:linux-cluster-bounces< at >redhat.com] On Behalf Of emmanuel segura
Sent: Friday, May 25, 2012 9:32 AM
To: linux clustering
Subject: Re: [Linux-cluster] Where can I get clvm



lvm2-cluster

2012/5/25 Chen, Ming Ming <ming-ming.chen< at >hp.com<mailto:ming-ming.chen< at >hp.com>>

I'm going to install and configure a CentOS 6.2 cluster. I need  CLVM  Is CLVM  included in  CentOS 6.2? If so, which package should I pick to install? If not, where can I get it?

Thanks

Ming








--
Linux-cluster mailing list
Linux-cluster< at >redhat.com<mailto:Linux-cluster< at >redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster




--
esta es mi vida e me la vivo hasta que dios quiera

Re: rgmanager is jamed

Nicolas Ross — 2012-05-25T18:16:11

Nicolas Ross a écrit :
(...)

I disabled cluster services with chkconfig, rebooter the whole 8 
servers, updated all, rebooted again, started cman and others, and then 
chkconfig'd back the cluster services.

All is working fine now.

Re: Where can I get clvm

Chen, Ming Ming — 2012-05-25T17:53:31

Thanks.

Ming





From: linux-cluster-bounces< at >redhat.com [mailto:linux-cluster-bounces< at >redhat.com] On Behalf Of emmanuel segura
Sent: Friday, May 25, 2012 9:32 AM
To: linux clustering
Subject: Re: [Linux-cluster] Where can I get clvm



lvm2-cluster

2012/5/25 Chen, Ming Ming <ming-ming.chen< at >hp.com<mailto:ming-ming.chen< at >hp.com>>

I'm going to install and configure a CentOS 6.2 cluster. I need  CLVM  Is CLVM  included in  CentOS 6.2? If so, which package should I pick to install? If not, where can I get it?

Thanks

Ming








--
Linux-cluster mailing list
Linux-cluster< at >redhat.com<mailto:Linux-cluster< at >redhat.com>
https://www.redhat.com/mailman/listinfo/linux-cluster




--
esta es mi vida e me la vivo hasta que dios quiera

Re: Where can I get clvm

emmanuel segura — 2012-05-25T16:31:48

lvm2-cluster

2012/5/25 Chen, Ming Ming <ming-ming.chen< at >hp.com>

Where to get CLVM

Chen, Ming Ming — 2012-05-25T16:27:53

I'm going to install and configure a CentOS 6.2 cluster. I need  CLVM  Is CLVM  included in  CentOS 6.2? If so, which package should I pick to install? If not, where can I get it?
Thanks
Ming

Where can I get clvm

Chen, Ming Ming — 2012-05-25T16:21:51

I'm going to install and configure a CentOS 6.2 cluster. I need  CLVM  Is CLVM  included in  CentOS 6.2? If so, which package should I pick to install? If not, where can I get it?

Thanks

Ming

rgmanager is jamed

Nicolas Ross — 2012-05-25T16:20:43

I am in the process of upgrading one of our cluster from RHEL 6.1 to 
6.2. It's an 8-node cluster.

I started with one node. Stop all cluster resources, cman, rgmanager et 
al. yum update, reboot, move to next. The first one did ok.

On the second one, rgmanager started, but doesn't seem to connect to 
other nodes. I found this in dmesg :

INFO: task rgmanager:2901 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rgmanager     D 0000000000000000     0  2901   2900 0x00000080
  ffff880667299d48 0000000000000082 0000000000000000 ffff8806656aa318
  ffff88066729c378 0000000000000001 ffff880665bb31b0 00007fffc6c6fa20
  ffff88066635a678 ffff880667299fd8 000000000000f4e8 ffff88066635a678
Call Trace:
  [<ffffffff814ee6fe>] __mutex_lock_slowpath+0x13e/0x180
  [<ffffffff814ee59b>] mutex_lock+0x2b/0x50
  [<ffffffffa02c192c>] dlm_new_lockspace+0x3c/0xa30 [dlm]
  [<ffffffff8115f74c>] ? __kmalloc+0x20c/0x220
  [<ffffffffa02ca94d>] device_write+0x30d/0x7d0 [dlm]

Is it possible to use quorum for CTDB to prevent split-brain and removing lockfile in the cluster file system

黃曉偉 — 2012-05-24T04:22:20

Hello list,

We know that CTDB uses lockfile in the cluster file system to prevent
split-brain.
It is a really good design when all nodes in the cluster can mount the
cluster file system (e.g. GPFS/GFS/GlusterFS) and CTDB can work happily in
this assumption.
However, when split-brain happens, the disconnected private network
violates this assumption usually.
For example, we have four nodes (A, B, C, D) in the cluster and GlusterFS
is the beckend.
GlusterFS and CTDB on all nodes communicate to each other via private
network and CTDB manages the public network.
If node A is disconnected in the private network, there will be group (A)
and group (B,C,D) in our cluster.
The election of recovery master will be triggered after the disconnected
determination of CTDB, i.e. the CTDB elects a new recovery master for each
group after 26 (KeepaliveInterval*KeepaliveLimits+1 by default) seconds.
Then node A will be the recovery master of group (A) and some node (e.g. B)
will be the recovery master of group (B,C,D).
Now, A

Re: RHEL/CentOS-6 HA NFS Configuration Question

Benjamin Coddington — 2012-05-22T13:26:21

My apologies, this second location is the same as the first, and should be
http://marc.info/?l=linux-nfs&m=131914563520472&w=2

Ben

Re: RHEL/CentOS-6 HA NFS Configuration Question

Benjamin Coddington — 2012-05-22T13:05:32

Those running HA NFS should be aware of the following two NFSD open leaks.

The first is the nfs4_open_downgrade leak:
http://marc.info/?l=linux-nfs&m=131077202109185&w=2
https://bugzilla.redhat.com/show_bug.cgi?id=714153

Redhat supposedly fixed this, but I never saw the errata go by.. while we
waited for them to fix it, we went to an upstream kernel and got bit
by this one:

http://marc.info/?l=linux-nfs&m=131077202109185&w=2

NFSD open leaks will cause your filesystems to fail to umount, even after
waiting through your lease time.  You'll see the device's open count
will be non-zero (dmsetup info <device>), even though the filesystem
is unexported, and kernel nfsds are stopped.

We've been running our NFS4 HA cluster for a few months now on
a 3.2.5 kernel, and failover/recovery works well.

Ben

On May 16, 2012, at 2:19 PM, Colin Simpson wrote:

Re: RHEL/CentOS-6 HA NFS Configuration Question

Fabio M. Di Nitto — 2012-05-17T09:57:22

Hi Colin,

On 5/17/2012 11:47 AM, Colin Simpson wrote:

Understood. I can´t really say if userland or kernel would make any
difference in this specific unmount issue, but for "safety reasons" I
need to assume their design is the same and behave the same way. when/if
there will be a switch, we will need to look more deeply into it. With
current kernel implementation we (cluster guys) need to use this approach.


Yes one for rhel5 and one for rhel6, but they are both private at the
moment because they have customer data in it.

I expect that the workaround/fix (whatever you want to label it) will be
available via RHN in 2/3 weeks.

Fabio


--
Linux-cluster mailing list
Linux-cluster< at >redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Re: RHEL/CentOS-6 HA NFS Configuration Question

Colin Simpson — 2012-05-17T09:47:29

Thanks for all the useful information on this.

I realise the bz is not for this issue, I just included it as it has the
suggestion that nfsd should actually live in user space (which seems
sensible).

Out of interest is there a bz # for this issue?

Colin


On Thu, 2012-05-17 at 10:26 +0200, Fabio M. Di Nitto wrote:


________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.

Re: RHEL/CentOS-6 HA NFS Configuration Question

Colin Simpson — 2012-05-17T09:47:00

Sadly that doesn't work (usually). With nfsd the filesystem will still
refuse to umount (despite the force) as it's locked in kernel space
(where nfsd lives). And the service will just go to failed.

The best you can so is the probably self_fence="1" which is pretty
brutal and halts the node with the stuck fs mount, I believe.

Colin

On Thu, 2012-05-17 at 00:45 +0200, emmanuel segura wrote:


________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.

Re: Linux-cluster Digest, Vol 97, Issue 5

Fabio M. Di Nitto — 2012-05-17T09:38:04

Emmanuel,

On 5/17/2012 10:38 AM, emmanuel segura wrote:

Wow. that´s good planning of your.. so instead of rely on an explicit
order in cluster.conf, you add an extra layer of obfuscation by assuming
service.sh will always be the same.

Explicit config is more readable and make the service clear for the user
vs expecting a user to dig into some file where priorities are described
and that can change.


I seriously hope this is sarcasm.

Fabio

Re: Linux-cluster Digest, Vol 97, Issue 5

emmanuel segura — 2012-05-17T08:38:21

Fabio

The Ip it's the last to start, as sayed before look vim
/usr/share/cluster/service.sh

a have a cluster configured like that and i can tell i never found the
problem
==============================
=============================

Look this, the order start of resources isn't based on the order in the xml
/etc/cluster.conf, i see you work in Redhat and you don't know this Mama Mia
===========================================================
 <special tag="rgmanager">
        <attributes root="1" maxinstances="1"/>
        <child type="lvm" start="1" stop="9"/>
        <child type="fs" start="2" stop="8"/>
        <child type="clusterfs" start="3" stop="7"/>
        <child type="netfs" start="4" stop="6"/>
        <child type="nfsexport" start="5" stop="5"/>

        <child type="nfsclient" start="6" stop="4"/>

        <child type="ip" start="7" stop="2"/>
        <child type="smb" start="8" stop="3"/>
        <child type="script" start="9" stop="1"/>
    </special>
=======================================

Re: Linux-cluster Digest, Vol 97, Issue 5

emmanuel segura — 2012-05-17T08:37:24

Fabio

The Ip it's the last to start, as sayed before look vim
/usr/share/cluster/service.sh

a have a cluster configured like that and i can tell i never found the
problem
===========================================================

Look this, the order start of resources isn't based on the order in the xml
/etc/cluster.conf, i see you work in Redhat and i don't know this Mama Mia
===========================================================
 <special tag="rgmanager">
        <attributes root="1" maxinstances="1"/>
        <child type="lvm" start="1" stop="9"/>
        <child type="fs" start="2" stop="8"/>
        <child type="clusterfs" start="3" stop="7"/>
        <child type="netfs" start="4" stop="6"/>
        <child type="nfsexport" start="5" stop="5"/>

        <child type="nfsclient" start="6" stop="4"/>

        <child type="ip" start="7" stop="2"/>
        <child type="smb" start="8" stop="3"/>
        <child type="script" start="9" stop="1"/>
    </special>
==========================================

Re: RHEL/CentOS-6 HA NFS Configuration Question

Fabio M. Di Nitto — 2012-05-17T08:26:46

Yes, I am aware the issue since I have been investigating it in details
for the past couple of weeks.


First, the bz you mention below is unrelated to the unmount problem we
are discussing. clustered nfsd locks are a slightly different story.

There are two issues here:

1) cluster users expectations
2) nfsd internal design

(and note I am not blaming either cluster or nfsd here)

Generally cluster users expect to be able to do things like (fake meta
config):

<service1..
 <fs1..
  <nfsexport1..
   <nfsclient1..
    <ip1..
....
<service2
 <fs2..
  <nfsexport2..
   <nfsclient2..
    <ip2..

and be able to move services around cluster nodes without problem. Note
that it is irrelevant of the fs used. It can be clustered or not.

This setup does unfortunately clash with nfsd design.

When shutdown of a service happens (due to stop or relocation is
indifferent):

ip is removed
exportfs -u .....
(and that's where we hit the nfsd design limitation)
umount fs..

By design (tho I can't say exactly why it is done th

Re: Linux-cluster Digest, Vol 97, Issue 5

Fabio M. Di Nitto — 2012-05-17T08:02:54


That is also wrong.

<ip has to be the last one to start or you will hit race conditions at
service startup.

Fabio