gmane.comp.java.hadoop.hbase.user

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:32:00

I believe there were cascading failures which got these deep nodes
containing still to be replicated WAL(s) - I suspect there is either some
parsing bug or something which is causing the replication source to not
work - also which version are you using - does it have
https://issues.apache.org/jira/browse/HBASE-8207 - since you use hyphens in
our paths. One way to get back up is to delete these nodes but then you
lose data in these WAL(s)...


On Wed, May 22, 2013 at 2:22 PM, Amit Mor <amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

Amit Mor — 2013-05-22T21:22:06

 va-p-hbase-02-d,60020,1369249862401


On Thu, May 23, 2013 at 12:20 AM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:20:22

Basically

ls /hbase/rs and what do you see for va-p-02-d ?


On Wed, May 22, 2013 at 2:19 PM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:19:45

Can you do ls /hbase/rs and see what you get for 02-d - instead of looking
in /replication/, could you look in /hbase/replication/rs - I want to see
if the timestamps are matching or not ?

Varun


On Wed, May 22, 2013 at 2:17 PM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:17:57

I see - so looks okay - there's just a lot of deep nesting in there - if
you look into these you nodes by doing ls - you should see a bunch of
WAL(s) which still need to be replicated...

Varun


On Wed, May 22, 2013 at 2:16 PM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:16:01

2013-05-22 15:31:25,929 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for *
/hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1-va-p-hbase-01-c,60020,1369042378287-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475/va-p-hbase-01-c%2C60020%2C1369042378287.1369220050719
*
*
*
*01->[01->02->02]->01*

*Looks like a bunch of cascading failures causing this deep nesting... *


On Wed, May 22, 2013 at 2:09 PM, Amit Mor <amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

Amit Mor — 2013-05-22T21:15:26

I found this:

[zk: va-p-zookeeper-01-c:2181(CONNECTED) 17] ls
/hbase/replication/rs/va-p-hbase-02-d,60020,1369249862401
[1-va-p-hbase-02-e,60020,1369042377129-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-01-d,60020,1369042382584-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475,
1,
1-va-p-hbase-02-e,60020,1369233253407-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-01-e,60020,1369233254969-va-p-hbase-02-e,60020,1369233253407-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-02-d,60020,1369042368330-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-01-e,60020,1369042368595-va-p-hbase-02-c,60020,1369042377731-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-01-c,60020,1369233253404-va-p-hbase-02-e,60020,1369233253407-va-p-hbase-02-d,60020,1369233252475,
1-va-p-hbase-01-d,60020,1369233257617-va-p-

Re: Simple but Reliable configuration for pseudo distributed mode on ubuntu

Stack — 2013-05-22T21:12:56

What did the logs show regards who could not find who? (I would like to
answer Jay Vyas but thought I'd ask here first to see if could see what
tight-coupling to /etc/hosts we are guilty of).

Thanks,
St.Ack


On Wed, May 22, 2013 at 12:46 PM, Stephen Boesch <javadba-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Simple but Reliable configuration for pseudo distributed mode on ubuntu

Stack — 2013-05-22T21:10:06

How could we improve the doc Stephen?  Was the problem that it was only in
the quick start section?  Thanks.
St.Ack


On Wed, May 22, 2013 at 12:46 PM, Stephen Boesch <javadba-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

Amit Mor — 2013-05-22T21:09:03

empty return:

[zk: va-p-zookeeper-01-c:2181(CONNECTED) 10] ls
/hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1
[]



On Thu, May 23, 2013 at 12:05 AM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T21:05:14

Do an "ls" not a get here and give the output ?

ls /hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1


On Wed, May 22, 2013 at 1:53 PM, amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org <
amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

2013-05-22T20:53:33

[zk: va-p-zookeeper-01-c:2181(CONNECTED) 3] get
/hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1

cZxid = 0x60281c1de
ctime = Wed May 22 15:11:17 EDT 2013
mZxid = 0x60281c1de
mtime = Wed May 22 15:11:17 EDT 2013
pZxid = 0x60281c1de
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0



On Wed, May 22, 2013 at 11:49 PM, Ted Yu <yuzhihong-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

Ted Yu — 2013-05-22T20:49:35

What does this command show you ?

get /hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1

Cheers

On Wed, May 22, 2013 at 1:46 PM, amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org <
amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

2013-05-22T20:46:25

ls /hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379
[1]
[zk: va-p-zookeeper-01-c:2181(CONNECTED) 2] ls
/hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379/1
[]

I'm on hbase-0.94.2-cdh4.2.1

Thanks


On Wed, May 22, 2013 at 11:40 PM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T20:40:41

Also what version of HBase are you running ?


On Wed, May 22, 2013 at 1:38 PM, Varun Sharma <varun-qCC3EHmTGpXby3iVrkZq2A< at >public.gmane.org> wrote:

Re: Poor HBase map-reduce scan performance

Ted Yu — 2013-05-22T20:39:55

Sandy:
Do you think the following JIRA would help with what you expect in this
regard ?

HBASE-8420 Port HBASE-6874 Implement prefetching for scanners from 0.89-fb

Cheers

On Wed, May 22, 2013 at 1:29 PM, Sandy Pratt <prattrs-dv/VyGpifdQAvxtiuMwx3w< at >public.gmane.org> wrote:

Re: RS crash upon replication

Varun Sharma — 2013-05-22T20:38:57

Basically,

You had va-p-hbase-02 crash - that caused all the replication related data
in zookeeper to be moved to va-p-hbase-01 and have it take over for
replicating 02's logs. Now each region server also maintains an in-memory
state of whats in ZK, it seems like when you start up 01, its trying to
replicate the 02 logs underneath but its failing to because that data is
not in ZK. This is somewhat weird...

Can you open the zookeepeer shell and do

ls /hbase/replication/rs/va-p-hbase-01-c,60020,1369249873379

And give the output ?


On Wed, May 22, 2013 at 1:27 PM, amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org <
amit.mor.mail-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Poor HBase map-reduce scan performance

Sandy Pratt — 2013-05-22T20:29:29

I found this thread on search-hadoop.com just now because I've been
wrestling with the same issue for a while and have as yet been unable to
solve it.  However, I think I have an idea of the problem.  My theory is
based on assumptions about what's going on in HBase and HDFS internally,
so please correct me if I'm wrong.

Briefly, I think the issue is that sequential reads from HDFS are
pipelined, whereas sequential reads from HBase are not.  Therefore,
sequential reads from HDFS tend to keep the IO subsystem saturated, while
sequential reads from HBase allow it to idle for a relatively large
proportion of time.

To make this more concrete, suppose that I'm reading N bytes of data from
a file in HDFS.  I issue the calls to open the file and begin to read
(from an InputStream, for example).  As I'm reading byte 1 of the stream
at my client, the datanode is reading byte M where 1 < M <= N from disk.
Thus, three activities tend to happen concurrently for the most part
(disregarding the beginning and end of the f

RS crash upon replication

2013-05-22T20:27:21

Hi,

This is bad ... and happened twice: I had my replication-slave cluster
offlined. I performed quite a massive Merge operation on it and after a
couple of hours it had finished and I returned it back online. At the same
time, the replication-master RS machines crashed (see first crash
http://pastebin.com/1msNZ2tH) with the first exception being:

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for
/hbase/replication/rs/va-p-hbase-01-c,60020,1369233253404/1-va-p-hbase-01-c,60020,1369042378287-va-p-hbase-02-c,60020,1369042377731/va-p-hbase-01-c%2C60020%2C1369042378287.1369220050719
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
        at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.setData(ZKUti

Re: HBase is not running.

Yves S. Garret — 2013-05-22T19:35:54

How does Zookeeper fit into this picture?  For hbase.zookeeper.quorum,
I have it set to localhost.  Would I need to include Zookeeper or start it
up in some way in order to get it to run?


On Wed, May 22, 2013 at 1:44 PM, Jay Vyas <jayunit100-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Simple but Reliable configuration for pseudo distributed mode on ubuntu

Jay Vyas — 2013-05-22T19:51:35

clearly there is a very high coupling to the /etc/hosts file in hbase.  Is
this necessary?


On Wed, May 22, 2013 at 3:46 PM, Stephen Boesch <javadba-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote: