http://permalink.gmane.org/gmane.comp.db.cassandra.user hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.db.cassandra.user/33609 <pre> If you don't use replace_token, this won't work at all. You'll get "attempt to bootstrap node into range of live node" type error, because the old ip will still own the token. If you do use replace_token, there have been historical cases where you get (non-service-impacting) ghost node state. replace_token is your friend. note however that replace_token is effectively a bootstrap, so you will end up with duplicate/extra data. :) =Rob </pre> Robert Coli 2013-05-24T17:01:16 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33608 <pre>Hi aaron an thanks, results that sounds like a bug. If you are mixing the CL levels such that R + W &lt;= N then it's expected behaviour. I think it's a bug, it concern only some keys (~200 over 120 000 keys) on one column family, As I understand it, if R+W &lt;= N it's expected behaviour at a moment, but read repair will correct the inconsistent data (related to read_repair_chance value) and the next query will return a consistent data , right ? Here is an exemple of a result that i have on a key (keyspace RF 3), 3 differents replicas : [default&lt; at &gt;prod] ASSUME contacts_timeordered KEYS AS ascii; [default&lt; at &gt;prod] get contacts_timeordered['1425185-IGNORED']; =&gt; (column=1a927740-97ec-11e2-ab16-a1afd66a735e, value=363838353936, timestamp=1364505108842098) =&gt; (column=1a93d5b0-97ec-11e2-888c-2bf068e0f754, value=31373930303330, timestamp=1364505108851088) =&gt; (column=b5c559c0-9d0f-11e2-8682-f7ecd4112689, value=32343130303930, timestamp=1365070157421869) =&gt; (column=7ba22b90-b48b-11e2-a2c2-914573921d9f, value=32353031353</pre> Kais Ahmed 2013-05-24T16:58:59 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33607 <pre> D'oh, knew I should have suggested underlying hardware failure. "Input/Output Error" is almost always failed disk... Bootstrap only ever occurs from one node/vnode per range being bootstrapped. It would be nice if things which encounter corrupt sstables (perhaps during streaming) had clearer exceptions regarding the failure. But I don't believe that it is likely for bootstrap to be changed to bootstrap from more than one node... =Rob </pre> Robert Coli 2013-05-24T16:58:50 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33606 <pre> You can check via lsof whether the file is open by cassandra, but if it is corrupt it was probably not successfully opened. If it's not open by cassandra, you can do whatever you want to it. Actually if it's open by cassandra you can also do whatever you want to, except that whatever you do won't happen until cassandra releases its filehandle. =Rob </pre> Robert Coli 2013-05-24T16:54:23 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33605 <pre>I seem to remember problems with ghost nodes, etc. and I seem to remember if you are replacing a node and you don’t use the same ip, this can cause issues. Is this correct? We would like the new node to keep the same token, and the same host name but are wondering if we can change the ip since we are bringing the new node up in a new rac that has a 10Gb switch. Thanks, Dean </pre> Hiller, Dean 2013-05-24T16:01:50 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33604 <pre>The short answer is yes, you can rely on the ordering of keys being consistent. They will always be returned in partitioner order. This is pretty much implied by the existence of the token() function so it's not going to change (if only because changing it would break people). -- Sylvain On Fri, May 24, 2013 at 4:52 PM, Ondřej Černoš &lt;cernoso&lt; at &gt;gmail.com&gt; wrote: </pre> Sylvain Lebresne 2013-05-24T15:34:28 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33603 <pre>Lol, yup, I just found out that is the issue!!!! Thanks, Dean On 5/24/13 8:55 AM, "Yuki Morishita" &lt;mor.yuki&lt; at &gt;gmail.com&gt; wrote: </pre> Hiller, Dean 2013-05-24T15:06:14 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33602 <pre>For anyone else that might be interested, when the stream hangs, there is no exceptions around that time frame as to what exactly happened and why it hung(there is an exception just not informative at all). We did find other exceptions that we "thought" were unrelated though days before. We found a corrupt sstable exception and took the file name and it happened to be the same file name of the file that it was hung on. We then tried to mv the file and got a Input/Output Error so now we know something is really wrong and could be a hardware issue(which is sad since this hardware is not even a year old yet). Anyways, I hope that helps someone else out. Bootstrapping a node does not fair well if one of the nodes has corrupt sstables which is funny as I thought it should just read from one of the other nodes on failure……(anyway to tell a node to bootstrap from a different node?) Should I file a ticket on this as I think you should be able to reproduce by somehow making a corrupt sstable file or somethi</pre> Hiller, Dean 2013-05-24T15:04:12 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33601 <pre>hmm, I only can say it may caused by corrupt SSTable... Stream hang on unexpected error was fixed in 1.2.5 (https://issues.apache.org/jira/browse/CASSANDRA-5229). On Fri, May 24, 2013 at 6:56 AM, Hiller, Dean &lt;Dean.Hiller&lt; at &gt;nrel.gov&gt; wrote: </pre> Yuki Morishita 2013-05-24T14:55:52 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33600 <pre>Hi all, I need to support a legacy API where page offset and limit are on the input of the API call (it used to be mapped directly to offset and limit MySQL select options). The data are pretty small (like really small, some hundreds of thousands narrow rows maximum - I use Cassandra for its multiple-dc and HA capabilities, not for "big data"). I know the token(key) function and its use for paging, but unfortunately I cannot change the API to a version where last key on previous page and limit would be provided. What I thought I would do - though it is violating good Cassandra practices like "don't fetch all keys" - is the following: select _key_ from table limit _offset_value_; select _columns_ from table where token(_key_) &gt; token(_last_key_from_the_select_above_); The first select tells me where the offset begins and the second one queries for the page. The paged queries will not be performed too often, so performance is not such a big deal here. This construct however depends on repeatable orderin</pre> Ondřej Černoš 2013-05-24T14:52:09 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33599 <pre>Delete the data, and associated index/filter/etc files. (all the files with the same number). If you are reading at one, bring the node up and disable thrift then run repair. If you are reading at quorum or higher, not a big need to stop thrift. Just run repair. On Fri, May 24, 2013 at 10:07 AM, Hiller, Dean &lt;Dean.Hiller&lt; at &gt;nrel.gov&gt; wrote: </pre> Edward Capriolo 2013-05-24T14:39:57 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33598 <pre>We have a corrupt sstable databus5-nreldata-ib-36763-Data.db. How do we safely blow this away? (and then we would run repair to make sure all data is still there)… Can we just move the file out from under cassandra? (or would cassandra freak out?) Thanks, Dean </pre> Hiller, Dean 2013-05-24T14:07:06 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33597 <pre>Hi all, I am trying to migrate some some Hector's RangeSlicesQuery to Astyanax, but the only method I have found is getKeyRange[1] which in turn has four parameters startKey,endKey, startToken, endToken, and count. The thing is that I am not sure what are the startToken and endToken parameters used for if I am already controlling the range with the startKey and endKey parameters. Could anyone show me some light on this manner please? Thanks in advance. Renato M. [1] http://www.srcrr.com/java/astyanax/1.0.5/reference/com/netflix/astyanax/thrift/ThriftColumnFamilyQueryImpl-source.html </pre> Renato Marroquín Mogrovejo 2013-05-24T13:53:07 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33596 <pre>Hi Aaron, We have a pretty big key space and have found to get a decent key cache hit rate it needs to be quite large. We get 3 SStables per read at the 99th percentile, 2 at the 98th and 95th, 1 below that. No problems at the moment with GC, but we're still quite early in our Cassandra adventure. Thanks for the advice Chris From: aaron morton [mailto:aaron&lt; at &gt;thelastpickle.com] Sent: 23 May 2013 23:50 To: user&lt; at &gt;cassandra.apache.org Subject: Re: High performance disk io I am currently trying to really study the effect of the width of a row (being in multiple sstables) vs its 95th percentile read time. I'd be interested to see your findings. Is use 3+ SSTables per read as (from cfhistograms) as a warning sign to dig deeper in the data model. Also the type of query impacts on the number of SSTables per read, queries by column name can short circuit and may be served from (say) 0 or 1 sstables even if the row is spread out. -We don't change anything and just keep upping</pre> Christopher Wirt 2013-05-24T13:38:06 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33595 <pre>The exception on that node was just this ERROR [Thread-6056] 2013-05-22 14:47:59,416 CassandraDaemon.java (line 132) Exception in thread Thread[Thread-6056,5,main] java.lang.IndexOutOfBoundsException at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:75) at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMay Throw(CompressedInputStream.java:151) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662) On 5/23/13 9:51 AM, "Yuki Morishita" &lt;mor.yuki&lt; at &gt;gmail.com&gt; wrote: </pre> Hiller, Dean 2013-05-24T11:56:13 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33594 <pre>By "it is still there" I mean that when I do get request in Cassandra cli I get the column, as well as when I try to read the column using Hector. I don't think it is a matter of tombstone. I have the default gc_grace_seconds and I run repair weekly (will run on Sunday). Other columns for same CF expire as expected after their 10 seconds pass. Thanks, Tamar *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] tamar&lt; at &gt;tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Thu, May 23, 2013 at 11:20 PM, Robert Coli &lt;rcoli&lt; at &gt;eventbrite.com&gt; wrote: </pre> Tamar Fraenkel 2013-05-24T10:26:37 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33593 <pre>Hmmm... In general it seems that for some reason Cassandra reads invalid value when trying to get key length (it should be ~100-150, but it gets 2048), then basing on this value it reads too much data and when trying to read next key's length again it reads some garbage translating it to a gargantuan int, that has nothing to do with real data layout in file, so in the end it crashes. I paste here some more logs (it crashes in a bit different place in code when streaming only one file, than it happened on production, but due to steps I have to take to reproduce it, I'm sure it's for the same root cause). I've modified code a bit to print debug messages in a few more places and I print the "real" key in DecoratedKey: ----- HERE STARTS THE LAST VALID KEY/VALUE PAIR INFO [Thread-8] 2013-05-23 13:07:06,766 ByteBufferUtil.java (line 370) readShortLength-&gt;readUnsignedShort: 121 INFO [Thread-8] 2013-05-23 13:07:06,766 ByteBufferUtil.java (line 407) READ with length 121 INFO [Thread-8] 2013-05-23 1</pre> Michal Michalski 2013-05-24T10:25:25 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33592 <pre> Heisenbug :D (never heard this name before :-) ) I thought so too, but I finally managed to reproduce it locally (it requires 3 nodes, one of them needs to have a specific token assigned), the rest just have to be present in cluster; plus the special, "invalid" file too of course), so it's probably token/partitioner related, but for now I have no bloody idea how. Thus, I'm rather sure that rebuilding/replacing node will not work :-( I also thought it might be a very weird DeflateCompressor bug causing it to produce invalid output for some strange characters (not very possible, but I had no better idea ;-) ). However, I tested compression/decompression and it's fine. D'oh! M. </pre> Michal Michalski 2013-05-24T08:46:17 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33591 <pre>I am sorry if I was not clear. I was using nodes to refer machines (or vice versa). Let me put in another way...  The application is composed of multiple instances of an executable. The application runs on multiple machines concurrently. All the instances are going to issue the same CQL command to and try to create exactly same namespace and column families. Thank you Emalayan ________________________________ From: Arthur Zubarev &lt;Arthur.Zubarev&lt; at &gt;Aol.com&gt; To: Emalayan Vairavanathan &lt;svemalayan&lt; at &gt;yahoo.com&gt;; user&lt; at &gt;cassandra.apache.org Sent: Thursday, 23 May 2013 1:15 PM Subject: Re: Creating namespace and column family from multiple nodes concurrently so where the multiple nodes are? I am just puzzled   From: Emalayan Vairavanathan Sent: Thursday, May 23, 2013 3:43 PM To: Arthur Zubarev ; user&lt; at &gt;cassandra.apache.org Subject: Re: Creating namespace and column family from multiple nodes concurrently   "Would each device/machine have its own keyspace?"   No. All the machines are going to run the ex</pre> Emalayan Vairavanathan 2013-05-24T05:14:27 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33590 <pre>I'd be interested to see your findings. Is use 3+ SSTables per read as (from cfhistograms) as a warning sign to dig deeper in the data model. Also the type of query impacts on the number of SSTables per read, queries by column name can short circuit and may be served from (say) 0 or 1 sstables even if the row is spread out. 800MB is a very high key cache and may result in poor GC performance which is ultimately going to hurt your read latency. Pay attention to what GC is doing, both ParNew and CMS and reduce the key cache if needed. When ParNew runs the server is stalled. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand &lt; at &gt;aaronmorton http://www.thelastpickle.com On 24/05/2013, at 3:16 AM, Edward Capriolo &lt;edlinuxguru&lt; at &gt;gmail.com&gt; wrote: </pre> aaron morton 2013-05-23T22:49:57 http://permalink.gmane.org/gmane.comp.db.cassandra.user/33589 <pre>Thanks, when and were is the talk ? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand &lt; at &gt;aaronmorton http://www.thelastpickle.com On 23/05/2013, at 6:42 AM, Peter Lin &lt;woolfel&lt; at &gt;gmail.com&gt; wrote: </pre> aaron morton 2013-05-23T22:38:37 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.db.cassandra.user