gmane.comp.jakarta.lucene.devel

multivalued field FieldCache

world hello — 2013-05-23T19:34:19








how can I retrieve the value out of a multivalued field in a customized function query?


I used the code but get exceptions - "can not use FieldCache on multivalued field"


////////////////////////////////////////////////////////////////////////////////////
public ValueSource parse(FunctionQParser fp) throws ParseException {


        try { 
            ValueSource vs = fp.parseValueSource();
       }
       catch (...)
       {
       }
}/////////////////////////////////////////////////////////////////////////
Thanks.
-Frank

DistributedUpdateProcessor's isReplayOrPeerSync

Shalin Shekhar Mangar — 2013-05-23T19:00:55

I see the following in solr's DistributedUpdateProcessor:

boolean isReplayOrPeersync = (cmd.getFlags() & (UpdateCommand.REPLAY |
UpdateCommand.REPLAY)) != 0;

Shouldn't that be the following?

boolean isReplayOrPeersync = (cmd.getFlags() & (UpdateCommand.REPLAY |
UpdateCommand.PEER_SYNC)) != 0;

[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_21) - Build # 2853 - Failure!

Policeman Jenkins Server — 2013-05-23T18:56:44

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/2853/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:63381 returned non ok status:500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:63381 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([531B68B6094DCB94:D2FDE6AE7E12ABA8]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 10030 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4]   2> 2208444 T6695 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 2208450 T6695 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\solrtest-ChaosMonkeyShardSplitTest-1369335172070
[junit4:junit4]   2> 2208453 T6695 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 2208454 T6696 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 2208571 T6695 oasc.ZkTestServer.run start zk server on port:63362
[junit4:junit4]   2> 2208574 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2208595 T6702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >43f4e098 name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2208596 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2208597 T6695 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 2208620 T6697 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ed2bc8bef0000, likely client has closed socket
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2208624 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2208629 T6704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >11de6b88 name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2208631 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2208631 T6695 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 2208643 T6695 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 2208650 T6695 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 2208657 T6695 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 2208665 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2208666 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2208678 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 2208679 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 2208687 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2208688 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2208696 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 2208697 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 2208704 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 2208705 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 2208713 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2208714 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2208721 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2208722 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2208729 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2208730 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2208737 T6695 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2208738 T6695 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2208746 T6697 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ed2bc8bef0001, likely client has closed socket
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2209283 T6695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2209289 T6695 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:63369
[junit4:junit4]   2> 2209289 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2209290 T6695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2209290 T6695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364
[junit4:junit4]   2> 2209291 T6695 oasc.CoreContainer$Initializer.initialize looking for solr config file: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\solr.xml
[junit4:junit4]   2> 2209291 T6695 oasc.CoreContainer.<init> New CoreContainer 864829384
[junit4:junit4]   2> 2209293 T6695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\'
[junit4:junit4]   2> 2209293 T6695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\'
[junit4:junit4]   2> 2209431 T6695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2209431 T6695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2209432 T6695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2209432 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2209433 T6695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2209433 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2209435 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2209435 T6695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2209435 T6695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2209436 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2209446 T6695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2209447 T6695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63362/solr
[junit4:junit4]   2> 2209448 T6695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2209451 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2209471 T6715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >57013766 name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2209471 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2209476 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2209476 T6697 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ed2bc8bef0002, likely client has closed socket
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2209486 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2209490 T6717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >4bbf47b3 name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2209491 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2209495 T6695 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 2209506 T6695 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 2209514 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 2209519 T6695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63369_
[junit4:junit4]   2> 2209523 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63369_
[junit4:junit4]   2> 2209531 T6695 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 2209547 T6695 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 2209553 T6695 oasc.Overseer.start Overseer (id=89740749837369347-127.0.0.1:63369_-n_0000000000) starting
[junit4:junit4]   2> 2209563 T6695 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 2209575 T6719 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 2209575 T6695 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 2209585 T6695 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 2209590 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2209597 T6718 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2209605 T6720 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\collection1
[junit4:junit4]   2> 2209605 T6720 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 2209606 T6720 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2209606 T6720 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 2209609 T6720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\collection1\'
[junit4:junit4]   2> 2209611 T6720 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2209612 T6720 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364/collection1/lib/README' to classloader
[junit4:junit4]   2> 2209697 T6720 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2209785 T6720 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2209788 T6720 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2209803 T6720 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2210586 T6720 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2210590 T6720 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2210594 T6720 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2210600 T6720 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2210647 T6720 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2210647 T6720 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369335172364\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/control/data\
[junit4:junit4]   2> 2210649 T6720 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >2d062bb6
[junit4:junit4]   2> 2210649 T6720 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2210650 T6720 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/control/data\
[junit4:junit4]   2> 2210650 T6720 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/control/data\index/
[junit4:junit4]   2> 2210650 T6720 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\control\data\index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2210651 T6720 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/control/data\index
[junit4:junit4]   2> 2210657 T6720 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1913c779 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >163b3350),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2210657 T6720 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2210661 T6720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2210661 T6720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2210662 T6720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2210662 T6720 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2210663 T6720 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2210663 T6720 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2210663 T6720 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2210665 T6720 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2210665 T6720 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2210677 T6720 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2210685 T6720 oass.SolrIndexSearcher.<init> Opening Searcher< at >4d2a117a main
[junit4:junit4]   2> 2210685 T6720 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\control\data\tlog
[junit4:junit4]   2> 2210688 T6720 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2210689 T6720 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2210692 T6721 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >4d2a117a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2210694 T6720 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2210694 T6720 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2211363 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2211364 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"1",
[junit4:junit4]   2>   "numShards":"1",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"control_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63369_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63369"}
[junit4:junit4]   2> 2211364 T6718 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2211364 T6718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2211379 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2211871 T6720 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2211871 T6720 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63369 collection:control_collection shard:shard1
[junit4:junit4]   2> 2211873 T6720 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2211894 T6720 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2211902 T6720 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2211902 T6720 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2211902 T6720 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63369/collection1/
[junit4:junit4]   2> 2211903 T6720 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2211903 T6720 oasc.SyncStrategy.syncToMe http://127.0.0.1:63369/collection1/ has no replicas
[junit4:junit4]   2> 2211903 T6720 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63369/collection1/
[junit4:junit4]   2> 2211904 T6720 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2213148 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2213167 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2213232 T6720 oasc.ZkController.register We are http://127.0.0.1:63369/collection1/ and leader is http://127.0.0.1:63369/collection1/
[junit4:junit4]   2> 2213232 T6720 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63369
[junit4:junit4]   2> 2213232 T6720 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2213234 T6720 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2213234 T6720 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2213237 T6720 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2213239 T6695 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
[junit4:junit4]   2> 2213239 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2213242 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2213252 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 2213255 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2213260 T6724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6639bc4c name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2213261 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2213262 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2213267 T6695 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 2213770 T6695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2213795 T6695 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:63381
[junit4:junit4]   2> 2213796 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2213796 T6695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2213797 T6695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885
[junit4:junit4]   2> 2213797 T6695 oasc.CoreContainer$Initializer.initialize looking for solr config file: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\solr.xml
[junit4:junit4]   2> 2213799 T6695 oasc.CoreContainer.<init> New CoreContainer 1821608151
[junit4:junit4]   2> 2213800 T6695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\'
[junit4:junit4]   2> 2213800 T6695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\'
[junit4:junit4]   2> 2213937 T6695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2213937 T6695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2213937 T6695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2213937 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2213938 T6695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2213938 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2213939 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2213940 T6695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2213940 T6695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2213941 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2213950 T6695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2213950 T6695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63362/solr
[junit4:junit4]   2> 2213950 T6695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2213954 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2213968 T6735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >645c6401 name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2213969 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2213972 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2213972 T6697 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ed2bc8bef0005, likely client has closed socket
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2213980 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2213985 T6737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >61cf0e66 name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2213985 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2214000 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2214940 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2214941 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"1",
[junit4:junit4]   2>   "numShards":"1",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"control_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63369_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63369"}
[junit4:junit4]   2> 2214953 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2214954 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2214953 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2215182 T6695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63381_
[junit4:junit4]   2> 2215185 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63381_
[junit4:junit4]   2> 2215191 T6724 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2215191 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2215191 T6737 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2215192 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2215192 T6717 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2215192 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2215206 T6738 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\collection1
[junit4:junit4]   2> 2215207 T6738 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2215208 T6738 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2215208 T6738 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2215211 T6738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\collection1\'
[junit4:junit4]   2> 2215215 T6738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2215216 T6738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885/collection1/lib/README' to classloader
[junit4:junit4]   2> 2215285 T6738 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2215376 T6738 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2215380 T6738 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2215390 T6738 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2216218 T6738 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2216223 T6738 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2216225 T6738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2216232 T6738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2216275 T6738 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2216277 T6738 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369335176885\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty1\
[junit4:junit4]   2> 2216277 T6738 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >2d062bb6
[junit4:junit4]   2> 2216278 T6738 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2216278 T6738 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty1\
[junit4:junit4]   2> 2216279 T6738 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty1\index/
[junit4:junit4]   2> 2216279 T6738 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty1\index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2216279 T6738 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty1\index
[junit4:junit4]   2> 2216283 T6738 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2216283 T6738 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2216289 T6738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2216289 T6738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2216290 T6738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2216291 T6738 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2216291 T6738 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2216293 T6738 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2216293 T6738 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2216294 T6738 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2216295 T6738 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2216307 T6738 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2216314 T6738 oass.SolrIndexSearcher.<init> Opening Searcher< at >3f0696b main
[junit4:junit4]   2> 2216314 T6738 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty1\tlog
[junit4:junit4]   2> 2216320 T6738 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2216320 T6738 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2216327 T6739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >3f0696b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2216333 T6738 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2216333 T6738 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2216721 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2216722 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63381_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63381"}
[junit4:junit4]   2> 2216723 T6718 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 2216723 T6718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2216735 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2216735 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2216735 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2217514 T6738 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2217514 T6738 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63381 collection:collection1 shard:shard1
[junit4:junit4]   2> 2217516 T6738 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2217535 T6738 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2217542 T6738 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2217542 T6738 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2217543 T6738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63381/collection1/
[junit4:junit4]   2> 2217543 T6738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2217543 T6738 oasc.SyncStrategy.syncToMe http://127.0.0.1:63381/collection1/ has no replicas
[junit4:junit4]   2> 2217544 T6738 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63381/collection1/
[junit4:junit4]   2> 2217544 T6738 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 2218502 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2218525 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218525 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218525 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218574 T6738 oasc.ZkController.register We are http://127.0.0.1:63381/collection1/ and leader is http://127.0.0.1:63381/collection1/
[junit4:junit4]   2> 2218574 T6738 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63381
[junit4:junit4]   2> 2218574 T6738 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2218575 T6738 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2218575 T6738 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2218580 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2218581 T6695 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
[junit4:junit4]   2> 2218582 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2218582 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2219065 T6695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2219071 T6695 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:63390
[junit4:junit4]   2> 2219071 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2219072 T6695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2219073 T6695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209
[junit4:junit4]   2> 2219073 T6695 oasc.CoreContainer$Initializer.initialize looking for solr config file: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\solr.xml
[junit4:junit4]   2> 2219075 T6695 oasc.CoreContainer.<init> New CoreContainer 973141773
[junit4:junit4]   2> 2219075 T6695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\'
[junit4:junit4]   2> 2219075 T6695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\'
[junit4:junit4]   2> 2219227 T6695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2219228 T6695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2219229 T6695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2219230 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2219230 T6695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2219231 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2219231 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2219233 T6695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2219234 T6695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2219235 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2219247 T6695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2219249 T6695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63362/solr
[junit4:junit4]   2> 2219249 T6695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2219252 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2219268 T6751 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1258e71d name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2219268 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2219273 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2219273 T6697 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ed2bc8bef0007, likely client has closed socket
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2219282 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2219286 T6753 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1e89b6d8 name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2219287 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2219296 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2220205 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2220206 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63381_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63381"}
[junit4:junit4]   2> 2220216 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220217 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220216 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220216 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220359 T6695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63390_
[junit4:junit4]   2> 2220367 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63390_
[junit4:junit4]   2> 2220374 T6737 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2220374 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2220374 T6753 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2220376 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2220376 T6717 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2220376 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2220377 T6724 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2220378 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2220391 T6754 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\collection1
[junit4:junit4]   2> 2220391 T6754 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2220393 T6754 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2220393 T6754 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2220396 T6754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\collection1\'
[junit4:junit4]   2> 2220398 T6754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2220399 T6754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209/collection1/lib/README' to classloader
[junit4:junit4]   2> 2220479 T6754 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2220564 T6754 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2220567 T6754 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2220578 T6754 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2221247 T6754 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2221251 T6754 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2221253 T6754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2221263 T6754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2221481 T6754 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2221481 T6754 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369335182209\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty2\
[junit4:junit4]   2> 2221481 T6754 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >2d062bb6
[junit4:junit4]   2> 2221482 T6754 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2221483 T6754 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty2\
[junit4:junit4]   2> 2221484 T6754 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty2\index/
[junit4:junit4]   2> 2221484 T6754 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty2\index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2221485 T6754 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty2\index
[junit4:junit4]   2> 2221491 T6754 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2221492 T6754 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2221498 T6754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2221498 T6754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2221499 T6754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2221500 T6754 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2221501 T6754 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2221501 T6754 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2221502 T6754 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2221504 T6754 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2221505 T6754 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2221522 T6754 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2221534 T6754 oass.SolrIndexSearcher.<init> Opening Searcher< at >7d4de6fd main
[junit4:junit4]   2> 2221534 T6754 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty2\tlog
[junit4:junit4]   2> 2221535 T6754 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2221536 T6754 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2221557 T6754 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2221557 T6754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2221559 T6755 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >7d4de6fd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2221829 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2221830 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"3",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63390_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63390"}
[junit4:junit4]   2> 2221830 T6718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2221831 T6718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 2221845 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2221845 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2221846 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2221848 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2222618 T6754 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2222618 T6754 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63390 collection:collection1 shard:shard2
[junit4:junit4]   2> 2222620 T6754 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 2222639 T6754 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2222646 T6754 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2222647 T6754 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2222647 T6754 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63390/collection1/
[junit4:junit4]   2> 2222648 T6754 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2222648 T6754 oasc.SyncStrategy.syncToMe http://127.0.0.1:63390/collection1/ has no replicas
[junit4:junit4]   2> 2222648 T6754 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63390/collection1/
[junit4:junit4]   2> 2222649 T6754 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 2223444 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2223471 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2223472 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2223472 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2223471 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2223522 T6754 oasc.ZkController.register We are http://127.0.0.1:63390/collection1/ and leader is http://127.0.0.1:63390/collection1/
[junit4:junit4]   2> 2223522 T6754 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63390
[junit4:junit4]   2> 2223522 T6754 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2223523 T6754 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2223523 T6754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2223526 T6754 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2223528 T6695 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
[junit4:junit4]   2> 2223529 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2223529 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2223963 T6695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2223968 T6695 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:63399
[junit4:junit4]   2> 2223969 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2223969 T6695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2223970 T6695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156
[junit4:junit4]   2> 2223971 T6695 oasc.CoreContainer$Initializer.initialize looking for solr config file: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\solr.xml
[junit4:junit4]   2> 2223971 T6695 oasc.CoreContainer.<init> New CoreContainer 939463483
[junit4:junit4]   2> 2223972 T6695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\'
[junit4:junit4]   2> 2223973 T6695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\'
[junit4:junit4]   2> 2224087 T6695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2224088 T6695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2224088 T6695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2224088 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2224089 T6695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2224089 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2224089 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2224089 T6695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2224091 T6695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2224091 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2224099 T6695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2224101 T6695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63362/solr
[junit4:junit4]   2> 2224101 T6695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2224104 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2224125 T6767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >60e49ad name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2224126 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2224130 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2224141 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2224147 T6769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >2eefe80a name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2224147 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2224158 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2225069 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2225071 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"3",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63390_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63390"}
[junit4:junit4]   2> 2225085 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225086 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225086 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225086 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225086 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225218 T6695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63399_
[junit4:junit4]   2> 2225220 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63399_
[junit4:junit4]   2> 2225227 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225229 T6724 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2225230 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2225230 T6753 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2225231 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2225231 T6737 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2225231 T6717 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2225232 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2225231 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2225238 T6769 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2225248 T6770 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\collection1
[junit4:junit4]   2> 2225248 T6770 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2225249 T6770 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2225250 T6770 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2225252 T6770 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\collection1\'
[junit4:junit4]   2> 2225254 T6770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2225255 T6770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156/collection1/lib/README' to classloader
[junit4:junit4]   2> 2225321 T6770 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2225385 T6770 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2225388 T6770 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2225398 T6770 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2226056 T6770 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2226060 T6770 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2226062 T6770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2226069 T6770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2226111 T6770 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2226111 T6770 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369335187156\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\
[junit4:junit4]   2> 2226112 T6770 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >2d062bb6
[junit4:junit4]   2> 2226112 T6770 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2226113 T6770 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\
[junit4:junit4]   2> 2226113 T6770 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index/
[junit4:junit4]   2> 2226113 T6770 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty3\index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2226114 T6770 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index
[junit4:junit4]   2> 2226118 T6770 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >12e232fe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3dd6a071),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2226118 T6770 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2226122 T6770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2226122 T6770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2226123 T6770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2226124 T6770 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2226124 T6770 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2226124 T6770 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2226125 T6770 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2226126 T6770 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2226126 T6770 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2226146 T6770 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2226153 T6770 oass.SolrIndexSearcher.<init> Opening Searcher< at >9b3367c main
[junit4:junit4]   2> 2226153 T6770 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty3\tlog
[junit4:junit4]   2> 2226155 T6770 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2226155 T6770 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2226161 T6771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >9b3367c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2226163 T6770 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2226163 T6770 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2226680 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2226682 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"4",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63399_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63399"}
[junit4:junit4]   2> 2226682 T6718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2226682 T6718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2226696 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2226696 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2226697 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2226696 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2226696 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2227224 T6770 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2227224 T6770 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63399 collection:collection1 shard:shard1
[junit4:junit4]   2> 2227230 T6770 oasc.ZkController.register We are http://127.0.0.1:63399/collection1/ and leader is http://127.0.0.1:63381/collection1/
[junit4:junit4]   2> 2227230 T6770 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63399
[junit4:junit4]   2> 2227231 T6770 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 2227231 T6770 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4952 name=collection1 org.apache.solr.core.SolrCore< at >640524a1 url=http://127.0.0.1:63399/collection1 node=127.0.0.1:63399_ C4952_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:63399_, base_url=http://127.0.0.1:63399}
[junit4:junit4]   2> 2227232 T6772 C4952 P63399 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 2227233 T6772 C4952 P63399 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2227234 T6772 C4952 P63399 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 2227234 T6772 C4952 P63399 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2227234 T6770 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2227238 T6772 C4952 P63399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2227240 T6695 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
[junit4:junit4]   2> 2227242 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2227242 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2227319 T6728 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2227688 T6695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2227692 T6695 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:63409
[junit4:junit4]   2> 2227692 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2227692 T6695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2227692 T6695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882
[junit4:junit4]   2> 2227694 T6695 oasc.CoreContainer$Initializer.initialize looking for solr config file: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\solr.xml
[junit4:junit4]   2> 2227694 T6695 oasc.CoreContainer.<init> New CoreContainer 279156940
[junit4:junit4]   2> 2227694 T6695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\'
[junit4:junit4]   2> 2227695 T6695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\'
[junit4:junit4]   2> 2227802 T6695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2227803 T6695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2227803 T6695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2227804 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2227806 T6695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2227807 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2227807 T6695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2227808 T6695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2227808 T6695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2227809 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2227818 T6695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2227819 T6695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63362/solr
[junit4:junit4]   2> 2227820 T6695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2227822 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2227842 T6784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >5dc408c8 name:ZooKeeperConnection Watcher:127.0.0.1:63362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2227842 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2227847 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2227852 T6695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2227858 T6786 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >33de2aa4 name:ZooKeeperConnection Watcher:127.0.0.1:63362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2227858 T6695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2227868 T6695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2228293 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2228294 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"4",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63399_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63399"}
[junit4:junit4]   2> 2228303 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228304 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228304 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228304 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228304 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228304 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228375 T6728 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 2228375 T6728 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:63399_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1056 
[junit4:junit4]   2> 2228929 T6695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63409_
[junit4:junit4]   2> 2228939 T6695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63409_
[junit4:junit4]   2> 2228945 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228948 T6786 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228948 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2228949 T6724 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228949 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2228950 T6753 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228950 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2228948 T6737 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228952 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2228950 T6717 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228952 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2228953 T6769 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2228966 T6787 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\collection1
[junit4:junit4]   2> 2228967 T6787 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2228968 T6787 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2228968 T6787 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2228972 T6787 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\collection1\'
[junit4:junit4]   2> 2228975 T6787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2228977 T6787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/C:/Users/JenkinsSlave/workspace/Lucene-Solr-trunk-Windows/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882/collection1/lib/README' to classloader
[junit4:junit4]   2> 2229066 T6787 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2229153 T6787 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2229158 T6787 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2229451 T6787 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2230173 T6787 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2230176 T6787 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2230178 T6787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2230185 T6787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2230232 T6787 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2230233 T6787 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369335190882\collection1\, dataDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\
[junit4:junit4]   2> 2230233 T6787 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >2d062bb6
[junit4:junit4]   2> 2230234 T6787 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2230235 T6787 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\
[junit4:junit4]   2> 2230235 T6787 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index/
[junit4:junit4]   2> 2230235 T6787 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty4\index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2230235 T6787 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index
[junit4:junit4]   2> 2230239 T6787 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a8f623f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >48a741f2),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2230239 T6787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2230242 T6787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2230242 T6787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2230244 T6787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2230244 T6787 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2230245 T6787 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2230245 T6787 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2230245 T6787 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2230246 T6787 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2230246 T6787 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2230255 T6787 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2230263 T6787 oass.SolrIndexSearcher.<init> Opening Searcher< at >12c8b018 main
[junit4:junit4]   2> 2230264 T6787 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069\jetty4\tlog
[junit4:junit4]   2> 2230264 T6787 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2230264 T6787 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2230272 T6788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >12c8b018 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2230274 T6787 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2230275 T6787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> ASYNC  NEW_CORE C4953 name=collection1 org.apache.solr.core.SolrCore< at >640524a1 url=http://127.0.0.1:63399/collection1 node=127.0.0.1:63399_ C4953_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:63399_, base_url=http://127.0.0.1:63399}
[junit4:junit4]   2> 2230434 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:63381/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 2230434 T6772 C4953 P63399 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:63399 START replicas=[http://127.0.0.1:63381/collection1/] nUpdates=100
[junit4:junit4]   2> 2230435 T6772 C4953 P63399 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2230435 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 2230435 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 2230435 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 2230436 T6772 C4953 P63399 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2230437 T6772 C4953 P63399 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:63381/collection1/. core=collection1
[junit4:junit4]   2> 2230437 T6772 C4953 P63399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4954 name=collection1 org.apache.solr.core.SolrCore< at >7ba30948 url=http://127.0.0.1:63381/collection1 node=127.0.0.1:63381_ C4954_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63381_, base_url=http://127.0.0.1:63381, leader=true}
[junit4:junit4]   2> 2230446 T6728 C4954 P63381 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 2230451 T6731 C4954 P63381 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2230454 T6731 C4954 P63381 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2230455 T6731 C4954 P63381 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2230455 T6731 C4954 P63381 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2230455 T6731 C4954 P63381 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2230457 T6731 C4954 P63381 oass.SolrIndexSearcher.<init> Opening Searcher< at >3cd26a0f realtime
[junit4:junit4]   2> 2230457 T6731 C4954 P63381 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2230457 T6731 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 2230457 T6772 C4953 P63399 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2230457 T6772 C4953 P63399 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2230460 T6733 C4954 P63381 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2230461 T6733 C4954 P63381 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 2230462 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2230462 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2230463 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2230465 T6729 C4954 P63381 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2230466 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 2230467 T6772 C4953 P63399 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index.20130523195314084
[junit4:junit4]   2> 2230467 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >77854fb8 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3867b0c0) fullCopy=false
[junit4:junit4]   2> 2230469 T6730 C4954 P63381 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 2230470 T6772 C4953 P63399 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2230470 T6772 C4953 P63399 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2230470 T6772 C4953 P63399 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 2230473 T6772 C4953 P63399 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >12e232fe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3dd6a071),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >12e232fe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3dd6a071),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2230474 T6772 C4953 P63399 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2230474 T6772 C4953 P63399 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2230474 T6772 C4953 P63399 oass.SolrIndexSearcher.<init> Opening Searcher< at >12b17f68 main
[junit4:junit4]   2> 2230476 T6771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >12b17f68 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2230477 T6772 C4953 P63399 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index.20130523195314084 [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index.20130523195314084;done=true>>]
[junit4:junit4]   2> 2230477 T6772 C4953 P63399 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index.20130523195314084
[junit4:junit4]   2> 2230477 T6772 C4953 P63399 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty3\index.20130523195314084
[junit4:junit4]   2> 2230477 T6772 C4953 P63399 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 2230477 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 2230478 T6772 C4953 P63399 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2230478 T6772 C4953 P63399 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2230481 T6772 C4953 P63399 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 2231539 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2231540 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"5",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63409_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63409"}
[junit4:junit4]   2> 2231540 T6718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2231541 T6718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 2231554 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"4",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63399_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63399"}
[junit4:junit4]   2> 2231562 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2231563 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2231563 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2231563 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2231563 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2231564 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2232180 T6787 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2232180 T6787 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63409 collection:collection1 shard:shard2
[junit4:junit4]   2> 2232186 T6787 oasc.ZkController.register We are http://127.0.0.1:63409/collection1/ and leader is http://127.0.0.1:63390/collection1/
[junit4:junit4]   2> 2232186 T6787 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63409
[junit4:junit4]   2> 2232186 T6787 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 2232186 T6787 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4955 name=collection1 org.apache.solr.core.SolrCore< at >5172174b url=http://127.0.0.1:63409/collection1 node=127.0.0.1:63409_ C4955_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:63409_, base_url=http://127.0.0.1:63409}
[junit4:junit4]   2> 2232187 T6791 C4955 P63409 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 2232188 T6787 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2232188 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2232189 T6791 C4955 P63409 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 2232189 T6791 C4955 P63409 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2232189 T6695 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
[junit4:junit4]   2> 2232190 T6695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2232191 T6695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2232191 T6791 C4955 P63409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2232199 T6749 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2232200 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2232201 T6695 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 2232202 T6695 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 2232202 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2233004 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2233005 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"5",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63409_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63409"}
[junit4:junit4]   2> 2233017 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233018 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233019 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233018 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233018 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233018 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2233149 T6749 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 2233149 T6749 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:63409_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=951 
[junit4:junit4]   2> 2233154 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2234105 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C4955_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:63409_, base_url=http://127.0.0.1:63409}
[junit4:junit4]   2> 2235051 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:63390/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 2235051 T6791 C4955 P63409 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:63409 START replicas=[http://127.0.0.1:63390/collection1/] nUpdates=100
[junit4:junit4]   2> 2235051 T6791 C4955 P63409 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2235051 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 2235051 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 2235052 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 2235052 T6791 C4955 P63409 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2235052 T6791 C4955 P63409 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:63390/collection1/. core=collection1
[junit4:junit4]   2> 2235052 T6791 C4955 P63409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4956 name=collection1 org.apache.solr.core.SolrCore< at >5ac883c8 url=http://127.0.0.1:63390/collection1 node=127.0.0.1:63390_ C4956_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63390_, base_url=http://127.0.0.1:63390, leader=true}
[junit4:junit4]   2> 2235060 T6744 C4956 P63390 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 2235063 T6749 C4956 P63390 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2235063 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2235067 T6749 C4956 P63390 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2235068 T6749 C4956 P63390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2235068 T6749 C4956 P63390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2235069 T6749 C4956 P63390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2235069 T6749 C4956 P63390 oass.SolrIndexSearcher.<init> Opening Searcher< at >18274888 realtime
[junit4:junit4]   2> 2235069 T6749 C4956 P63390 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2235069 T6749 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 2235071 T6791 C4955 P63409 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2235071 T6791 C4955 P63409 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2235074 T6746 C4956 P63390 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2235075 T6746 C4956 P63390 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 2235075 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2235075 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2235076 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2235077 T6747 C4956 P63390 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2235078 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 2235079 T6791 C4955 P63409 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index.20130523195318696
[junit4:junit4]   2> 2235079 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >472b86a3 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >26a6697b) fullCopy=false
[junit4:junit4]   2> 2235082 T6748 C4956 P63390 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 2235083 T6791 C4955 P63409 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2235084 T6791 C4955 P63409 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2235084 T6791 C4955 P63409 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 2235087 T6791 C4955 P63409 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a8f623f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >48a741f2),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a8f623f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >48a741f2),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2235088 T6791 C4955 P63409 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2235089 T6791 C4955 P63409 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2235089 T6791 C4955 P63409 oass.SolrIndexSearcher.<init> Opening Searcher< at >685f69af main
[junit4:junit4]   2> 2235089 T6788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >685f69af main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2235090 T6791 C4955 P63409 oasc.CachingDirectoryFactory.closeCacheValue looking to close .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index.20130523195318696 [CachedDir<<refCount=0;path=.\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index.20130523195318696;done=true>>]
[junit4:junit4]   2> 2235090 T6791 C4955 P63409 oasc.CachingDirectoryFactory.close Closing directory: .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index.20130523195318696
[junit4:junit4]   2> 2235090 T6791 C4955 P63409 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: .\org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369335172069/jetty4\index.20130523195318696
[junit4:junit4]   2> 2235091 T6791 C4955 P63409 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 2235091 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 2235091 T6791 C4955 P63409 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2235092 T6791 C4955 P63409 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2235101 T6791 C4955 P63409 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 2235881 T6718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2235882 T6718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"5",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:63409_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:63409"}
[junit4:junit4]   2> 2235893 T6769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235894 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235895 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235895 T6753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235895 T6724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235895 T6717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2236014 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2236016 T6695 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C4957 name=collection1 org.apache.solr.core.SolrCore< at >6bf9772d url=http://127.0.0.1:63369/collection1 node=127.0.0.1:63369_ C4957_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:63369_, base_url=http://127.0.0.1:63369, leader=true}
[junit4:junit4]   2> 2236026 T6709 C4957 P63369 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2236028 T6709 C4957 P63369 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1913c779 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >163b3350),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2236029 T6709 C4957 P63369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2236030 T6709 C4957 P63369 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1913c779 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >163b3350),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1913c779 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >163b3350),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2236030 T6709 C4957 P63369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2236030 T6709 C4957 P63369 oass.SolrIndexSearcher.<init> Opening Searcher< at >7aea01ab main
[junit4:junit4]   2> 2236032 T6709 C4957 P63369 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2236033 T6721 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >7aea01ab main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2236033 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2> 2236039 T6745 C4956 P63390 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2236039 T6745 C4956 P63390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >376ffda5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >172f6812),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2236039 T6745 C4956 P63390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2236039 T6745 C4956 P63390 oass.SolrIndexSearcher.<init> Opening Searcher< at >127d90ab main
[junit4:junit4]   2> 2236040 T6745 C4956 P63390 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2236041 T6755 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >127d90ab main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2236042 T6745 C4956 P63390 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:63381/collection1/, StdNode: http://127.0.0.1:63399/collection1/, StdNode: http://127.0.0.1:63409/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 2236049 T6732 C4954 P63381 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C4953_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63399_, base_url=http://127.0.0.1:63399}
[junit4:junit4]   2> 2236058 T6764 C4953 P63399 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C4955_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63409_, base_url=http://127.0.0.1:63409}
[junit4:junit4]   2> 2236060 T6777 C4955 P63409 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2236060 T6732 C4954 P63381 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2008d93f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1db4153f),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2236061 T6732 C4954 P63381 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2236062 T6732 C4954 P63381 oass.SolrIndexSearcher.<init> Opening Searcher< at >f974f3f main
[junit4:junit4]   2> 2236063 T6732 C4954 P63381 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2236060 T6764 C4953 P63399 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >12e232fe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3dd6a071),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >12e232fe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3dd6a071),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2236064 T6764 C4953 P63399 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2236064 T6739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >f974f3f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2236061 T6777 C4955 P63409 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a8f623f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >48a741f2),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a8f623f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >48a741f2),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2236065 T6777 C4955 P63409 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2236065 T6732 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4]   2> 2236064 T6764 C4953 P63399 oass.SolrIndexSearcher.<init> Opening Searcher< at >32515b37 main
[junit4:junit4]   2> 2236066 T6764 C4953 P63399 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2236067 T6777 C4955 P63409 oass.SolrIndexSearcher.<init> Opening Searcher< at >4a957e4b main
[junit4:junit4]   2> 2236067 T6771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >32515b37 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2236068 T6764 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 2236070 T6777 C4955 P63409 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2236069 T6788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >4a957e4b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2236071 T6777 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 2236071 T6745 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 32
[junit4:junit4]   2> 2236073 T6695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2236076 T6733 C4954 P63381 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2236077 T6765 C4953 P63399 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2236079 T6746 C4956 P63390 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2236081 T6779 C4955 P63409 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2237986 T6712 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435852028355018752)} 0 2
[junit4:junit4]   2> 2237996 T6778 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435852028360261632&update.from=http://127.0.0.1:63390/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435852028360261632)} 0 3
[junit4:junit4]   2> 2238001 T6763 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435852028364455936&update.from=http://127.0.0.1:63381/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435852028364455936)} 0 2
[junit4:junit4]   2> 2238002 T6731 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435852028364455936)} 0 9
[junit4:junit4]   2> 2238003 T6749 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435852028360261632)} 0 14
[junit4:junit4]   2> 2238006 T6708 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1435852028378087424)]} 0 0
[junit4:junit4]   2> 2238015 T6764 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435852028385427456)]} 0 0
[junit4:junit4]   2> 2238017 T6732 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1435852028385427456)]} 0 4
[junit4:junit4]   2> 2238018 T6745 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 7
[junit4:junit4]   2> 2238021 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1435852028393816064)]} 0 0
[junit4:junit4]   2> 2238026 T6765 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435852028396961792)]} 0 0
[junit4:junit4]   2> 2238027 T6733 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1435852028396961792)]} 0 3
[junit4:junit4]   2> 2238029 T6710 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1435852028402204672)]} 0 0
[junit4:junit4]   2> 2238038 T6777 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435852028408496128)]} 0 0
[junit4:junit4]   2> 2238039 T6746 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435852028408496128)]} 0 4
[junit4:junit4]   2> 2238040 T6729 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2]} 0 7
[junit4:junit4]   2> 2238043 T6711 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1435852028416884736)]} 0 0
[junit4:junit4]   2> 2238049 T6779 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435852028420030464)]} 0 0
[junit4:junit4]   2> 2238050 T6747 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1435852028420030464)]} 0 4
[junit4:junit4]   2> 2238052 T6713 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1435852028426321920)]} 0 0
[junit4:junit4]   2> 2238060 T6760 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435852028430516224)]} 0 0
[junit4:junit4]   2> 2238061 T6730 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1435852028430516224)]} 0 5
[junit4:junit4]   2> 2238062 T6748 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4]} 0 8
[junit4:junit4]   2> 2238064 T6712 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1435852028438904832)]} 0 0
[junit4:junit4]   2> 2238069 T6780 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435852028441001984)]} 0 0
[junit4:junit4]   2> 2238071 T6744 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1435852028441001984)]} 0 5
[junit4:junit4]   2> 2238073 T6708 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1435852028448342016)]} 0 0
[junit4:junit4]   2> 2238080 T6781 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435852028454633472)]} 0 0
[junit4:junit4]   2> 2238081 T6749 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1435852028454633472)]} 0 2
[junit4:junit4]   2> 2238082 T6728 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6]} 0 6
[junit4:junit4]   2> 2238085 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1435852028460924928)]} 0 0
[junit4:junit4]   2> 2238092 T6782 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435852028467216384)]} 0 0
[junit4:junit4]   2> 2238093 T6745 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1435852028467216384)]} 0 2
[junit4:junit4]   2> 2238094 T6731 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7]} 0 5
[junit4:junit4]   2> 2238098 T6710 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1435852028474556416)]} 0 0
[junit4:junit4]   2> 2238106 T6761 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435852028480847872)]} 0 0
[junit4:junit4]   2> 2238107 T6732 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1435852028480847872)]} 0 3
[junit4:junit4]   2> 2238108 T6746 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8]} 0 7
[junit4:junit4]   2> 2238111 T6711 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1435852028488187904)]} 0 0
[junit4:junit4]   2> 2238117 T6778 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435852028491333632)]} 0 0
[junit4:junit4]   2> 2238117 T6747 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1435852028491333632)]} 0 3
[junit4:junit4]   2> 2238121 T6713 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1435852028498673664)]} 0 0
[junit4:junit4]   2> 2238127 T6762 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435852028501819392)]} 0 0
[junit4:junit4]   2> 2238128 T6733 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1435852028501819392)]} 0 4
[junit4:junit4]   2> 2238130 T6712 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1435852028508110848)]} 0 0
[junit4:junit4]   2> 2238135 T6763 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435852028510208000)]} 0 0
[junit4:junit4]   2> 2238136 T6729 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1435852028510208000)]} 0 4
[junit4:junit4]   2> 2238138 T6708 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1435852028516499456)]} 0 0
[junit4:junit4]   2> 2238143 T6764 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435852028519645184)]} 0 0
[junit4:junit4]   2> 2238143 T6730 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1435852028519645184)]} 0 2
[junit4:junit4]   2> 2238146 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1435852028524888064)]} 0 0
[junit4:junit4]   2> 2238155 T6765 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435852028531179520)]} 0 0
[junit4:junit4]   2> 2238156 T6728 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1435852028531179520)]} 0 4
[junit4:junit4]   2> 2238156 T6748 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13]} 0 6
[junit4:junit4]   2> 2238159 T6710 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1435852028538519552)]} 0 0
[junit4:junit4]   2> 2238165 T6760 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435852028541665280)]} 0 0
[junit4:junit4]   2> 2238167 T6731 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1435852028541665280)]} 0 5
[junit4:junit4]   2> 2238169 T6711 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1435852028549005312)]} 0 0
[junit4:junit4]   2> 2238180 T6761 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435852028555296768)]} 0 1
[junit4:junit4]   2> 2238181 T6732 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1435852028555296768)]} 0 6
[junit4:junit4]   2> 2238181 T6744 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15]} 0 9
[junit4:junit4]   2> 2238184 T6713 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1435852028564733952)]} 0 0
[junit4:junit4]   2> 2238194 T6762 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435852028571025408)]} 0 1
[junit4:junit4]   2> 2238194 T6733 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1435852028571025408)]} 0 4
[junit4:junit4]   2> 2238195 T6749 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16]} 0 8
[junit4:junit4]   2> 2238198 T6712 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1435852028578365440)]} 0 0
[junit4:junit4]   2> 2238207 T6777 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435852028585705472)]} 0 0
[junit4:junit4]   2> 2238208 T6745 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[17 (1435852028585705472)]} 0 4
[junit4:junit4]   2> 2238209 T6729 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17]} 0 8
[junit4:junit4]   2> 2238211 T6708 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1435852028593045504)]} 0 0
[junit4:junit4]   2> 2238219 T6779 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435852028599336960)]} 0 0
[junit4:junit4]   2> 2238220 T6746 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1435852028599336960)]} 0 3
[junit4:junit4]   2> 2238220 T6730 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18]} 0 5
[junit4:junit4]   2> 2238225 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1435852028606676992)]} 0 1
[junit4:junit4]   2> 2238233 T6780 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435852028614017024)]} 0 0
[junit4:junit4]   2> 2238235 T6747 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[19 (1435852028614017024)]} 0 4
[junit4:junit4]   2> 2238236 T6728 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19]} 0 8
[junit4:junit4]   2> 2238238 T6710 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1435852028621357056)]} 0 0
[junit4:junit4]   2> 2238246 T6763 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435852028627648512)]} 0 0
[junit4:junit4]   2> 2238247 T6731 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[20 (1435852028627648512)]} 0 3
[junit4:junit4]   2> 2238248 T6748 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20]} 0 6
[junit4:junit4]   2> 2238250 T6711 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1435852028633939968)]} 0 0
[junit4:junit4]   2> 2238256 T6781 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435852028637085696)]} 0 0
[junit4:junit4]   2> 2238257 T6744 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1435852028637085696)]} 0 4
[junit4:junit4]   2> 2238259 T6713 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1435852028643377152)]} 0 0
[junit4:junit4]   2> 2238267 T6764 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435852028648620032)]} 0 0
[junit4:junit4]   2> 2238268 T6732 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[22 (1435852028648620032)]} 0 4
[junit4:junit4]   2> 2238269 T6749 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22]} 0 7
[junit4:junit4]   2> 2238271 T6712 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1435852028655960064)]} 0 0
[junit4:junit4]   2> 2238279 T6782 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435852028662251520)]} 0 0
[junit4:junit4]   2> 2238280 T6745 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[23 (1435852028662251520)]} 0 3
[junit4:junit4]   2> 2238281 T6733 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23]} 0 7
[junit4:junit4]   2> 2238284 T6708 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1435852028669591552)]} 0 0
[junit4:junit4]   2> 2238291 T6765 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435852028674834432)]} 0 0
[junit4:junit4]   2> 2238292 T6729 C4954 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1435852028674834432)]} 0 3
[junit4:junit4]   2> 2238296 T6746 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24]} 0 9
[junit4:junit4]   2> 2238303 T6709 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435852028688465920)]} 0 2
[junit4:junit4]   2> 2238309 T6778 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435852028692660224)]} 0 0
[junit4:junit4]   2> 2238310 T6747 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435852028692660224)]} 0 4
[junit4:junit4]   2> 2238312 T6710 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435852028698951680)]} 0 0
[junit4:junit4]   2> 2238319 T6777 C4955 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435852028703145984)]} 0 0
[junit4:junit4]   2> 2238320 T6748 C4956 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435852028703145984)]} 0 4
[junit4:junit4]   2> 2238322 T6711 C4957 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1435852028709437440)]} 0 0
[junit4:junit4]   2> 2238329 T6760 C4953 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435852028713631744)]} 0 0
[junit4:junit4]   2> ASYNC  NEW_CORE C4958 name=collection1 org.apache.solr.core.SolrCore< at >7ba30948 url=http://127.0.0.1:63381/collection1 node=127.0.0.1:63381_ C4958_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63381_, base_url=http://127.0.0.1:63381, leader=true}
[junit4:junit4]   2> 2238331 T6730 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1435852028713631744)]} 0 5
[junit4:junit4]   2> ASYNC  NEW_CORE C4959 name=collection1 org.apache.solr.core.SolrCore< at >6bf9772d url=http://127.0.0.1:63369/collection1 node=127.0.0.1:63369_ C4959_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:63369_, base_url=http://127.0.0.1:63369, leader=true}
[junit4:junit4]   2> 2238350 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1435852028737748992)]} 0 1
[junit4:junit4]   2> ASYNC  NEW_CORE C4960 name=collection1 org.apache.solr.core.SolrCore< at >640524a1 url=http://127.0.0.1:63399/collection1 node=127.0.0.1:63399_ C4960_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63399_, base_url=http://127.0.0.1:63399}
[junit4:junit4]   2> 2238356 T6761 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435852028741943296)]} 0 0
[junit4:junit4]   2> 2238357 T6728 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1435852028741943296)]} 0 4
[junit4:junit4]   2> 2238360 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1435852028749283328)]} 0 0
[junit4:junit4]   2> ASYNC  NEW_CORE C4961 name=collection1 org.apache.solr.core.SolrCore< at >5172174b url=http://127.0.0.1:63409/collection1 node=127.0.0.1:63409_ C4961_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63409_, base_url=http://127.0.0.1:63409}
[junit4:junit4]   2> 2238368 T6779 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435852028755574784)]} 0 0
[junit4:junit4]   2> ASYNC  NEW_CORE C4962 name=collection1 org.apache.solr.core.SolrCore< at >5ac883c8 url=http://127.0.0.1:63390/collection1 node=127.0.0.1:63390_ C4962_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:63390_, base_url=http://127.0.0.1:63390, leader=true}
[junit4:junit4]   2> 2238369 T6744 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1435852028755574784)]} 0 3
[junit4:junit4]   2> 2238370 T6731 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29]} 0 7
[junit4:junit4]   2> 2238373 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1435852028762914816)]} 0 0
[junit4:junit4]   2> 2238379 T6780 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435852028766060544)]} 0 0
[junit4:junit4]   2> 2238380 T6749 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1435852028766060544)]} 0 4
[junit4:junit4]   2> 2238383 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1435852028773400576)]} 0 0
[junit4:junit4]   2> 2238389 T6781 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435852028777594880)]} 0 0
[junit4:junit4]   2> 2238391 T6745 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1435852028777594880)]} 0 4
[junit4:junit4]   2> 2238392 T6732 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31]} 0 6
[junit4:junit4]   2> 2238394 T6710 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1435852028784934912)]} 0 0
[junit4:junit4]   2> 2238401 T6762 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435852028790177792)]} 0 0
[junit4:junit4]   2> 2238402 T6729 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[32 (1435852028790177792)]} 0 3
[junit4:junit4]   2> 2238403 T6747 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32]} 0 6
[junit4:junit4]   2> 2238406 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1435852028797517824)]} 0 0
[junit4:junit4]   2> 2238412 T6778 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1435852028800663552)]} 0 0
[junit4:junit4]   2> 2238412 T6748 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1435852028800663552)]} 0 3
[junit4:junit4]   2> 2238415 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435852028806955008)]} 0 0
[junit4:junit4]   2> 2238422 T6764 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1435852028810100736)]} 0 0
[junit4:junit4]   2> 2238423 T6730 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435852028810100736)]} 0 5
[junit4:junit4]   2> 2238425 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1435852028817440768)]} 0 0
[junit4:junit4]   2> 2238433 T6765 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1435852028822683648)]} 0 0
[junit4:junit4]   2> 2238434 T6728 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[35 (1435852028822683648)]} 0 4
[junit4:junit4]   2> 2238435 T6744 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35]} 0 7
[junit4:junit4]   2> 2238437 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1435852028830023680)]} 0 0
[junit4:junit4]   2> 2238442 T6760 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1435852028832120832)]} 0 0
[junit4:junit4]   2> 2238444 T6731 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1435852028832120832)]} 0 5
[junit4:junit4]   2> 2238447 T6711 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1435852028840509440)]} 0 0
[junit4:junit4]   2> 2238453 T6777 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1435852028844703744)]} 0 0
[junit4:junit4]   2> 2238454 T6749 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[37 (1435852028844703744)]} 0 3
[junit4:junit4]   2> 2238455 T6733 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37]} 0 5
[junit4:junit4]   2> 2238457 T6710 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435852028850995200)]} 0 0
[junit4:junit4]   2> 2238462 T6761 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1435852028854140928)]} 0 0
[junit4:junit4]   2> 2238463 T6732 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435852028854140928)]} 0 3
[junit4:junit4]   2> 2238465 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1435852028859383808)]} 0 0
[junit4:junit4]   2> 2238474 T6779 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1435852028865675264)]} 0 0
[junit4:junit4]   2> 2238475 T6746 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[39 (1435852028865675264)]} 0 4
[junit4:junit4]   2> 2238476 T6729 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39]} 0 7
[junit4:junit4]   2> 2238478 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1435852028873015296)]} 0 0
[junit4:junit4]   2> 2238485 T6763 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1435852028878258176)]} 0 0
[junit4:junit4]   2> 2238486 T6730 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1435852028878258176)]} 0 3
[junit4:junit4]   2> 2238487 T6745 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40]} 0 6
[junit4:junit4]   2> 2238489 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1435852028884549632)]} 0 0
[junit4:junit4]   2> 2238494 T6762 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1435852028887695360)]} 0 0
[junit4:junit4]   2> 2238495 T6728 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1435852028887695360)]} 0 3
[junit4:junit4]   2> 2238498 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1435852028893986816)]} 0 0
[junit4:junit4]   2> 2238502 T6764 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1435852028897132544)]} 0 0
[junit4:junit4]   2> 2238503 T6731 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1435852028897132544)]} 0 2
[junit4:junit4]   2> 2238506 T6711 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1435852028902375424)]} 0 0
[junit4:junit4]   2> 2238513 T6765 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1435852028907618304)]} 0 0
[junit4:junit4]   2> 2238514 T6733 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[43 (1435852028907618304)]} 0 3
[junit4:junit4]   2> 2238515 T6747 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43]} 0 6
[junit4:junit4]   2> 2238517 T6710 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1435852028913909760)]} 0 0
[junit4:junit4]   2> 2238525 T6760 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1435852028920201216)]} 0 0
[junit4:junit4]   2> 2238526 T6732 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1435852028920201216)]} 0 3
[junit4:junit4]   2> 2238527 T6748 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44]} 0 6
[junit4:junit4]   2> 2238529 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1435852028926492672)]} 0 0
[junit4:junit4]   2> 2238537 T6780 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1435852028931735552)]} 0 0
[junit4:junit4]   2> 2238538 T6744 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[45 (1435852028931735552)]} 0 4
[junit4:junit4]   2> 2238539 T6729 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45]} 0 7
[junit4:junit4]   2> 2238541 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1435852028939075584)]} 0 0
[junit4:junit4]   2> 2238547 T6761 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1435852028943269888)]} 0 0
[junit4:junit4]   2> 2238548 T6730 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[46 (1435852028943269888)]} 0 3
[junit4:junit4]   2> 2238549 T6749 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46]} 0 6
[junit4:junit4]   2> 2238552 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1435852028950609920)]} 0 0
[junit4:junit4]   2> 2238558 T6763 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1435852028954804224)]} 0 0
[junit4:junit4]   2> 2238560 T6728 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1435852028954804224)]} 0 4
[junit4:junit4]   2> 2238560 T6746 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47]} 0 6
[junit4:junit4]   2> 2238564 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1435852028963192832)]} 0 0
[junit4:junit4]   2> 2238570 T6762 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1435852028967387136)]} 0 0
[junit4:junit4]   2> 2238571 T6731 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[48 (1435852028967387136)]} 0 3
[junit4:junit4]   2> 2238572 T6745 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48]} 0 6
[junit4:junit4]   2> 2238575 T6711 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1435852028974727168)]} 0 0
[junit4:junit4]   2> 2238581 T6764 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1435852028978921472)]} 0 0
[junit4:junit4]   2> 2238582 T6733 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1435852028978921472)]} 0 3
[junit4:junit4]   2> 2238583 T6747 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49]} 0 5
[junit4:junit4]   2> 2238586 T6710 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435852028986261504)]} 0 0
[junit4:junit4]   2> 2238592 T6765 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1435852028990455808)]} 0 0
[junit4:junit4]   2> 2238593 T6732 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435852028990455808)]} 0 3
[junit4:junit4]   2> 2238596 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1435852028996747264)]} 0 0
[junit4:junit4]   2> 2238602 T6760 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1435852028999892992)]} 0 0
[junit4:junit4]   2> 2238603 T6729 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1435852028999892992)]} 0 4
[junit4:junit4]   2> 2238605 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1435852029006184448)]} 0 0
[junit4:junit4]   2> 2238611 T6761 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1435852029009330176)]} 0 1
[junit4:junit4]   2> 2238612 T6730 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1435852029009330176)]} 0 4
[junit4:junit4]   2> 2238614 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1435852029015621632)]} 0 0
[junit4:junit4]   2> 2238618 T6782 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1435852029018767360)]} 0 0
[junit4:junit4]   2> 2238619 T6748 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1435852029018767360)]} 0 2
[junit4:junit4]   2> 2238622 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1435852029024010240)]} 0 0
[junit4:junit4]   2> 2238629 T6763 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1435852029029253120)]} 0 0
[junit4:junit4]   2> 2238630 T6728 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[54 (1435852029029253120)]} 0 3
[junit4:junit4]   2> 2238630 T6744 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54]} 0 5
[junit4:junit4]   2> 2238634 T6711 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1435852029036593152)]} 0 0
[junit4:junit4]   2> 2238642 T6762 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1435852029041836032)]} 0 0
[junit4:junit4]   2> 2238643 T6731 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1435852029041836032)]} 0 4
[junit4:junit4]   2> 2238643 T6749 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55]} 0 6
[junit4:junit4]   2> 2238646 T6710 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1435852029049176064)]} 0 0
[junit4:junit4]   2> 2238654 T6781 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1435852029054418944)]} 0 0
[junit4:junit4]   2> 2238656 T6746 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[56 (1435852029054418944)]} 0 5
[junit4:junit4]   2> 2238656 T6733 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56]} 0 7
[junit4:junit4]   2> 2238658 T6713 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435852029061758976)]} 0 0
[junit4:junit4]   2> 2238664 T6778 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1435852029064904704)]} 0 0
[junit4:junit4]   2> 2238665 T6745 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435852029064904704)]} 0 4
[junit4:junit4]   2> 2238668 T6712 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1435852029072244736)]} 0 0
[junit4:junit4]   2> 2238674 T6777 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1435852029075390464)]} 0 0
[junit4:junit4]   2> 2238675 T6747 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1435852029075390464)]} 0 4
[junit4:junit4]   2> 2238678 T6708 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435852029082730496)]} 0 0
[junit4:junit4]   2> 2238682 T6764 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1435852029085876224)]} 0 0
[junit4:junit4]   2> 2238684 T6732 C4958 P63381 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435852029085876224)]} 0 3
[junit4:junit4]   2> 2238686 T6709 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1435852029091119104)]} 0 0
[junit4:junit4]   2> 2238692 T6779 C4961 P63409 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63390/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1435852029094264832)]} 0 0
[junit4:junit4]   2> 2238694 T6748 C4962 P63390 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1435852029094264832)]} 0 5
[junit4:junit4]   2> 2238696 T6711 C4959 P63369 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1435852029101604864)]} 0 0
[junit4:junit4]   2> 2238704 T6765 C4960 P63399 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:63381/collection1/&upda

[...truncated too long message...]

:junit4]   2> 2266351 T6786 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2267354 T6786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2267354 T6786 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2267354 T6695 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 63369
[junit4:junit4]   2> 2267368 T6695 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 2267369 T6695 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:63362 63362
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=531B68B6094DCB94 -Dtests.slow=true -Dtests.locale=sk_SK -Dtests.timezone=Africa/Ndjamena -Dtests.file.encoding=Cp1252
[junit4:junit4] ERROR   58.9s | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:63381 returned non ok status:500, message:Server Error
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([531B68B6094DCB94:D2FDE6AE7E12ABA8]:0)
[junit4:junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]    > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4]    > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 2267379 T6695 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {id=MockFixedIntBlock(blockSize=1067), _version_=Pulsing41(freqCutoff=17 minBlockSize=83 maxBlockSize=164)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=sk_SK, timezone=Africa/Ndjamena
[junit4:junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=1,free=124606784,total=236478464
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestWordDelimiterFilterFactory, TestDynamicFieldCollectionResource, OpenExchangeRatesOrgProviderTest, IndexSchemaTest, TestSearchPerf, TestCSVResponseWriter, CSVRequestHandlerTest, TestRandomDVFaceting, HardAutoCommitTest, AnalysisAfterCoreReloadTest, UnloadDistributedZkTest, PingRequestHandlerTest, TestRecovery, SpatialFilterTest, OutputWriterTest, TestClassNameShortening, TestReplicationHandler, TestHashPartitioner, TestTrie, TestRealTimeGet, CoreAdminHandlerTest, IndexBasedSpellCheckerTest, TestFieldTypeCollectionResource, ConvertedLegacyTest, DirectSolrSpellCheckerTest, TestLFUCache, TestSchemaVersionResource, TestNumberUtils, SortByFunctionTest, ExternalFileFieldSortTest, DateMathParserTest, FieldAnalysisRequestHandlerTest, SuggesterFSTTest, TestArbitraryIndexDir, DocValuesTest, ChaosMonkeyNothingIsSafeTest, SuggesterTest, RequestHandlersTest, UpdateRequestProcessorFactoryTest, TestPerFieldSimilarity, TestPseudoReturnFields, DocumentBuilderTest, BadComponentTest, TermsComponentTest, SolrIndexSplitterTest, ScriptEngineTest, OverseerCollectionProcessorTest, TestFoldingMultitermQuery, TestPluginEnable, BasicDistributedZkTest, UUIDFieldTest, TestFieldResource, TestLazyCores, ZkCLITest, TestMultiCoreConfBootstrap, TestFastOutputStream, TestConfig, TestValueSourceCache, MinimalSchemaTest, TestIndexSearcher, TestCloudManagedSchema, SolrPluginUtilsTest, TestSolrQueryParser, QueryParsingTest, MultiTermTest, CircularListTest, TestShardHandlerFactory, OverseerTest, TestFastWriter, SpellCheckCollatorTest, TestStressLucene, RegexBoostProcessorTest, SuggesterWFSTTest, TestSchemaNameResource, SpellingQueryConverterTest, CoreContainerCoreInitFailuresTest, TestJoin, CachingDirectoryFactoryTest, RAMDirectoryFactoryTest, TestDefaultSimilarityFactory, TestPropInjectDefaults, SearchHandlerTest, TestSort, XmlUpdateRequestHandlerTest, ZkControllerTest, TestSolrQueryParserDefaultOperatorResource, DocValuesMultiTest, DistributedSpellCheckComponentTest, BasicZkTest, PreAnalyzedFieldTest, TestDistributedSearch, SuggesterTSTTest, TestReversedWildcardFilterFactory, TestUpdate, TestSchemaResource, EchoParamsTest, BasicFunctionalityTest, TestComponentsName, TestIndexingPerformance, LukeRequestHandlerTest, TestAtomicUpdateErrorCases, SystemInfoHandlerTest, ChaosMonkeySafeLeaderTest, StandardRequestHandlerTest, TestFieldTypeResource, PathHierarchyTokenizerFactoryTest, FullSolrCloudDistribCmdsTest, SOLR749Test, PrimUtilsTest, SolrRequestParserTest, ResponseLogComponentTest, TestIBSimilarityFactory, JsonLoaderTest, LoggingHandlerTest, AliasIntegrationTest, DistributedQueryElevationComponentTest, TestPropInject, TestSolrXMLSerializer, BasicDistributedZk2Test, NoCacheHeaderTest, JSONWriterTest, TestRTGBase, TestMergePolicyConfig, LeaderElectionIntegrationTest, TestCopyFieldCollectionResource, TestSolrQueryParserResource, DirectUpdateHandlerOptimizeTest, TestCSVLoader, TestPostingsSolrHighlighter, ClusterStateTest, TestGroupingSearch, FastVectorHighlighterTest, TestExtendedDismaxParser, TestBM25SimilarityFactory, UniqFieldsUpdateProcessorFactoryTest, QueryResultKeyTest, TestStressRecovery, QueryEqualityTest, DirectUpdateHandlerTest, TestCodecSupport, BadIndexSchemaTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, ShardRoutingCustomTest, SliceStateUpdateTest, AutoCommitTest, TestDistributedGrouping, TestFaceting, TermVectorComponentDistributedTest, TestStressReorder, TestReload, TestStressVersions, DistributedTermsComponentTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, TestFiltering, TestFunctionQuery, TestBadConfig, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, CurrencyFieldXmlFileTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, FileBasedSpellCheckerTest, TermVectorComponentTest, MoreLikeThisHandlerTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, TestJmxIntegration, TestCollationField, ReturnFieldsTest, MBeansHandlerTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, TestBinaryResponseWriter, HighlighterConfigTest, TestQuerySenderListener, IndexReaderFactoryTest, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, TestSystemIdResolver, TestSuggestSpellingConverter, DateFieldTest, DOMUtilTest, TestSolrJ, TestLRUCache, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, FileUtilsTest, SolrTestCaseJ4Test, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed in 58.96s, 1 test, 1 error <<< FAILURES!

[...truncated 54 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:380: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:360: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:181: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:437: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1240: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 error, 18 ignored (12 assumptions)

Total time: 62 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe< at >lucene.apache.org
For additional commands, e-mail: dev-help< at >lucene.apache.org

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 5733 - Failure!

Policeman Jenkins Server — 2013-05-23T17:19:46

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/5733/
Java: 32bit/jdk1.6.0_45 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 230 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
at __randomizedtesting.SeedInfo.seed([3198F45C014BF633:B07E7A447614960F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 9038 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4:junit4]   2> 425859 T1145 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_ml/
[junit4:junit4]   2> 425862 T1145 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1369328642833
[junit4:junit4]   2> 425863 T1145 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 425863 T1146 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 425963 T1145 oasc.ZkTestServer.run start zk server on port:41518
[junit4:junit4]   2> 425964 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426266 T1152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >a0e67c name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426266 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426267 T1145 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 426271 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426272 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1c33175 name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426273 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426273 T1145 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 426276 T1145 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 426279 T1145 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 426281 T1145 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 426283 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 426284 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 426288 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 426289 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 426292 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 426293 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 426295 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 426296 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 426299 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 426299 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 426302 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 426303 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 426305 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 426306 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 426309 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 426310 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 426312 T1145 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 426312 T1145 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 426416 T1145 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 426419 T1145 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:46766
[junit4:junit4]   2> 426420 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 426420 T1145 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 426421 T1145 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285
[junit4:junit4]   2> 426421 T1145 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/solr.xml
[junit4:junit4]   2> 426422 T1145 oasc.CoreContainer.<init> New CoreContainer 2882690
[junit4:junit4]   2> 426423 T1145 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/'
[junit4:junit4]   2> 426423 T1145 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/'
[junit4:junit4]   2> 426491 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285
[junit4:junit4]   2> 426492 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/collection1
[junit4:junit4]   2> 426493 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/collection1/conf
[junit4:junit4]   2> 426501 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/collection1/conf/xslt
[junit4:junit4]   2> 426530 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/collection1/lib
[junit4:junit4]   2> 426531 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/collection1/lib/classes
[junit4:junit4]   2> 426533 T1145 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/conf/core.properties, adding to cores
[junit4:junit4]   2> 426537 T1145 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 426537 T1145 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 426538 T1145 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 426538 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 426539 T1145 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 426539 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 426540 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 426540 T1145 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 426541 T1145 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 426541 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 426546 T1145 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 426547 T1145 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41518/solr
[junit4:junit4]   2> 426547 T1145 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 426548 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426550 T1165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >185824c name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426550 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426552 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 426556 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426557 T1167 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >c38614 name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426558 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426559 T1145 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 426562 T1145 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 426565 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 426567 T1145 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46766__ml
[junit4:junit4]   2> 426568 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46766__ml
[junit4:junit4]   2> 426571 T1145 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 426576 T1145 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 426578 T1145 oasc.Overseer.start Overseer (id=89740321936834563-127.0.0.1:46766__ml-n_0000000000) starting
[junit4:junit4]   2> 426581 T1145 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 426584 T1169 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 426584 T1145 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 426586 T1145 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 426588 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 426590 T1168 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 426593 T1145 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 426594 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 426594 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 426598 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 426599 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426601 T1172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >d55aac name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426601 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426602 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 426604 T1145 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 426675 T1145 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 426681 T1145 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:45571
[junit4:junit4]   2> 426682 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 426682 T1145 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 426682 T1145 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575
[junit4:junit4]   2> 426683 T1145 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/solr.xml
[junit4:junit4]   2> 426683 T1145 oasc.CoreContainer.<init> New CoreContainer 16970409
[junit4:junit4]   2> 426683 T1145 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/'
[junit4:junit4]   2> 426684 T1145 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/'
[junit4:junit4]   2> 426725 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575
[junit4:junit4]   2> 426726 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/collection1
[junit4:junit4]   2> 426726 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/collection1/conf
[junit4:junit4]   2> 426731 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/collection1/conf/xslt
[junit4:junit4]   2> 426747 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/collection1/lib
[junit4:junit4]   2> 426747 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/collection1/lib/classes
[junit4:junit4]   2> 426749 T1145 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/conf/core.properties, adding to cores
[junit4:junit4]   2> 426751 T1145 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 426751 T1145 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 426751 T1145 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 426752 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 426752 T1145 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 426752 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 426752 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 426753 T1145 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 426753 T1145 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 426753 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 426756 T1145 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 426757 T1145 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41518/solr
[junit4:junit4]   2> 426757 T1145 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 426758 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426759 T1183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6c23be name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426759 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426761 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 426764 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 426766 T1185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1bd77f name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 426766 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 426770 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 427773 T1145 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45571__ml
[junit4:junit4]   2> 427774 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45571__ml
[junit4:junit4]   2> 427777 T1185 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 427777 T1172 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 427777 T1167 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 427782 T1145 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 427783 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 427784 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 427855 T1145 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 427857 T1145 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:52140
[junit4:junit4]   2> 427857 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 427857 T1145 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 427858 T1145 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757
[junit4:junit4]   2> 427858 T1145 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/solr.xml
[junit4:junit4]   2> 427858 T1145 oasc.CoreContainer.<init> New CoreContainer 14497045
[junit4:junit4]   2> 427859 T1145 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/'
[junit4:junit4]   2> 427859 T1145 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/'
[junit4:junit4]   2> 427912 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757
[junit4:junit4]   2> 427913 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/collection1
[junit4:junit4]   2> 427914 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/collection1/conf
[junit4:junit4]   2> 427918 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/collection1/conf/xslt
[junit4:junit4]   2> 427934 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/collection1/lib
[junit4:junit4]   2> 427935 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/collection1/lib/classes
[junit4:junit4]   2> 427936 T1145 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/conf/core.properties, adding to cores
[junit4:junit4]   2> 427938 T1145 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 427938 T1145 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 427939 T1145 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 427939 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 427939 T1145 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 427939 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 427940 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 427940 T1145 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 427940 T1145 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 427941 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 427943 T1145 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 427944 T1145 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41518/solr
[junit4:junit4]   2> 427944 T1145 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 427945 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 427946 T1197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1d09e59 name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 427947 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 427948 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 427951 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 427952 T1199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >45bf2d name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 427953 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 427956 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 428959 T1145 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52140__ml
[junit4:junit4]   2> 428960 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52140__ml
[junit4:junit4]   2> 428962 T1185 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 428962 T1199 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 428962 T1167 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 428962 T1172 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 428966 T1145 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 428966 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 428966 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 429031 T1145 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 429033 T1145 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:52879
[junit4:junit4]   2> 429033 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 429034 T1145 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 429034 T1145 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938
[junit4:junit4]   2> 429034 T1145 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/solr.xml
[junit4:junit4]   2> 429035 T1145 oasc.CoreContainer.<init> New CoreContainer 8486515
[junit4:junit4]   2> 429035 T1145 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/'
[junit4:junit4]   2> 429035 T1145 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/'
[junit4:junit4]   2> 429076 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938
[junit4:junit4]   2> 429076 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/collection1
[junit4:junit4]   2> 429077 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/collection1/conf
[junit4:junit4]   2> 429081 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/collection1/conf/xslt
[junit4:junit4]   2> 429097 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/collection1/lib
[junit4:junit4]   2> 429098 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/collection1/lib/classes
[junit4:junit4]   2> 429099 T1145 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/conf/core.properties, adding to cores
[junit4:junit4]   2> 429101 T1145 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 429101 T1145 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 429102 T1145 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 429102 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 429102 T1145 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 429103 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 429103 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 429103 T1145 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 429103 T1145 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 429104 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 429107 T1145 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 429107 T1145 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41518/solr
[junit4:junit4]   2> 429107 T1145 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 429108 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 429109 T1211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >18a5e8e name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 429110 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 429111 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 429114 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 429115 T1213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >161a4cf name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 429115 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 429118 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 430120 T1145 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52879__ml
[junit4:junit4]   2> 430122 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52879__ml
[junit4:junit4]   2> 430125 T1185 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 430125 T1199 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 430125 T1213 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 430125 T1167 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 430125 T1172 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 430129 T1145 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 430129 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 430130 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 430214 T1145 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 430216 T1145 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:42212
[junit4:junit4]   2> 430216 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 430216 T1145 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 430217 T1145 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102
[junit4:junit4]   2> 430217 T1145 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/solr.xml
[junit4:junit4]   2> 430217 T1145 oasc.CoreContainer.<init> New CoreContainer 31468389
[junit4:junit4]   2> 430218 T1145 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/'
[junit4:junit4]   2> 430218 T1145 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/'
[junit4:junit4]   2> 430263 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102
[junit4:junit4]   2> 430265 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/collection1
[junit4:junit4]   2> 430265 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/collection1/conf
[junit4:junit4]   2> 430274 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/collection1/conf/xslt
[junit4:junit4]   2> 430305 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/collection1/lib
[junit4:junit4]   2> 430306 T1145 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/collection1/lib/classes
[junit4:junit4]   2> 430308 T1145 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/conf/core.properties, adding to cores
[junit4:junit4]   2> 430312 T1145 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 430312 T1145 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 430313 T1145 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 430313 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 430314 T1145 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 430314 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 430314 T1145 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 430315 T1145 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 430315 T1145 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 430315 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 430318 T1145 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 430319 T1145 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41518/solr
[junit4:junit4]   2> 430319 T1145 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 430320 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 430321 T1225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1a54605 name:ZooKeeperConnection Watcher:127.0.0.1:41518 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 430322 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 430323 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 430326 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 430327 T1227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >7db50a name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 430327 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 430331 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 431333 T1145 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42212__ml
[junit4:junit4]   2> 431334 T1145 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42212__ml
[junit4:junit4]   2> 431338 T1172 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431339 T1167 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431339 T1199 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431338 T1213 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431339 T1227 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431339 T1185 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 431345 T1145 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 431345 T1145 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 431346 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 431348 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 431351 T1179 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 431352 T1167 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 431353 T1169 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4:junit4]   2>   "operation":"createcollection",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "maxShardsPerNode":null,
[junit4:junit4]   2>   "createNodeSet":null,
[junit4:junit4]   2>   "name":"nodes_used_collection",
[junit4:junit4]   2>   "replicationFactor":"2"}
[junit4:junit4]   2> 431353 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:42212__ml
[junit4:junit4]   2> 431353 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:52140__ml
[junit4:junit4]   2> 431354 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:46766__ml
[junit4:junit4]   2> 431354 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:52879__ml
[junit4:junit4]   2> 431355 T1220 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 431355 T1161 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 431356 T1161 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 431356 T1194 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 431355 T1220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 431357 T1194 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 431357 T1161 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 431358 T1161 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 431358 T1206 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 431358 T1206 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 431359 T1161 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 431359 T1161 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 431358 T1220 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 431358 T1194 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 431360 T1220 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 431359 T1206 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 431360 T1194 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 431361 T1206 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 431362 T1161 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 431362 T1220 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 431362 T1194 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 431363 T1206 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 431364 T1220 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 431364 T1161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/'
[junit4:junit4]   2> 431364 T1194 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 431366 T1206 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 431366 T1220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/'
[junit4:junit4]   2> 431366 T1194 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/'
[junit4:junit4]   2> 431368 T1206 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/'
[junit4:junit4]   2> 431415 T1206 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 431415 T1161 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 431415 T1220 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 431425 T1194 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 431467 T1206 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 431468 T1206 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 431468 T1194 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 431470 T1220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 431473 T1220 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 431474 T1161 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 431474 T1194 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 431475 T1161 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 431476 T1206 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
[junit4:junit4]   2> 431479 T1220 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
[junit4:junit4]   2> 431481 T1194 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
[junit4:junit4]   2> 431482 T1161 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
[junit4:junit4]   2> 432062 T1220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 432071 T1220 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 432076 T1220 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 432091 T1220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432095 T1220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432097 T1161 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 432099 T1220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432101 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432102 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432102 T1220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432104 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432104 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432105 T1220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432105 T1220 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data/
[junit4:junit4]   2> 432105 T1161 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 432106 T1220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 432107 T1220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432108 T1220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data
[junit4:junit4]   2> 432108 T1220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data/index/
[junit4:junit4]   2> 432108 T1220 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 432109 T1220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data/index
[junit4:junit4]   2> 432110 T1161 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 432111 T1220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1d68af5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >ecdc51),segFN=segments_1,generation=1}
[junit4:junit4]   2> 432112 T1220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 432114 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 432115 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 432116 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 432117 T1220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 432118 T1220 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 432118 T1220 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 432119 T1220 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 432120 T1220 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 432120 T1220 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 432124 T1220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 432126 T1161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432127 T1220 oass.SolrIndexSearcher.<init> Opening Searcher< at >1637b37 main
[junit4:junit4]   2> 432128 T1220 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/nodes_used_collection_shard1_replica1/data/tlog
[junit4:junit4]   2> 432129 T1220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 432129 T1220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 432132 T1161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432135 T1233 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher< at >1637b37 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 432137 T1161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432138 T1220 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
[junit4:junit4]   2> 432138 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432139 T1206 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 432140 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432140 T1161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432142 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432143 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432143 T1161 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432143 T1161 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data/
[junit4:junit4]   2> 432144 T1161 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 432145 T1161 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432145 T1161 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data
[junit4:junit4]   2> 432146 T1206 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 432146 T1161 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data/index/
[junit4:junit4]   2> 432146 T1161 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 432147 T1161 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data/index
[junit4:junit4]   2> 432149 T1206 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 432149 T1161 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >9bd99f lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1cce50e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 432150 T1161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 432152 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 432153 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 432153 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 432154 T1161 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 432155 T1161 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 432155 T1161 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 432156 T1161 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 432157 T1161 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 432157 T1161 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 432160 T1194 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 432160 T1161 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 432161 T1206 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432163 T1161 oass.SolrIndexSearcher.<init> Opening Searcher< at >1dff693 main
[junit4:junit4]   2> 432164 T1161 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/nodes_used_collection_shard2_replica1/data/tlog
[junit4:junit4]   2> 432165 T1161 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 432165 T1161 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 432167 T1206 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432168 T1194 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 432171 T1161 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
[junit4:junit4]   2> 432172 T1206 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432172 T1234 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher< at >1dff693 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 432173 T1194 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 432175 T1206 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432175 T1206 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432176 T1206 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432178 T1206 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432178 T1206 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432179 T1206 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432179 T1206 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data/
[junit4:junit4]   2> 432180 T1206 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 432180 T1206 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432181 T1206 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data
[junit4:junit4]   2> 432182 T1206 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data/index/
[junit4:junit4]   2> 432182 T1206 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 432183 T1206 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data/index
[junit4:junit4]   2> 432186 T1206 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >c5b7ff lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >b0852c),segFN=segments_1,generation=1}
[junit4:junit4]   2> 432186 T1206 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 432189 T1206 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 432189 T1194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432190 T1206 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 432191 T1206 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 432192 T1206 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 432192 T1206 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 432193 T1206 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 432193 T1206 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 432194 T1206 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 432194 T1206 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 432196 T1194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 432197 T1206 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 432200 T1206 oass.SolrIndexSearcher.<init> Opening Searcher< at >1f5bc main
[junit4:junit4]   2> 432200 T1194 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432201 T1206 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/nodes_used_collection_shard2_replica2/data/tlog
[junit4:junit4]   2> 432202 T1194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432202 T1194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432202 T1206 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 432203 T1194 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 432203 T1206 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 432204 T1194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 432204 T1194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 432205 T1194 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432205 T1194 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data/
[junit4:junit4]   2> 432206 T1194 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 432206 T1194 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 432207 T1194 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data
[junit4:junit4]   2> 432207 T1235 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher< at >1f5bc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 432208 T1194 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data/index/
[junit4:junit4]   2> 432208 T1194 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 432209 T1194 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data/index
[junit4:junit4]   2> 432210 T1206 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
[junit4:junit4]   2> 432211 T1194 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >e2243c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >59e8a8),segFN=segments_1,generation=1}
[junit4:junit4]   2> 432212 T1194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 432214 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 432215 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 432215 T1194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 432216 T1194 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 432217 T1194 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 432217 T1194 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 432218 T1194 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 432218 T1194 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 432219 T1194 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 432222 T1194 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 432225 T1194 oass.SolrIndexSearcher.<init> Opening Searcher< at >1a40908 main
[junit4:junit4]   2> 432226 T1194 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/nodes_used_collection_shard1_replica2/data/tlog
[junit4:junit4]   2> 432227 T1194 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 432227 T1194 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 432231 T1236 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher< at >1a40908 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 432233 T1194 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
[junit4:junit4]   2> 432598 T1168 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 432599 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42212__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42212/_ml"}
[junit4:junit4]   2> 432599 T1168 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with numShards 2
[junit4:junit4]   2> 432603 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46766__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46766/_ml"}
[junit4:junit4]   2> 432606 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52879__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52879/_ml"}
[junit4:junit4]   2> 432610 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52140__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52140/_ml"}
[junit4:junit4]   2> 432614 T1227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 432614 T1167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 432614 T1199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 432614 T1213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 432614 T1172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 432614 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433141 T1220 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 433141 T1220 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:42212/_ml collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 433142 T1220 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
[junit4:junit4]   2> 433148 T1220 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 433150 T1220 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=180000
[junit4:junit4]   2> 433174 T1161 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 433174 T1161 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:46766/_ml collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 433175 T1161 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4]   2> 433181 T1161 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 433184 T1161 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179999
[junit4:junit4]   2> 433212 T1206 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 433212 T1206 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:52879/_ml collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 433234 T1194 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 433235 T1194 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:52140/_ml collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 433651 T1220 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 433652 T1220 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 433652 T1220 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42212/_ml/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 433653 T1220 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:42212/_ml START replicas=[http://127.0.0.1:52140/_ml/nodes_used_collection_shard1_replica2/] nUpdates=100
[junit4:junit4]   2> 433654 T1220 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:42212/_ml DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 433654 T1220 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 433654 T1220 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 433655 T1220 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42212/_ml/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 433655 T1220 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
[junit4:junit4]   2> ASYNC  NEW_CORE C1813 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >faf68b url=http://127.0.0.1:52140/_ml/nodes_used_collection_shard1_replica2 node=127.0.0.1:52140__ml C1813_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=down, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:52140__ml, base_url=http://127.0.0.1:52140/_ml}
[junit4:junit4]   2> 433660 T1192 C1813 P52140 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_ml path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 433685 T1161 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 433685 T1161 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 433686 T1161 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46766/_ml/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 433686 T1161 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:46766/_ml START replicas=[http://127.0.0.1:52879/_ml/nodes_used_collection_shard2_replica2/] nUpdates=100
[junit4:junit4]   2> 433687 T1161 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:46766/_ml DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 433687 T1161 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 433688 T1161 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 433688 T1161 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46766/_ml/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 433688 T1161 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
[junit4:junit4]   2> ASYNC  NEW_CORE C1814 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >2b26a4 url=http://127.0.0.1:52879/_ml/nodes_used_collection_shard2_replica2 node=127.0.0.1:52879__ml C1814_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=down, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:52879__ml, base_url=http://127.0.0.1:52879/_ml}
[junit4:junit4]   2> 433694 T1208 C1814 P52879 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_ml path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 434122 T1168 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 434136 T1227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434137 T1199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434137 T1172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434137 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434137 T1167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434137 T1213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 434161 T1220 oasc.ZkController.register We are http://127.0.0.1:42212/_ml/nodes_used_collection_shard1_replica1/ and leader is http://127.0.0.1:42212/_ml/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 434162 T1220 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica1 baseURL=http://127.0.0.1:42212/_ml
[junit4:junit4]   2> 434162 T1220 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 434163 T1220 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=active
[junit4:junit4]   2> 434164 T1220 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 434166 T1220 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2810 
[junit4:junit4]   2> 434194 T1161 oasc.ZkController.register We are http://127.0.0.1:46766/_ml/nodes_used_collection_shard2_replica1/ and leader is http://127.0.0.1:46766/_ml/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 434195 T1161 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica1 baseURL=http://127.0.0.1:46766/_ml
[junit4:junit4]   2> 434195 T1161 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 434195 T1161 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=active
[junit4:junit4]   2> 434198 T1161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 434199 T1161 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2844 
[junit4:junit4]   2> 434216 T1206 oasc.ZkController.register We are http://127.0.0.1:52879/_ml/nodes_used_collection_shard2_replica2/ and leader is http://127.0.0.1:46766/_ml/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 434216 T1206 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica2 baseURL=http://127.0.0.1:52879/_ml
[junit4:junit4]   2> 434217 T1206 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 434217 T1206 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 434217 T1239 C1814 P52879 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard2_replica2 recoveringAfterStartup=true
[junit4:junit4]   2> 434218 T1239 C1814 P52879 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 434218 T1206 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 434218 T1239 C1814 P52879 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=recovering
[junit4:junit4]   2> 434219 T1206 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2861 
[junit4:junit4]   2> 434220 T1239 C1814 P52879 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 434223 T1163 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52879__ml_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 434238 T1194 oasc.ZkController.register We are http://127.0.0.1:52140/_ml/nodes_used_collection_shard1_replica2/ and leader is http://127.0.0.1:42212/_ml/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 434239 T1194 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica2 baseURL=http://127.0.0.1:52140/_ml
[junit4:junit4]   2> 434239 T1194 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 434239 T1194 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 434240 T1240 C1813 P52140 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard1_replica2 recoveringAfterStartup=true
[junit4:junit4]   2> 434240 T1240 C1813 P52140 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 434240 T1194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 434240 T1240 C1813 P52140 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=recovering
[junit4:junit4]   2> 434241 T1194 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2885 
[junit4:junit4]   2> 434241 T1169 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: nodes_used_collection
[junit4:junit4]   2> 434242 T1240 C1813 P52140 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 434244 T1185 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
[junit4:junit4]   2> 434245 T1222 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52140__ml_nodes_used_collection_shard1_replica2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 434245 T1167 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 434245 T1169 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=2810},core=nodes_used_collection_shard1_replica1},null={responseHeader={status=0,QTime=2844},core=nodes_used_collection_shard2_replica1},null={responseHeader={status=0,QTime=2861},core=nodes_used_collection_shard2_replica2},null={responseHeader={status=0,QTime=2885},core=nodes_used_collection_shard1_replica2}}}
[junit4:junit4]   2> 434246 T1179 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2} status=0 QTime=2896 
[junit4:junit4]   2> 434247 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 434248 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 434258 T1242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1be28a8 name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 434258 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 434260 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 434262 T1145 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 434263 T1145 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 434264 T1244 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >94b5af name:ZooKeeperConnection Watcher:127.0.0.1:41518/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 434265 T1145 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 434266 T1145 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 434271 T1181 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=3&maxShardsPerNode=2&name=awholynewcollection_0&replicationFactor=3&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 434272 T1167 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 434274 T1169 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4:junit4]   2>   "operation":"createcollection",
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "maxShardsPerNode":"2",
[junit4:junit4]   2>   "createNodeSet":null,
[junit4:junit4]   2>   "name":"awholynewcollection_0",
[junit4:junit4]   2>   "replicationFactor":"3"}
[junit4:junit4]   2> 434274 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica1 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:52879__ml
[junit4:junit4]   2> 434274 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica2 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:45571__ml
[junit4:junit4]   2> 434274 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica3 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:46766__ml
[junit4:junit4]   2> 434275 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica1 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:42212__ml
[junit4:junit4]   2> 434275 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica2 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:52140__ml
[junit4:junit4]   2> 434275 T1179 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2
[junit4:junit4]   2> 434276 T1179 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434276 T1160 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3
[junit4:junit4]   2> 434276 T1160 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434277 T1192 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2
[junit4:junit4]   2> 434277 T1192 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434275 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica3 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:52879__ml
[junit4:junit4]   2> 434277 T1160 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434278 T1160 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434279 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica1 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:45571__ml
[junit4:junit4]   2> 434277 T1208 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1
[junit4:junit4]   2> 434280 T1208 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434277 T1223 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1
[junit4:junit4]   2> 434281 T1208 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434276 T1179 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434281 T1208 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434281 T1223 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434280 T1160 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 434283 T1160 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434280 T1207 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3
[junit4:junit4]   2> 434283 T1207 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434279 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica2 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:46766__ml
[junit4:junit4]   2> 434278 T1192 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434284 T1169 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica3 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:42212__ml
[junit4:junit4]   2> 434283 T1180 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1
[junit4:junit4]   2> 434285 T1162 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2
[junit4:junit4]   2> 434285 T1162 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434285 T1220 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3
[junit4:junit4]   2> 434286 T1220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434283 T1208 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 434283 T1223 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434282 T1179 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434287 T1223 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434287 T1208 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434286 T1220 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434288 T1220 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434288 T1179 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434286 T1162 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434289 T1162 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434285 T1160 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434285 T1207 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434290 T1207 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434285 T1180 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434284 T1192 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434291 T1162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/'
[junit4:junit4]   2> 434291 T1208 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434291 T1220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/'
[junit4:junit4]   2> 434288 T1223 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434292 T1180 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434293 T1180 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434294 T1208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/'
[junit4:junit4]   2> 434292 T1207 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/'
[junit4:junit4]   2> 434292 T1179 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434295 T1180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/'
[junit4:junit4]   2> 434295 T1160 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/'
[junit4:junit4]   2> 434293 T1192 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434300 T1223 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434300 T1179 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/'
[junit4:junit4]   2> 434304 T1192 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434309 T1223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/'
[junit4:junit4]   2> 434314 T1192 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/'
[junit4:junit4]   2> 434348 T1162 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434348 T1208 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434371 T1207 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434421 T1192 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434449 T1180 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434462 T1220 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434463 T1223 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434473 T1160 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434500 T1179 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 434507 T1162 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434511 T1208 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434514 T1162 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434514 T1208 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434531 T1162 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica2] Schema name=test
[junit4:junit4]   2> 434533 T1208 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica1] Schema name=test
[junit4:junit4]   2> 434531 T1207 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434562 T1207 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434563 T1223 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434568 T1207 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica3] Schema name=test
[junit4:junit4]   2> 434570 T1223 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434580 T1223 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica1] Schema name=test
[junit4:junit4]   2> 434592 T1160 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434594 T1160 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434594 T1220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434596 T1220 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434600 T1160 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica3] Schema name=test
[junit4:junit4]   2> 434602 T1220 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica3] Schema name=test
[junit4:junit4]   2> 434620 T1192 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434622 T1192 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434628 T1192 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica2] Schema name=test
[junit4:junit4]   2> 434629 T1179 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434629 T1180 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 434631 T1179 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434632 T1180 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 434644 T1180 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica1] Schema name=test
[junit4:junit4]   2> 434671 T1179 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica2] Schema name=test
[junit4:junit4]   2> 435647 T1168 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 435648 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:42212__ml_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42212__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42212/_ml"}
[junit4:junit4]   2> 435651 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:46766__ml_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46766__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46766/_ml"}
[junit4:junit4]   2> 435654 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:52879__ml_nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52879__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52879/_ml"}
[junit4:junit4]   2> 435657 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:52140__ml_nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52140__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52140/_ml"}
[junit4:junit4]   2> 435662 T1227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435662 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435663 T1172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435662 T1213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435662 T1244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435662 T1199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435662 T1242 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435665 T1167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 436052 T1207 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436069 T1207 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436073 T1207 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436087 T1223 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436099 T1207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436103 T1207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436103 T1179 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436125 T1207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436112 T1223 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436127 T1207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436127 T1207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436128 T1207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436129 T1207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436129 T1207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436130 T1207 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436130 T1207 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica3] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data/
[junit4:junit4]   2> 436130 T1223 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436131 T1207 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436132 T1207 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436133 T1207 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data
[junit4:junit4]   2> 436133 T1179 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436133 T1207 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data/index/
[junit4:junit4]   2> 436134 T1207 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard2_replica3] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436135 T1207 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data/index
[junit4:junit4]   2> 436137 T1208 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436139 T1207 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >14af80a lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >377ff6),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436139 T1207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436137 T1179 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436152 T1207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436161 T1207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436162 T1207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436139 T1160 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436163 T1208 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436160 T1220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436158 T1223 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436164 T1162 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436164 T1207 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436169 T1179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436170 T1223 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436171 T1208 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436173 T1180 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436174 T1179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436175 T1162 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436175 T1223 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436178 T1160 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436179 T1162 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436179 T1179 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436169 T1207 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436181 T1207 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436181 T1180 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436181 T1207 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436182 T1207 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436181 T1179 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436183 T1207 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436185 T1180 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436183 T1160 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436183 T1220 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436186 T1207 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436183 T1223 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436183 T1179 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436187 T1223 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436188 T1208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436187 T1179 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436190 T1179 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436190 T1179 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436190 T1207 oass.SolrIndexSearcher.<init> Opening Searcher< at >1a09508 main
[junit4:junit4]   2> 436191 T1207 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard2_replica3/data/tlog
[junit4:junit4]   2> 436192 T1207 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436192 T1208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436188 T1223 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436192 T1207 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436190 T1179 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436194 T1179 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data/
[junit4:junit4]   2> 436194 T1179 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436195 T1179 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436196 T1220 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436196 T1179 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data
[junit4:junit4]   2> 436197 T1179 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data/index/
[junit4:junit4]   2> 436197 T1179 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436198 T1179 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data/index
[junit4:junit4]   2> 436199 T1192 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 436200 T1223 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436200 T1223 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436201 T1223 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436201 T1223 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data/
[junit4:junit4]   2> 436201 T1208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436201 T1223 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436202 T1223 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436203 T1208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436204 T1179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >11d1285 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >240ba4),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436205 T1179 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436204 T1250 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard2_replica3] Registered new searcher Searcher< at >1a09508 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436203 T1223 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data
[junit4:junit4]   2> 436206 T1180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436205 T1208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436213 T1208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436213 T1162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436212 T1180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436214 T1208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436215 T1208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436215 T1208 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436216 T1208 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data/
[junit4:junit4]   2> 436216 T1208 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436217 T1208 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436218 T1162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436208 T1179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436207 T1207 oasc.ZkController.publish publishing core=awholynewcollection_0_shard2_replica3 state=down
[junit4:junit4]   2> 436206 T1223 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data/index/
[junit4:junit4]   2> 436219 T1179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436219 T1208 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data
[junit4:junit4]   2> 436220 T1179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436219 T1192 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 436221 T1179 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436222 T1179 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436222 T1162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436218 T1180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436217 T1220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436224 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436225 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436225 T1162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436224 T1160 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436224 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436226 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436227 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436229 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436222 T1179 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436220 T1208 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data/index/
[junit4:junit4]   2> 436220 T1223 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436230 T1208 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436230 T1179 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436230 T1160 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436229 T1162 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436232 T1162 oasc.SolrCore.<init> [awholynewcollection_0_shard3_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data/
[junit4:junit4]   2> 436233 T1162 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436233 T1162 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436234 T1162 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data
[junit4:junit4]   2> 436235 T1162 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data/index/
[junit4:junit4]   2> 436235 T1162 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard3_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436236 T1162 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data/index
[junit4:junit4]   2> 436229 T1220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436227 T1180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436225 T1192 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 436236 T1160 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436232 T1179 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436238 T1179 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436239 T1160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436239 T1160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436240 T1160 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436231 T1208 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data/index
[junit4:junit4]   2> 436231 T1223 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data/index
[junit4:junit4]   2> 436238 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436241 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436242 T1162 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a0c00 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >37da60),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436242 T1162 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436242 T1160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436243 T1160 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436244 T1160 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436242 T1220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436245 T1162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436244 T1160 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica3] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data/
[junit4:junit4]   2> 436244 T1208 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >16ae150 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >19e8257),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436243 T1223 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1a11fd8 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >f1890),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436247 T1223 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436242 T1179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436242 T1180 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436249 T1223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436249 T1223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436247 T1208 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436247 T1160 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436246 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436251 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436251 T1220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436246 T1162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436252 T1179 oass.SolrIndexSearcher.<init> Opening Searcher< at >1f9adb4 main
[junit4:junit4]   2> 436252 T1162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436251 T1160 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436253 T1162 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436250 T1223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436249 T1180 oasc.SolrCore.<init> [awholynewcollection_0_shard3_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data/
[junit4:junit4]   2> 436254 T1160 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data
[junit4:junit4]   2> 436254 T1223 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436254 T1162 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436255 T1162 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436256 T1162 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436257 T1192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436253 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436257 T1220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436252 T1179 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard1_replica2/data/tlog
[junit4:junit4]   2> 436258 T1220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436257 T1162 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436257 T1208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436255 T1223 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436259 T1223 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436255 T1160 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data/index/
[junit4:junit4]   2> 436255 T1180 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436260 T1160 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica3] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436260 T1223 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436259 T1208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436259 T1162 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436259 T1179 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436258 T1220 oasc.SolrCore.<init> [awholynewcollection_0_shard3_replica3] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data/
[junit4:junit4]   2> 436262 T1192 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 436262 T1179 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436262 T1208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436261 T1223 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436261 T1160 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data/index
[junit4:junit4]   2> 436260 T1180 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436265 T1180 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data
[junit4:junit4]   2> 436266 T1180 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data/index/
[junit4:junit4]   2> 436266 T1180 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard3_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436265 T1223 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436264 T1208 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436264 T1162 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436262 T1220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436268 T1223 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436269 T1251 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica2] Registered new searcher Searcher< at >1f9adb4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436270 T1220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436271 T1220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data
[junit4:junit4]   2> 436268 T1208 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436271 T1208 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436267 T1192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436267 T1180 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data/index
[junit4:junit4]   2> 436266 T1160 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >133c431 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1a4fbe),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436272 T1179 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica2 state=down
[junit4:junit4]   2> 436272 T1208 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436271 T1220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data/index/
[junit4:junit4]   2> 436274 T1223 oass.SolrIndexSearcher.<init> Opening Searcher< at >17b2bef main
[junit4:junit4]   2> 436274 T1180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >67e699 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >10a9c28),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436275 T1180 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436274 T1192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436275 T1192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436273 T1160 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436276 T1192 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 436275 T1223 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard2_replica1/data/tlog
[junit4:junit4]   2> 436274 T1162 oass.SolrIndexSearcher.<init> Opening Searcher< at >149c9cb main
[junit4:junit4]   2> 436274 T1208 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436277 T1160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436274 T1220 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard3_replica3] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436278 T1192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 436279 T1220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data/index
[junit4:junit4]   2> 436278 T1160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436278 T1208 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436277 T1162 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard3_replica2/data/tlog
[junit4:junit4]   2> 436277 T1223 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436280 T1162 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436277 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436281 T1162 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436280 T1223 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436280 T1160 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436279 T1192 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 436283 T1160 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436282 T1220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >d5950c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >f322f1),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436284 T1220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436281 T1208 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436281 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436286 T1252 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard2_replica1] Registered new searcher Searcher< at >17b2bef main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436285 T1253 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard3_replica2] Registered new searcher Searcher< at >149c9cb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436283 T1160 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436287 T1160 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436288 T1160 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436283 T1192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436289 T1223 oasc.ZkController.publish publishing core=awholynewcollection_0_shard2_replica1 state=down
[junit4:junit4]   2> 436289 T1162 oasc.ZkController.publish publishing core=awholynewcollection_0_shard3_replica2 state=down
[junit4:junit4]   2> 436289 T1208 oass.SolrIndexSearcher.<init> Opening Searcher< at >f79daf main
[junit4:junit4]   2> 436288 T1160 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436287 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436287 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436291 T1180 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436291 T1160 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436291 T1208 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369328645938/awholynewcollection_0_shard1_replica1/data/tlog
[junit4:junit4]   2> 436290 T1192 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data/
[junit4:junit4]   2> 436293 T1208 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436292 T1180 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436293 T1180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436292 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436294 T1180 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436294 T1160 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436293 T1208 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436293 T1192 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >f864fe
[junit4:junit4]   2> 436295 T1180 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436294 T1220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436296 T1180 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436295 T1192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 436297 T1160 oass.SolrIndexSearcher.<init> Opening Searcher< at >12106d8 main
[junit4:junit4]   2> 436297 T1192 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data
[junit4:junit4]   2> 436298 T1192 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data/index/
[junit4:junit4]   2> 436298 T1192 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 436299 T1192 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data/index
[junit4:junit4]   2> 436296 T1220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436300 T1254 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica1] Registered new searcher Searcher< at >f79daf main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436298 T1180 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436297 T1160 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369328643285/awholynewcollection_0_shard1_replica3/data/tlog
[junit4:junit4]   2> 436301 T1192 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >97cabe lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >18e8627),segFN=segments_1,generation=1}
[junit4:junit4]   2> 436301 T1192 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 436300 T1220 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436302 T1220 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436302 T1220 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436303 T1220 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436301 T1160 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436304 T1220 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436303 T1192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 436303 T1180 oass.SolrIndexSearcher.<init> Opening Searcher< at >17b9ff2 main
[junit4:junit4]   2> 436302 T1208 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica1 state=down
[junit4:junit4]   2> 436305 T1180 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369328643575/awholynewcollection_0_shard3_replica1/data/tlog
[junit4:junit4]   2> 436305 T1192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 436304 T1160 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436306 T1220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436306 T1192 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 436307 T1180 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436307 T1180 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436307 T1192 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 436309 T1192 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 436309 T1192 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 436309 T1220 oass.SolrIndexSearcher.<init> Opening Searcher< at >fa68b0 main
[junit4:junit4]   2> 436310 T1220 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369328647102/awholynewcollection_0_shard3_replica3/data/tlog
[junit4:junit4]   2> 436311 T1220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436311 T1220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436312 T1255 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica3] Registered new searcher Searcher< at >12106d8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436309 T1192 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 436312 T1256 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard3_replica1] Registered new searcher Searcher< at >17b9ff2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436313 T1160 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica3 state=down
[junit4:junit4]   2> 436313 T1192 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 436314 T1192 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 436315 T1180 oasc.ZkController.publish publishing core=awholynewcollection_0_shard3_replica1 state=down
[junit4:junit4]   2> 436316 T1192 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 436317 T1257 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard3_replica3] Registered new searcher Searcher< at >fa68b0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436318 T1192 oass.SolrIndexSearcher.<init> Opening Searcher< at >1e6ac09 main
[junit4:junit4]   2> 436318 T1192 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369328644757/awholynewcollection_0_shard2_replica2/data/tlog
[junit4:junit4]   2> 436319 T1192 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 436319 T1192 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 436320 T1220 oasc.ZkController.publish publishing core=awholynewcollection_0_shard3_replica3 state=down
[junit4:junit4]   2> 436323 T1258 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard2_replica2] Registered new searcher Searcher< at >1e6ac09 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 436324 T1192 oasc.ZkController.publish publishing core=awholynewcollection_0_shard2_replica2 state=down
[junit4:junit4]   2> 436340 T1222 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52140__ml_nodes_used_collection_shard1_replica2, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 436341 T1163 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52879__ml_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 436341 T1222 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52140__ml_nodes_used_collection_shard1_replica2&state=recovering&nodeName=127.0.0.1:52140__ml&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard1_replica1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2096 
[junit4:junit4]   2> 436341 T1163 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52879__ml_nodes_used_collection_shard2_replica2&state=recovering&nodeName=127.0.0.1:52879__ml&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard2_replica1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2118 
[junit4:junit4]   2> 437173 T1168 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 437174 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard2_replica3",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52879__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52879/_ml"}
[junit4:junit4]   2> 437174 T1168 oasc.Overseer$ClusterStateUpdater.createCollection Create collection awholynewcollection_0 with numShards 3
[junit4:junit4]   2> 437177 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard1_replica2",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:45571__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:45571/_ml"}
[junit4:junit4]   2> 437180 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard2_replica1",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42212__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42212/_ml"}
[junit4:junit4]   2> 437182 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard3",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard3_replica2",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46766__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46766/_ml"}
[junit4:junit4]   2> 437185 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard1_replica1",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52879__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52879/_ml"}
[junit4:junit4]   2> 437188 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard1_replica3",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46766__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46766/_ml"}
[junit4:junit4]   2> 437191 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard3",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard3_replica1",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:45571__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:45571/_ml"}
[junit4:junit4]   2> 437193 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard3",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard3_replica3",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42212__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42212/_ml"}
[junit4:junit4]   2> 437198 T1168 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"3",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"awholynewcollection_0_shard2_replica2",
[junit4:junit4]   2>   "collection":"awholynewcollection_0",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:52140__ml",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:52140/_ml"}
[junit4:junit4]   2> 437202 T1244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437202 T1199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1167 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1242 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437203 T1172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 437220 T1207 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard2_replica3
[junit4:junit4]   2> 437220 T1207 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard2_replica3 address:http://127.0.0.1:52879/_ml collection:awholynewcollection_0 shard:shard2
[junit4:junit4]   2> 437221 T1207 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0/leader_elect/shard2/election
[junit4:junit4]   2> 437226 T1207 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 437228 T1207 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=3 found=1 timeoutin=180000
[junit4:junit4]   2> 437274 T1179 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica2
[junit4:junit4]   2> 437275 T1179 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica2 address:http://127.0.0.1:45571/_ml collection:awholynewcollection_0 shard:shard1
[junit4:junit4]   2> 437276 T1179 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0/leader_elect/shard1/election
[junit4:junit4]   2> 437282 T1179 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 437283 T1179 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=3 found=1 timeoutin=180000
[junit4:junit4]   2> 437291 T1223 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard2_replica1
[junit4:junit4]   2> 437291 T1162 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard3_replica2
[junit4:junit4]   2> 437291 T1223 oasc.ZkController.register Register replica - core:awholynewcollection_

[...truncated too long message...]

ect/election/89740321936834563-127.0.0.1:46766__ml-n_0000000000 (0)
[junit4:junit4]   1>    /solr/overseer_elect/leader (0)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"id":"89740321936834563-127.0.0.1:46766__ml-n_0000000000"}
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=3198F45C014BF633 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=Asia/Ashgabat -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE  281s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([3198F45C014BF633:B07E7A447614960F]:0)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
[junit4:junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
[junit4:junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 706691 T1145 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 280835 T1144 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4]   2> 707907 T1227 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_3/leader_elect/shard3/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 707908 T1227 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_3/leader_elect/shard2/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 707908 T1227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 707908 T1227 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 707908 T1227 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 707908 T1227 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_3/leader_elect/shard1/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {timestamp=PostingsFormat(name=MockRandom), range_facet_l=PostingsFormat(name=MockRandom), id=PostingsFormat(name=NestedPulsing), range_facet_sl=PostingsFormat(name=NestedPulsing), a_t=PostingsFormat(name=MockRandom), text=MockVariableIntBlock(baseBlockSize=43), range_facet_si=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=MockSep), other_tl1=PostingsFormat(name=MockSep), multiDefault=PostingsFormat(name=MockRandom), a_si=PostingsFormat(name=MockSep), intDefault=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=et, timezone=Asia/Ashgabat
[junit4:junit4]   2> NOTE: Linux 3.2.0-41-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=3,free=150117696,total=301727744
[junit4:junit4]   2> NOTE: All tests run in this JVM: [CachingDirectoryFactoryTest, HighlighterConfigTest, ShardRoutingCustomTest, TestSuggestSpellingConverter, CurrencyFieldOpenExchangeTest, TestDefaultSimilarityFactory, TestSolrQueryParserDefaultOperatorResource, ConvertedLegacyTest, TestZkChroot, TestFoldingMultitermQuery, IndexReaderFactoryTest, SortByFunctionTest, NotRequiredUniqueKeyTest, JSONWriterTest, TestFieldTypeResource, TestSolr4Spatial, BadIndexSchemaTest, TestIndexSearcher, DisMaxRequestHandlerTest, TestDefaultSearchFieldResource, LeaderElectionIntegrationTest, TimeZoneUtilsTest, TestSchemaNameResource, DirectSolrSpellCheckerTest, TestUtils, TestDistributedGrouping, TestCopyFieldCollectionResource, ChaosMonkeyShardSplitTest, IndexSchemaRuntimeFieldTest, NoCacheHeaderTest, TestReload, TestDynamicFieldCollectionResource, TestRTGBase, TestJmxMonitoredMap, SignatureUpdateProcessorFactoryTest, TestBM25SimilarityFactory, XmlUpdateRequestHandlerTest, TestCSVResponseWriter, TestPropInject, FieldAnalysisRequestHandlerTest, ZkSolrClientTest, TestShardHandlerFactory, OpenCloseCoreStressTest, CollectionsAPIDistributedZkTest]
[junit4:junit4] Completed on J1 in 282.11s, 1 test, 1 failure <<< FAILURES!

[...truncated 548 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:887: There were test failures: 296 suites, 1234 tests, 1 failure, 13 ignored (7 assumptions)

Total time: 42 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe< at >lucene.apache.org
For additional commands, e-mail: dev-help< at >lucene.apache.org

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 496 - Failure!

Policeman Jenkins Server — 2013-05-23T15:04:15

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/496/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:49889 returned non ok status:500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:49889 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([A45CB9EC96503665:25BA37F4E10F5659]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)


REGRESSION:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:51386 returned non ok status:500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:51386 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([A45CB9EC96503665:25BA37F4E10F5659]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:133)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 9584 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4]   2> 1647382 T2541 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 1647391 T2541 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1369319259695
[junit4:junit4]   2> 1647393 T2541 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 1647394 T2542 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 1647495 T2541 oasc.ZkTestServer.run start zk server on port:49882
[junit4:junit4]   2> 1647497 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1647511 T2548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >19614933 name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1647511 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1647512 T2541 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 1647522 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1647528 T2550 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >7cd4c5ac name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1647528 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1647528 T2541 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 1647537 T2541 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 1647545 T2541 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 1647554 T2541 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 1647565 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1647566 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1647582 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 1647583 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 1647592 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 1647593 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 1647601 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 1647602 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 1647610 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 1647611 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 1647625 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 1647627 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 1647639 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 1647640 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 1647648 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 1647649 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 1647657 T2541 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 1647659 T2541 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 1648422 T2541 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 1648439 T2541 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:49885
[junit4:junit4]   2> 1648441 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1648443 T2541 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1648444 T2541 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971
[junit4:junit4]   2> 1648445 T2541 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/solr.xml
[junit4:junit4]   2> 1648446 T2541 oasc.CoreContainer.<init> New CoreContainer 148598336
[junit4:junit4]   2> 1648447 T2541 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/'
[junit4:junit4]   2> 1648449 T2541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/'
[junit4:junit4]   2> 1648771 T2541 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 1648773 T2541 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1648774 T2541 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 1648775 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1648776 T2541 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1648777 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1648778 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1648779 T2541 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1648780 T2541 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1648781 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 1648830 T2541 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 1648831 T2541 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49882/solr
[junit4:junit4]   2> 1648832 T2541 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 1648835 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1648843 T2561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >743505ef name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1648844 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1648873 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 1648906 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1648922 T2563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >45cd74fb name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1648923 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1648933 T2541 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 1648949 T2541 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 1648966 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1648975 T2541 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49885_
[junit4:junit4]   2> 1648979 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49885_
[junit4:junit4]   2> 1648993 T2541 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1649014 T2541 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1649024 T2541 oasc.Overseer.start Overseer (id=89739707004026883-127.0.0.1:49885_-n_0000000000) starting
[junit4:junit4]   2> 1649050 T2541 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 1649067 T2565 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 1649073 T2541 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1649083 T2541 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 1649092 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1649113 T2564 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1649115 T2566 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/collection1
[junit4:junit4]   2> 1649116 T2566 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1649119 T2566 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1649120 T2566 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1649127 T2566 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/collection1/'
[junit4:junit4]   2> 1649129 T2566 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1649130 T2566 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/collection1/lib/README' to classloader
[junit4:junit4]   2> 1649262 T2566 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1649435 T2566 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1649445 T2566 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 1649457 T2566 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 1650531 T2566 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1650536 T2566 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1650540 T2566 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1650549 T2566 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1650628 T2566 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1650630 T2566 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369319259971/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data/
[junit4:junit4]   2> 1650630 T2566 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >7f7ee44
[junit4:junit4]   2> 1650631 T2566 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1650632 T2566 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data
[junit4:junit4]   2> 1650633 T2566 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data/index/
[junit4:junit4]   2> 1650633 T2566 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1650633 T2566 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data/index
[junit4:junit4]   2> 1650640 T2566 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >6019171e lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5fdb7768),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1650641 T2566 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1650645 T2566 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1650646 T2566 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1650647 T2566 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1650647 T2566 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1650649 T2566 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1650649 T2566 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1650649 T2566 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1650650 T2566 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1650650 T2566 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1650666 T2566 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1650675 T2566 oass.SolrIndexSearcher.<init> Opening Searcher< at >78e3882c main
[junit4:junit4]   2> 1650676 T2566 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/control/data/tlog
[junit4:junit4]   2> 1650677 T2566 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1650677 T2566 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1650680 T2567 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >78e3882c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1650682 T2566 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1650683 T2566 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1652136 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1652138 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"1",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"control_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49885_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49885"}
[junit4:junit4]   2> 1652138 T2564 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 1652138 T2564 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 1652148 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1652688 T2566 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1652688 T2566 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49885 collection:control_collection shard:shard1
[junit4:junit4]   2> 1652690 T2566 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 1652717 T2566 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 1652725 T2566 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 1652725 T2566 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 1652726 T2566 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49885/collection1/
[junit4:junit4]   2> 1652726 T2566 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 1652726 T2566 oasc.SyncStrategy.syncToMe http://127.0.0.1:49885/collection1/ has no replicas
[junit4:junit4]   2> 1652726 T2566 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49885/collection1/
[junit4:junit4]   2> 1652727 T2566 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 1653662 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1653691 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1653727 T2566 oasc.ZkController.register We are http://127.0.0.1:49885/collection1/ and leader is http://127.0.0.1:49885/collection1/
[junit4:junit4]   2> 1653728 T2566 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49885
[junit4:junit4]   2> 1653728 T2566 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 1653728 T2566 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 1653729 T2566 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1653734 T2566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1653738 T2541 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1653739 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1653740 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1653778 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 1653781 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1653788 T2570 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >3071e3aa name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1653788 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1653794 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1653805 T2541 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 1654523 T2541 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 1654530 T2541 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:49889
[junit4:junit4]   2> 1654532 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1654532 T2541 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1654533 T2541 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108
[junit4:junit4]   2> 1654533 T2541 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/solr.xml
[junit4:junit4]   2> 1654534 T2541 oasc.CoreContainer.<init> New CoreContainer 921667395
[junit4:junit4]   2> 1654535 T2541 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/'
[junit4:junit4]   2> 1654535 T2541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/'
[junit4:junit4]   2> 1654777 T2541 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 1654778 T2541 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1654779 T2541 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 1654779 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1654780 T2541 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1654780 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1654781 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1654782 T2541 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1654782 T2541 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1654783 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 1654817 T2541 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 1654818 T2541 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49882/solr
[junit4:junit4]   2> 1654819 T2541 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 1654821 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1654826 T2581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >27843da3 name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1654827 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1654833 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 1654859 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1654867 T2583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6bb469fd name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1654867 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1654881 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1655215 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1655217 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49885__collection1",
[junit4:junit4]   2>   "numShards":"1",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"control_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49885_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49885"}
[junit4:junit4]   2> 1655228 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1655229 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1655230 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1655894 T2541 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49889_
[junit4:junit4]   2> 1655898 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49889_
[junit4:junit4]   2> 1655905 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1655911 T2570 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 1655911 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1655912 T2583 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 1655912 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1655924 T2563 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 1655970 T2584 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/collection1
[junit4:junit4]   2> 1655971 T2584 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 1655972 T2584 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1655973 T2584 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 1655976 T2584 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/collection1/'
[junit4:junit4]   2> 1655978 T2584 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1655978 T2584 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/collection1/lib/README' to classloader
[junit4:junit4]   2> 1656068 T2584 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1656158 T2584 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1656163 T2584 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 1656169 T2584 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 1657301 T2584 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1657309 T2584 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1657314 T2584 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1657325 T2584 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1657431 T2584 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1657432 T2584 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369319266108/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1/
[junit4:junit4]   2> 1657433 T2584 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >7f7ee44
[junit4:junit4]   2> 1657434 T2584 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1657435 T2584 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1
[junit4:junit4]   2> 1657436 T2584 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1/index/
[junit4:junit4]   2> 1657437 T2584 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1657438 T2584 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1/index
[junit4:junit4]   2> 1657446 T2584 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1657447 T2584 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1657454 T2584 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1657455 T2584 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1657457 T2584 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1657459 T2584 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1657460 T2584 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1657461 T2584 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1657462 T2584 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1657464 T2584 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1657466 T2584 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1657500 T2584 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1657509 T2584 oass.SolrIndexSearcher.<init> Opening Searcher< at >5c7cde9d main
[junit4:junit4]   2> 1657509 T2584 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty1/tlog
[junit4:junit4]   2> 1657511 T2584 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1657511 T2584 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1657516 T2585 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >5c7cde9d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1657520 T2584 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1657521 T2584 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1658277 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1658279 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49889_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49889"}
[junit4:junit4]   2> 1658279 T2564 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 1658279 T2564 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 1658298 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1658299 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1658299 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1658527 T2584 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1658528 T2584 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49889 collection:collection1 shard:shard1
[junit4:junit4]   2> 1658530 T2584 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1658560 T2584 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 1658570 T2584 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 1658571 T2584 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 1658571 T2584 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49889/collection1/
[junit4:junit4]   2> 1658572 T2584 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 1658573 T2584 oasc.SyncStrategy.syncToMe http://127.0.0.1:49889/collection1/ has no replicas
[junit4:junit4]   2> 1658573 T2584 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49889/collection1/
[junit4:junit4]   2> 1658574 T2584 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1659816 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1659849 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1659854 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1659854 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1659907 T2584 oasc.ZkController.register We are http://127.0.0.1:49889/collection1/ and leader is http://127.0.0.1:49889/collection1/
[junit4:junit4]   2> 1659908 T2584 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49889
[junit4:junit4]   2> 1659908 T2584 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 1659909 T2584 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 1659910 T2584 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1659920 T2584 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1659925 T2541 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1659926 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1659927 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1660811 T2541 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 1660827 T2541 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:49892
[junit4:junit4]   2> 1660829 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1660830 T2541 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1660831 T2541 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289
[junit4:junit4]   2> 1660832 T2541 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/solr.xml
[junit4:junit4]   2> 1660833 T2541 oasc.CoreContainer.<init> New CoreContainer 1797047614
[junit4:junit4]   2> 1660836 T2541 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/'
[junit4:junit4]   2> 1660837 T2541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/'
[junit4:junit4]   2> 1661173 T2541 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 1661174 T2541 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1661175 T2541 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 1661177 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1661179 T2541 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1661180 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1661182 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1661183 T2541 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1661184 T2541 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1661186 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 1661222 T2541 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 1661225 T2541 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49882/solr
[junit4:junit4]   2> 1661228 T2541 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 1661237 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1661246 T2597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6a21cd50 name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1661247 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1661252 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 1661277 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1661292 T2599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >792d11b6 name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1661293 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1661311 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1661387 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1661389 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49889__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49889_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49889"}
[junit4:junit4]   2> 1661403 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1661404 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1661405 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1661405 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1662323 T2541 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49892_
[junit4:junit4]   2> 1662333 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49892_
[junit4:junit4]   2> 1662358 T2570 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 1662359 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1662359 T2599 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 1662360 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1662374 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 1662387 T2583 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 1662387 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1662391 T2563 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 1662418 T2600 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/collection1
[junit4:junit4]   2> 1662419 T2600 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 1662421 T2600 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1662422 T2600 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 1662435 T2600 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/collection1/'
[junit4:junit4]   2> 1662439 T2600 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1662440 T2600 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/collection1/lib/README' to classloader
[junit4:junit4]   2> 1662571 T2600 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1662694 T2600 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1662698 T2600 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 1662709 T2600 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 1664053 T2600 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1664061 T2600 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1664066 T2600 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1664078 T2600 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1664188 T2600 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1664188 T2600 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369319272289/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2/
[junit4:junit4]   2> 1664189 T2600 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >7f7ee44
[junit4:junit4]   2> 1664190 T2600 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1664191 T2600 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2
[junit4:junit4]   2> 1664192 T2600 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2/index/
[junit4:junit4]   2> 1664192 T2600 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1664194 T2600 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2/index
[junit4:junit4]   2> 1664201 T2600 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1664202 T2600 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1664209 T2600 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1664210 T2600 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1664212 T2600 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1664213 T2600 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1664215 T2600 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1664215 T2600 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1664216 T2600 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1664220 T2600 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1664224 T2600 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1664251 T2600 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1664261 T2600 oass.SolrIndexSearcher.<init> Opening Searcher< at >46cc74dc main
[junit4:junit4]   2> 1664262 T2600 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty2/tlog
[junit4:junit4]   2> 1664264 T2600 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1664264 T2600 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1664303 T2601 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >46cc74dc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1664310 T2600 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1664311 T2600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1664472 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1664475 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49892_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49892"}
[junit4:junit4]   2> 1664476 T2564 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 1664476 T2564 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 1664492 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1664492 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1664493 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1664501 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1665317 T2600 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1665317 T2600 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49892 collection:collection1 shard:shard2
[junit4:junit4]   2> 1665326 T2600 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 1665352 T2600 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 1665361 T2600 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 1665362 T2600 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 1665362 T2600 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49892/collection1/
[junit4:junit4]   2> 1665362 T2600 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 1665362 T2600 oasc.SyncStrategy.syncToMe http://127.0.0.1:49892/collection1/ has no replicas
[junit4:junit4]   2> 1665363 T2600 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49892/collection1/
[junit4:junit4]   2> 1665363 T2600 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 1666013 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1666042 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1666055 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1666056 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1666057 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1666107 T2600 oasc.ZkController.register We are http://127.0.0.1:49892/collection1/ and leader is http://127.0.0.1:49892/collection1/
[junit4:junit4]   2> 1666108 T2600 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49892
[junit4:junit4]   2> 1666109 T2600 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 1666109 T2600 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 1666109 T2600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1666115 T2600 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1666119 T2541 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1666120 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1666121 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1667085 T2541 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 1667095 T2541 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:49895
[junit4:junit4]   2> 1667098 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1667099 T2541 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1667101 T2541 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453
[junit4:junit4]   2> 1667107 T2541 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/solr.xml
[junit4:junit4]   2> 1667109 T2541 oasc.CoreContainer.<init> New CoreContainer 310909954
[junit4:junit4]   2> 1667110 T2541 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/'
[junit4:junit4]   2> 1667111 T2541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/'
[junit4:junit4]   2> 1667345 T2541 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 1667346 T2541 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1667348 T2541 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 1667349 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1667350 T2541 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1667351 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1667353 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1667354 T2541 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1667356 T2541 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1667358 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 1667405 T2541 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 1667406 T2541 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49882/solr
[junit4:junit4]   2> 1667407 T2541 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 1667410 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1667419 T2613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >634bb9b9 name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1667420 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1667435 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 1667475 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1667481 T2615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >5307c62 name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1667482 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1667506 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1667581 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1667584 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49892__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49892_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49892"}
[junit4:junit4]   2> 1667598 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1667598 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1667599 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1667599 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1667603 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1668518 T2541 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49895_
[junit4:junit4]   2> 1668521 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49895_
[junit4:junit4]   2> 1668528 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 1668542 T2563 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 1668546 T2570 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 1668547 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1668547 T2599 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 1668548 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1668548 T2583 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 1668549 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1668549 T2615 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 1668549 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1668577 T2616 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/collection1
[junit4:junit4]   2> 1668577 T2616 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 1668580 T2616 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1668581 T2616 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 1668585 T2616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/collection1/'
[junit4:junit4]   2> 1668588 T2616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1668589 T2616 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/collection1/lib/README' to classloader
[junit4:junit4]   2> 1668711 T2616 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1668858 T2616 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1668862 T2616 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 1668869 T2616 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 1670173 T2616 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1670179 T2616 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1670181 T2616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1670188 T2616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1670279 T2616 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1670280 T2616 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369319278453/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/
[junit4:junit4]   2> 1670280 T2616 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >7f7ee44
[junit4:junit4]   2> 1670281 T2616 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1670282 T2616 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3
[junit4:junit4]   2> 1670282 T2616 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index/
[junit4:junit4]   2> 1670283 T2616 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1670284 T2616 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index
[junit4:junit4]   2> 1670290 T2616 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13a5a693 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >354264e5),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1670291 T2616 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1670306 T2616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1670307 T2616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1670308 T2616 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1670310 T2616 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1670311 T2616 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1670311 T2616 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1670312 T2616 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1670313 T2616 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1670314 T2616 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1670337 T2616 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1670350 T2616 oass.SolrIndexSearcher.<init> Opening Searcher< at >2db4c4ab main
[junit4:junit4]   2> 1670352 T2616 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/tlog
[junit4:junit4]   2> 1670354 T2616 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1670355 T2616 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1670378 T2617 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >2db4c4ab main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1670382 T2616 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1670383 T2616 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1670636 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1670639 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49895_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49895"}
[junit4:junit4]   2> 1670639 T2564 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 1670640 T2564 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 1670657 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1670658 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1670658 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1670659 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1670660 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1671387 T2616 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1671388 T2616 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49895 collection:collection1 shard:shard1
[junit4:junit4]   2> 1671400 T2616 oasc.ZkController.register We are http://127.0.0.1:49895/collection1/ and leader is http://127.0.0.1:49889/collection1/
[junit4:junit4]   2> 1671401 T2616 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49895
[junit4:junit4]   2> 1671401 T2616 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 1671402 T2616 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2918 name=collection1 org.apache.solr.core.SolrCore< at >c86f559 url=http://127.0.0.1:49895/collection1 node=127.0.0.1:49895_ C2918_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49895_, base_url=http://127.0.0.1:49895}
[junit4:junit4]   2> 1671404 T2618 C2918 P49895 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 1671406 T2618 C2918 P49895 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 1671406 T2618 C2918 P49895 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 1671407 T2618 C2918 P49895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1671412 T2616 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1671415 T2618 C2918 P49895 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1671430 T2541 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1671431 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1671432 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1671461 T2577 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:49895__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 1672166 T2541 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 1672173 T2541 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:49899
[junit4:junit4]   2> 1672175 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1672176 T2541 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1672177 T2541 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780
[junit4:junit4]   2> 1672177 T2541 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/solr.xml
[junit4:junit4]   2> 1672178 T2541 oasc.CoreContainer.<init> New CoreContainer 1323580394
[junit4:junit4]   2> 1672179 T2541 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/'
[junit4:junit4]   2> 1672179 T2541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/'
[junit4:junit4]   2> 1672197 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1672199 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49895__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49895_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49895"}
[junit4:junit4]   2> 1672212 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1672213 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1672213 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1672213 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1672214 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1672463 T2577 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:49895__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 1672463 T2577 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49895__collection1&state=recovering&nodeName=127.0.0.1:49895_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002 
[junit4:junit4]   2> 1672471 T2541 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 1672471 T2541 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1672472 T2541 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 1672473 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1672473 T2541 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1672474 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1672475 T2541 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1672479 T2541 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1672483 T2541 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1672483 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 1672514 T2541 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 1672515 T2541 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49882/solr
[junit4:junit4]   2> 1672517 T2541 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 1672520 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1672525 T2630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6128d09 name:ZooKeeperConnection Watcher:127.0.0.1:49882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1672526 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1672531 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 1672561 T2541 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1672571 T2632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1b608a13 name:ZooKeeperConnection Watcher:127.0.0.1:49882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1672572 T2541 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1672585 T2541 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1673595 T2541 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49899_
[junit4:junit4]   2> 1673599 T2541 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49899_
[junit4:junit4]   2> 1673619 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 1673632 T2570 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673632 T2583 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673633 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1673632 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1673632 T2632 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673634 T2599 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673635 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1673636 T2615 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673637 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1673645 T2563 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 1673659 T2633 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/collection1
[junit4:junit4]   2> 1673659 T2633 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 1673661 T2633 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1673661 T2633 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 1673664 T2633 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/collection1/'
[junit4:junit4]   2> 1673666 T2633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1673666 T2633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/collection1/lib/README' to classloader
[junit4:junit4]   2> 1673796 T2633 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 1673957 T2633 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1673961 T2633 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 1673969 T2633 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> ASYNC  NEW_CORE C2919 name=collection1 org.apache.solr.core.SolrCore< at >c86f559 url=http://127.0.0.1:49895/collection1 node=127.0.0.1:49895_ C2919_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49895_, base_url=http://127.0.0.1:49895}
[junit4:junit4]   2> 1674466 T2618 C2919 P49895 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49889/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 1674467 T2618 C2919 P49895 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49895 START replicas=[http://127.0.0.1:49889/collection1/] nUpdates=100
[junit4:junit4]   2> 1674468 T2618 C2919 P49895 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 1674485 T2618 C2919 P49895 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 1674488 T2618 C2919 P49895 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 1674488 T2618 C2919 P49895 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 1674488 T2618 C2919 P49895 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 1674489 T2618 C2919 P49895 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49889/collection1/. core=collection1
[junit4:junit4]   2> 1674489 T2618 C2919 P49895 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2920 name=collection1 org.apache.solr.core.SolrCore< at >1099ea3 url=http://127.0.0.1:49889/collection1 node=127.0.0.1:49889_ C2920_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49889_, base_url=http://127.0.0.1:49889, leader=true}
[junit4:junit4]   2> 1674519 T2577 C2920 P49889 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14 
[junit4:junit4]   2> 1674526 T2574 C2920 P49889 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 1674593 T2574 C2920 P49889 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1674595 T2574 C2920 P49889 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1674596 T2574 C2920 P49889 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_2,generation=2}
[junit4:junit4]   2> 1674596 T2574 C2920 P49889 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 1674597 T2574 C2920 P49889 oass.SolrIndexSearcher.<init> Opening Searcher< at >5c0a67cf realtime
[junit4:junit4]   2> 1674598 T2574 C2920 P49889 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1674598 T2574 C2920 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 72
[junit4:junit4]   2> 1674600 T2618 C2919 P49895 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 1674601 T2618 C2919 P49895 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 1674625 T2576 C2920 P49889 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> ASYNC  NEW_CORE C2921 name=collection1 org.apache.solr.core.SolrCore< at >1099ea3 url=http://127.0.0.1:49889/collection1 node=127.0.0.1:49889_ C2921_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49889_, base_url=http://127.0.0.1:49889, leader=true}
[junit4:junit4]   2> 1674652 T2576 C2921 P49889 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=46 
[junit4:junit4]   2> ASYNC  NEW_CORE C2922 name=collection1 org.apache.solr.core.SolrCore< at >c86f559 url=http://127.0.0.1:49895/collection1 node=127.0.0.1:49895_ C2922_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49895_, base_url=http://127.0.0.1:49895}
[junit4:junit4]   2> 1674662 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 1674663 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 1674663 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 1674667 T2578 C2921 P49889 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 1674670 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 1674684 T2618 C2922 P49895 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index.20130523222806973
[junit4:junit4]   2> 1674687 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1e6c629 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >328053e0) fullCopy=false
[junit4:junit4]   2> 1674690 T2579 C2921 P49889 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 1674698 T2618 C2922 P49895 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 1674701 T2618 C2922 P49895 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 1674702 T2618 C2922 P49895 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 1674724 T2618 C2922 P49895 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13a5a693 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >354264e5),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13a5a693 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >354264e5),segFN=segments_2,generation=2}
[junit4:junit4]   2> 1674729 T2618 C2922 P49895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 1674735 T2618 C2922 P49895 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 1674735 T2618 C2922 P49895 oass.SolrIndexSearcher.<init> Opening Searcher< at >49035106 main
[junit4:junit4]   2> 1674737 T2617 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >49035106 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1674747 T2618 C2922 P49895 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index.20130523222806973 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index.20130523222806973;done=true>>]
[junit4:junit4]   2> 1674747 T2618 C2922 P49895 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index.20130523222806973
[junit4:junit4]   2> 1674760 T2618 C2922 P49895 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty3/index.20130523222806973
[junit4:junit4]   2> 1674761 T2618 C2922 P49895 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 1674761 T2618 C2922 P49895 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 1674761 T2618 C2922 P49895 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 1674762 T2618 C2922 P49895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1674765 T2618 C2922 P49895 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 1675199 T2633 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1675204 T2633 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1675208 T2633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1675218 T2633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1675262 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1675264 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49895__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49895_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49895"}
[junit4:junit4]   2> 1675280 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675281 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675281 T2632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675282 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675282 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675281 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1675330 T2633 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1675331 T2633 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369319283780/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/
[junit4:junit4]   2> 1675331 T2633 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >7f7ee44
[junit4:junit4]   2> 1675332 T2633 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1675333 T2633 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4
[junit4:junit4]   2> 1675335 T2633 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index/
[junit4:junit4]   2> 1675336 T2633 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1675337 T2633 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index
[junit4:junit4]   2> 1675349 T2633 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >795a1ad5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >515c915e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1675349 T2633 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1675354 T2633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1675355 T2633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1675360 T2633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1675362 T2633 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1675363 T2633 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1675363 T2633 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1675365 T2633 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1675367 T2633 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1675368 T2633 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1675389 T2633 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1675402 T2633 oass.SolrIndexSearcher.<init> Opening Searcher< at >38dce987 main
[junit4:junit4]   2> 1675403 T2633 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/tlog
[junit4:junit4]   2> 1675405 T2633 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1675405 T2633 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1675423 T2636 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >38dce987 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1675427 T2633 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1675428 T2633 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1676811 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1676815 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":null,
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49899_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49899"}
[junit4:junit4]   2> 1676815 T2564 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 1676815 T2564 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 1676828 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1676828 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1676828 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1676828 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1676829 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1676828 T2632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1677434 T2633 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1677435 T2633 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49899 collection:collection1 shard:shard2
[junit4:junit4]   2> 1677450 T2633 oasc.ZkController.register We are http://127.0.0.1:49899/collection1/ and leader is http://127.0.0.1:49892/collection1/
[junit4:junit4]   2> 1677451 T2633 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49899
[junit4:junit4]   2> 1677451 T2633 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 1677451 T2633 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2923 name=collection1 org.apache.solr.core.SolrCore< at >b4f831 url=http://127.0.0.1:49899/collection1 node=127.0.0.1:49899_ C2923_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49899_, base_url=http://127.0.0.1:49899}
[junit4:junit4]   2> 1677453 T2637 C2923 P49899 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 1677454 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 1677455 T2637 C2923 P49899 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 1677455 T2637 C2923 P49899 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1677457 T2633 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1677458 T2637 C2923 P49899 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1677478 T2541 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1677478 T2541 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1677479 T2541 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1677502 T2595 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:49899__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 1677515 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1677518 T2541 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 1677519 T2541 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 1677522 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1678352 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1678355 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49899__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49899_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49899"}
[junit4:junit4]   2> 1678367 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678367 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678368 T2632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678368 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678372 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678374 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1678505 T2595 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:49899__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 1678506 T2595 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49899__collection1&state=recovering&nodeName=127.0.0.1:49899_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1004 
[junit4:junit4]   2> 1678527 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1679532 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C2923_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49899_, base_url=http://127.0.0.1:49899}
[junit4:junit4]   2> 1680508 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49892/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 1680509 T2637 C2923 P49899 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49899 START replicas=[http://127.0.0.1:49892/collection1/] nUpdates=100
[junit4:junit4]   2> 1680510 T2637 C2923 P49899 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 1680510 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 1680510 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 1680511 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 1680511 T2637 C2923 P49899 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 1680511 T2637 C2923 P49899 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49892/collection1/. core=collection1
[junit4:junit4]   2> 1680511 T2637 C2923 P49899 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1680557 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C2924 name=collection1 org.apache.solr.core.SolrCore< at >2d1ec082 url=http://127.0.0.1:49892/collection1 node=127.0.0.1:49892_ C2924_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49892_, base_url=http://127.0.0.1:49892, leader=true}
[junit4:junit4]   2> 1680559 T2592 C2924 P49892 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 1680562 T2595 C2924 P49892 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 1680566 T2592 C2924 P49892 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1680567 T2592 C2924 P49892 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1680568 T2592 C2924 P49892 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_2,generation=2}
[junit4:junit4]   2> 1680568 T2592 C2924 P49892 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 1680570 T2592 C2924 P49892 oass.SolrIndexSearcher.<init> Opening Searcher< at >30937105 realtime
[junit4:junit4]   2> 1680570 T2592 C2924 P49892 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1680571 T2592 C2924 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 1680572 T2637 C2923 P49899 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 1680573 T2637 C2923 P49899 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 1680579 T2594 C2924 P49892 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 1680580 T2594 C2924 P49892 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 1680581 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 1680581 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 1680581 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 1680585 T2591 C2924 P49892 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1 
[junit4:junit4]   2> 1680587 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 1680590 T2637 C2923 P49899 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index.20130523222812890
[junit4:junit4]   2> 1680590 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >49b64b78 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >20b5177c) fullCopy=false
[junit4:junit4]   2> 1680594 T2593 C2924 P49892 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 1680596 T2637 C2923 P49899 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 1680599 T2637 C2923 P49899 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 1680599 T2637 C2923 P49899 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 1680602 T2637 C2923 P49899 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >795a1ad5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >515c915e),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >795a1ad5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >515c915e),segFN=segments_2,generation=2}
[junit4:junit4]   2> 1680603 T2637 C2923 P49899 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 1680603 T2637 C2923 P49899 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 1680604 T2637 C2923 P49899 oass.SolrIndexSearcher.<init> Opening Searcher< at >78f5bf54 main
[junit4:junit4]   2> 1680606 T2636 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >78f5bf54 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1680607 T2637 C2923 P49899 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index.20130523222812890 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index.20130523222812890;done=true>>]
[junit4:junit4]   2> 1680607 T2637 C2923 P49899 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index.20130523222812890
[junit4:junit4]   2> 1680608 T2637 C2923 P49899 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369319259694/jetty4/index.20130523222812890
[junit4:junit4]   2> 1680608 T2637 C2923 P49899 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 1680609 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 1680609 T2637 C2923 P49899 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 1680609 T2637 C2923 P49899 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1680613 T2637 C2923 P49899 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 1681449 T2564 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1681452 T2564 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:49899__collection1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"collection1",
[junit4:junit4]   2>   "collection":"collection1",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:49899_",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:49899"}
[junit4:junit4]   2> 1681469 T2599 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681470 T2563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681471 T2632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681472 T2583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681473 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681474 T2615 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 1681568 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1681572 T2541 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C2925 name=collection1 org.apache.solr.core.SolrCore< at >5c316451 url=http://127.0.0.1:49885/collection1 node=127.0.0.1:49885_ C2925_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49885_, base_url=http://127.0.0.1:49885, leader=true}
[junit4:junit4]   2> 1681605 T2557 C2925 P49885 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 1681614 T2557 C2925 P49885 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >6019171e lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5fdb7768),segFN=segments_1,generation=1}
[junit4:junit4]   2> 1681615 T2557 C2925 P49885 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 1681617 T2557 C2925 P49885 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >6019171e lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5fdb7768),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >6019171e lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5fdb7768),segFN=segments_2,generation=2}
[junit4:junit4]   2> 1681617 T2557 C2925 P49885 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 1681618 T2557 C2925 P49885 oass.SolrIndexSearcher.<init> Opening Searcher< at >4a99ac43 main
[junit4:junit4]   2> 1681619 T2557 C2925 P49885 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1681621 T2567 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >4a99ac43 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1681622 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 17
[junit4:junit4]   2> ASYNC  NEW_CORE C2926 name=collection1 org.apache.solr.core.SolrCore< at >2d1ec082 url=http://127.0.0.1:49892/collection1 node=127.0.0.1:49892_ C2926_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49892_, base_url=http://127.0.0.1:49892, leader=true}
[junit4:junit4]   2> 1681660 T2590 C2926 P49892 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 1681665 T2590 C2926 P49892 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >34e23e3c lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >7a70a267),segFN=segments_3,generation=3}
[junit4:junit4]   2> 1681666 T2590 C2926 P49892 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 1681667 T2590 C2926 P49892 oass.SolrIndexSearcher.<init> Opening Searcher< at >1c9051b4 main
[junit4:junit4]   2> 1681668 T2590 C2926 P49892 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1681670 T2601 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >1c9051b4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1681672 T2590 C2926 P49892 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:49889/collection1/, StdNode: http://127.0.0.1:49895/collection1/, StdNode: http://127.0.0.1:49899/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2927 name=collection1 org.apache.solr.core.SolrCore< at >1099ea3 url=http://127.0.0.1:49889/collection1 node=127.0.0.1:49889_ C2927_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49889_, base_url=http://127.0.0.1:49889, leader=true}
[junit4:junit4]   2> 1681711 T2579 C2927 P49889 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C2928 name=collection1 org.apache.solr.core.SolrCore< at >b4f831 url=http://127.0.0.1:49899/collection1 node=127.0.0.1:49899_ C2928_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49899_, base_url=http://127.0.0.1:49899}
[junit4:junit4]   2> 1681729 T2627 C2928 P49899 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C2929 name=collection1 org.apache.solr.core.SolrCore< at >c86f559 url=http://127.0.0.1:49895/collection1 node=127.0.0.1:49895_ C2929_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49895_, base_url=http://127.0.0.1:49895}
[junit4:junit4]   2> 1681735 T2606 C2929 P49895 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 1681737 T2627 C2928 P49899 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >795a1ad5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >515c915e),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >795a1ad5 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >515c915e),segFN=segments_3,generation=3}
[junit4:junit4]   2> 1681738 T2627 C2928 P49899 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 1681740 T2579 C2927 P49889 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >2157c280 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >5cc388de),segFN=segments_3,generation=3}
[junit4:junit4]   2> 1681741 T2579 C2927 P49889 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 1681742 T2627 C2928 P49899 oass.SolrIndexSearcher.<init> Opening Searcher< at >11190bf8 main
[junit4:junit4]   2> 1681743 T2627 C2928 P49899 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1681747 T2579 C2927 P49889 oass.SolrIndexSearcher.<init> Opening Searcher< at >49da4e65 main
[junit4:junit4]   2> 1681748 T2636 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >11190bf8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1681750 T2585 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >49da4e65 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1681748 T2579 C2927 P49889 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1681751 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 40
[junit4:junit4]   2> 1681752 T2627 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4]   2> 1681754 T2606 C2929 P49895 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13a5a693 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >354264e5),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13a5a693 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >354264e5),segFN=segments_3,generation=3}
[junit4:junit4]   2> 1681755 T2606 C2929 P49895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 1681761 T2606 C2929 P49895 oass.SolrIndexSearcher.<init> Opening Searcher< at >b93ab63 main
[junit4:junit4]   2> 1681762 T2606 C2929 P49895 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 1681764 T2617 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher< at >b93ab63 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 1681765 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4]   2> 1681768 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 108
[junit4:junit4]   2> 1681772 T2541 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1681781 T2577 C2927 P49889 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 1681787 T2607 C2929 P49895 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 1681796 T2594 C2926 P49892 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 1681801 T2628 C2928 P49899 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 1683828 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435835350256713728)} 0 3
[junit4:junit4]   2> 1683862 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435835350265102336&update.from=http://127.0.0.1:49889/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435835350265102336)} 0 2
[junit4:junit4]   2> 1683869 T2628 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435835350291316736&update.from=http://127.0.0.1:49892/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435835350291316736)} 0 2
[junit4:junit4]   2> 1683871 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435835350291316736)} 0 12
[junit4:junit4]   2> 1683872 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435835350265102336)} 0 38
[junit4:junit4]   2> 1683877 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1435835350309142528)]} 0 1
[junit4:junit4]   2> 1683901 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435835350329065472)]} 0 1
[junit4:junit4]   2> 1683902 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1435835350329065472)]} 0 8
[junit4:junit4]   2> 1683904 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 23
[junit4:junit4]   2> 1683915 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1435835350348988416)]} 0 1
[junit4:junit4]   2> 1683929 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435835350361571328)]} 0 1
[junit4:junit4]   2> 1683930 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1435835350361571328)]} 0 5
[junit4:junit4]   2> 1683931 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 9
[junit4:junit4]   2> 1683938 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1435835350374154240)]} 0 0
[junit4:junit4]   2> 1683964 T2624 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435835350388834304)]} 0 3
[junit4:junit4]   2> 1683967 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435835350388834304)]} 0 16
[junit4:junit4]   2> 1683971 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2]} 0 23
[junit4:junit4]   2> 1683975 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1435835350412951552)]} 0 0
[junit4:junit4]   2> 1683985 T2623 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435835350420291584)]} 0 1
[junit4:junit4]   2> 1683986 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[3 (1435835350420291584)]} 0 5
[junit4:junit4]   2> 1683987 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3]} 0 8
[junit4:junit4]   2> 1683991 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1435835350429728768)]} 0 1
[junit4:junit4]   2> 1684006 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435835350438117376)]} 0 1
[junit4:junit4]   2> 1684007 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1435835350438117376)]} 0 8
[junit4:junit4]   2> 1684009 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4]} 0 15
[junit4:junit4]   2> 1684013 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1435835350452797440)]} 0 0
[junit4:junit4]   2> 1684024 T2625 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435835350461186048)]} 0 1
[junit4:junit4]   2> 1684026 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[5 (1435835350461186048)]} 0 5
[junit4:junit4]   2> 1684028 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5]} 0 10
[junit4:junit4]   2> 1684031 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1435835350471671808)]} 0 0
[junit4:junit4]   2> 1684037 T2626 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435835350474817536)]} 0 0
[junit4:junit4]   2> 1684039 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1435835350474817536)]} 0 5
[junit4:junit4]   2> 1684043 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1435835350484254720)]} 0 1
[junit4:junit4]   2> 1684050 T2627 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435835350487400448)]} 0 1
[junit4:junit4]   2> 1684052 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1435835350487400448)]} 0 6
[junit4:junit4]   2> 1684057 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1435835350498934784)]} 0 1
[junit4:junit4]   2> 1684066 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435835350503129088)]} 0 1
[junit4:junit4]   2> 1684073 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1435835350503129088)]} 0 12
[junit4:junit4]   2> 1684080 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1435835350522003456)]} 0 1
[junit4:junit4]   2> 1684091 T2628 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435835350528294912)]} 0 1
[junit4:junit4]   2> 1684093 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1435835350528294912)]} 0 8
[junit4:junit4]   2> 1684101 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1435835350544023552)]} 0 2
[junit4:junit4]   2> 1684114 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435835350553460736)]} 0 1
[junit4:junit4]   2> 1684116 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[10 (1435835350553460736)]} 0 8
[junit4:junit4]   2> 1684117 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10]} 0 12
[junit4:junit4]   2> 1684123 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1435835350567092224)]} 0 2
[junit4:junit4]   2> 1684131 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435835350573383680)]} 0 0
[junit4:junit4]   2> 1684133 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1435835350573383680)]} 0 6
[junit4:junit4]   2> 1684138 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1435835350582820864)]} 0 1
[junit4:junit4]   2> 1684145 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435835350588063744)]} 0 0
[junit4:junit4]   2> 1684147 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1435835350588063744)]} 0 6
[junit4:junit4]   2> 1684151 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1435835350596452352)]} 0 1
[junit4:junit4]   2> 1684161 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435835350603792384)]} 0 1
[junit4:junit4]   2> 1684162 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1435835350603792384)]} 0 5
[junit4:junit4]   2> 1684163 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13]} 0 9
[junit4:junit4]   2> 1684167 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1435835350614278144)]} 0 0
[junit4:junit4]   2> 1684185 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435835350630006784)]} 0 0
[junit4:junit4]   2> 1684186 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1435835350630006784)]} 0 4
[junit4:junit4]   2> 1684188 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14]} 0 14
[junit4:junit4]   2> 1684192 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1435835350640492544)]} 0 1
[junit4:junit4]   2> 1684203 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435835350645735424)]} 0 2
[junit4:junit4]   2> 1684204 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1435835350645735424)]} 0 7
[junit4:junit4]   2> 1684208 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1435835350657269760)]} 0 0
[junit4:junit4]   2> 1684219 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435835350665658368)]} 0 1
[junit4:junit4]   2> 1684220 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1435835350665658368)]} 0 5
[junit4:junit4]   2> 1684221 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16]} 0 9
[junit4:junit4]   2> 1684224 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1435835350674046976)]} 0 0
[junit4:junit4]   2> 1684231 T2624 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435835350678241280)]} 0 0
[junit4:junit4]   2> 1684232 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1435835350678241280)]} 0 4
[junit4:junit4]   2> 1684235 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1435835350685581312)]} 0 0
[junit4:junit4]   2> 1684240 T2625 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435835350688727040)]} 0 0
[junit4:junit4]   2> 1684241 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1435835350688727040)]} 0 3
[junit4:junit4]   2> 1684246 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1435835350696067072)]} 0 1
[junit4:junit4]   2> 1684252 T2626 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435835350700261376)]} 0 1
[junit4:junit4]   2> 1684253 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1435835350700261376)]} 0 4
[junit4:junit4]   2> 1684262 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1435835350712844288)]} 0 1
[junit4:junit4]   2> 1684275 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435835350718087168)]} 0 1
[junit4:junit4]   2> 1684276 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1435835350718087168)]} 0 10
[junit4:junit4]   2> 1684280 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1435835350732767232)]} 0 0
[junit4:junit4]   2> 1684288 T2627 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435835350736961536)]} 0 1
[junit4:junit4]   2> 1684289 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1435835350736961536)]} 0 5
[junit4:junit4]   2> 1684293 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1435835350746398720)]} 0 0
[junit4:junit4]   2> 1684300 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435835350750593024)]} 0 1
[junit4:junit4]   2> 1684301 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1435835350750593024)]} 0 5
[junit4:junit4]   2> 1684305 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1435835350758981632)]} 0 0
[junit4:junit4]   2> 1684315 T2628 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435835350763175936)]} 0 1
[junit4:junit4]   2> 1684317 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1435835350763175936)]} 0 8
[junit4:junit4]   2> 1684320 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1435835350774710272)]} 0 0
[junit4:junit4]   2> 1684328 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435835350778904576)]} 0 0
[junit4:junit4]   2> 1684331 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1435835350778904576)]} 0 7
[junit4:junit4]   2> 1684341 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435835350795681792)]} 0 1
[junit4:junit4]   2> 1684351 T2623 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435835350799876096)]} 0 1
[junit4:junit4]   2> 1684352 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435835350799876096)]} 0 8
[junit4:junit4]   2> 1684357 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435835350813507584)]} 0 1
[junit4:junit4]   2> 1684365 T2624 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435835350816653312)]} 0 0
[junit4:junit4]   2> 1684366 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435835350816653312)]} 0 6
[junit4:junit4]   2> 1684370 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1435835350826090496)]} 0 1
[junit4:junit4]   2> 1684376 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435835350830284800)]} 0 0
[junit4:junit4]   2> 1684378 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1435835350830284800)]} 0 5
[junit4:junit4]   2> 1684382 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1435835350838673408)]} 0 1
[junit4:junit4]   2> 1684392 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435835350847062016)]} 0 0
[junit4:junit4]   2> 1684394 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1435835350847062016)]} 0 5
[junit4:junit4]   2> 1684395 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28]} 0 9
[junit4:junit4]   2> 1684399 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1435835350857547776)]} 0 0
[junit4:junit4]   2> 1684405 T2625 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435835350860693504)]} 0 0
[junit4:junit4]   2> 1684406 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1435835350860693504)]} 0 4
[junit4:junit4]   2> 1684410 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1435835350869082112)]} 0 1
[junit4:junit4]   2> 1684418 T2626 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435835350874324992)]} 0 0
[junit4:junit4]   2> 1684424 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1435835350874324992)]} 0 9
[junit4:junit4]   2> 1684427 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30]} 0 14
[junit4:junit4]   2> 1684434 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1435835350893199360)]} 0 1
[junit4:junit4]   2> 1684447 T2627 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435835350903685120)]} 0 1
[junit4:junit4]   2> 1684448 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1435835350903685120)]} 0 5
[junit4:junit4]   2> 1684450 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31]} 0 10
[junit4:junit4]   2> 1684454 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1435835350914170880)]} 0 1
[junit4:junit4]   2> 1684461 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435835350918365184)]} 0 1
[junit4:junit4]   2> 1684462 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1435835350918365184)]} 0 5
[junit4:junit4]   2> 1684466 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1435835350927802368)]} 0 1
[junit4:junit4]   2> 1684476 T2628 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1435835350935142400)]} 0 1
[junit4:junit4]   2> 1684477 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1435835350935142400)]} 0 5
[junit4:junit4]   2> 1684479 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33]} 0 9
[junit4:junit4]   2> 1684483 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435835350944579584)]} 0 1
[junit4:junit4]   2> 1684496 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1435835350950871040)]} 0 1
[junit4:junit4]   2> 1684497 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435835350950871040)]} 0 10
[junit4:junit4]   2> 1684501 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1435835350964502528)]} 0 1
[junit4:junit4]   2> 1684515 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1435835350969745408)]} 0 1
[junit4:junit4]   2> 1684516 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1435835350969745408)]} 0 11
[junit4:junit4]   2> 1684521 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1435835350985474048)]} 0 0
[junit4:junit4]   2> 1684528 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1435835350989668352)]} 0 1
[junit4:junit4]   2> 1684530 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1435835350989668352)]} 0 5
[junit4:junit4]   2> 1684533 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1435835350998056960)]} 0 0
[junit4:junit4]   2> 1684540 T2623 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1435835351002251264)]} 0 0
[junit4:junit4]   2> 1684542 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1435835351002251264)]} 0 5
[junit4:junit4]   2> 1684554 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435835351017979904)]} 0 5
[junit4:junit4]   2> 1684563 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1435835351024271360)]} 0 1
[junit4:junit4]   2> 1684565 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435835351024271360)]} 0 8
[junit4:junit4]   2> 1684569 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1435835351034757120)]} 0 1
[junit4:junit4]   2> 1684584 T2624 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1435835351047340032)]} 0 1
[junit4:junit4]   2> 1684586 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1435835351047340032)]} 0 6
[junit4:junit4]   2> 1684590 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1435835351057825792)]} 0 1
[junit4:junit4]   2> 1684597 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1435835351060971520)]} 0 1
[junit4:junit4]   2> 1684598 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1435835351060971520)]} 0 5
[junit4:junit4]   2> 1684602 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1435835351070408704)]} 0 1
[junit4:junit4]   2> 1684609 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1435835351073554432)]} 0 1
[junit4:junit4]   2> 1684610 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1435835351073554432)]} 0 5
[junit4:junit4]   2> 1684614 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1435835351081943040)]} 0 1
[junit4:junit4]   2> 1684628 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1435835351091380224)]} 0 1
[junit4:junit4]   2> 1684630 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1435835351091380224)]} 0 9
[junit4:junit4]   2> 1684638 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1435835351108157440)]} 0 0
[junit4:junit4]   2> 1684650 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1435835351116546048)]} 0 0
[junit4:junit4]   2> 1684652 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[43 (1435835351116546048)]} 0 6
[junit4:junit4]   2> 1684653 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43]} 0 11
[junit4:junit4]   2> 1684659 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1435835351129128960)]} 0 1
[junit4:junit4]   2> 1684667 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1435835351134371840)]} 0 1
[junit4:junit4]   2> 1684668 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1435835351134371840)]} 0 5
[junit4:junit4]   2> 1684671 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1435835351142760448)]} 0 1
[junit4:junit4]   2> 1684680 T2625 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1435835351149051904)]} 0 0
[junit4:junit4]   2> 1684685 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[45 (1435835351149051904)]} 0 8
[junit4:junit4]   2> 1684689 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45]} 0 15
[junit4:junit4]   2> 1684693 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1435835351165829120)]} 0 1
[junit4:junit4]   2> 1684715 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1435835351184703488)]} 0 1
[junit4:junit4]   2> 1684717 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[46 (1435835351184703488)]} 0 6
[junit4:junit4]   2> 1684718 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46]} 0 10
[junit4:junit4]   2> 1684722 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1435835351196237824)]} 0 1
[junit4:junit4]   2> 1684732 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1435835351202529280)]} 0 1
[junit4:junit4]   2> 1684733 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1435835351202529280)]} 0 5
[junit4:junit4]   2> 1684734 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47]} 0 9
[junit4:junit4]   2> 1684737 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1435835351211966464)]} 0 0
[junit4:junit4]   2> 1684746 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1435835351218257920)]} 0 0
[junit4:junit4]   2> 1684747 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[48 (1435835351218257920)]} 0 4
[junit4:junit4]   2> 1684749 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48]} 0 9
[junit4:junit4]   2> 1684753 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1435835351227695104)]} 0 1
[junit4:junit4]   2> 1684776 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1435835351240278016)]} 0 1
[junit4:junit4]   2> 1684778 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1435835351240278016)]} 0 16
[junit4:junit4]   2> 1684779 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49]} 0 23
[junit4:junit4]   2> 1684783 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435835351260200960)]} 0 1
[junit4:junit4]   2> 1684790 T2610 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1435835351264395264)]} 0 1
[junit4:junit4]   2> 1684792 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435835351264395264)]} 0 6
[junit4:junit4]   2> 1684795 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1435835351272783872)]} 0 0
[junit4:junit4]   2> 1684805 T2606 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1435835351279075328)]} 0 1
[junit4:junit4]   2> 1684806 T2579 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1435835351279075328)]} 0 5
[junit4:junit4]   2> 1684808 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51]} 0 10
[junit4:junit4]   2> 1684812 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1435835351289561088)]} 0 1
[junit4:junit4]   2> 1684822 T2611 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1435835351297949696)]} 0 0
[junit4:junit4]   2> 1684825 T2577 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[52 (1435835351297949696)]} 0 7
[junit4:junit4]   2> 1684829 T2593 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52]} 0 14
[junit4:junit4]   2> 1684839 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1435835351318921216)]} 0 1
[junit4:junit4]   2> 1684849 T2627 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1435835351324164096)]} 0 1
[junit4:junit4]   2> 1684851 T2591 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1435835351324164096)]} 0 7
[junit4:junit4]   2> 1684857 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1435835351336747008)]} 0 2
[junit4:junit4]   2> 1684870 T2608 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1435835351343038464)]} 0 1
[junit4:junit4]   2> 1684872 T2575 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1435835351343038464)]} 0 10
[junit4:junit4]   2> 1684884 T2554 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1435835351366107136)]} 0 1
[junit4:junit4]   2> 1684893 T2607 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1435835351371350016)]} 0 1
[junit4:junit4]   2> 1684895 T2574 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1435835351371350016)]} 0 6
[junit4:junit4]   2> 1684900 T2557 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1435835351381835776)]} 0 1
[junit4:junit4]   2> 1684914 T2628 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1435835351387078656)]} 0 2
[junit4:junit4]   2> 1684919 T2595 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1435835351387078656)]} 0 15
[junit4:junit4]   2> 1684924 T2558 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435835351407001600)]} 0 1
[junit4:junit4]   2> 1684932 T2623 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1435835351412244480)]} 0 1
[junit4:junit4]   2> 1684934 T2592 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435835351412244480)]} 0 6
[junit4:junit4]   2> 1684939 T2559 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1435835351422730240)]} 0 1
[junit4:junit4]   2> 1684952 T2624 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1435835351434264576)]} 0 1
[junit4:junit4]   2> 1684953 T2594 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[58 (1435835351434264576)]} 0 5
[junit4:junit4]   2> 1684955 T2576 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58]} 0 11
[junit4:junit4]   2> 1684959 T2555 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435835351443701760)]} 0 1
[junit4:junit4]   2> 1684966 T2609 C2929 P49895 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1435835351447896064)]} 0 1
[junit4:junit4]   2> 1684967 T2578 C2927 P49889 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435835351447896064)]} 0 5
[junit4:junit4]   2> 1684972 T2556 C2925 P49885 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1435835351457333248)]} 0 0
[junit4:junit4]   2> 1684981 T2626 C2928 P49899 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49892/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1435835351464673280)]} 0 0
[junit4:junit4]   2> 1684983 T2590 C2926 P49892 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49889/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[60 (1435835351464673280)]} 0 5
[junit4:junit4]   2> 1

[...truncated too long message...]

2
[junit4:junit4]   2> !!!! WARNING: best effort to remove /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-1369321237572 FAILED !!!!!
[junit4:junit4]   2> 3677549 T7247 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 3677550 T7247 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51379 51379
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=A45CB9EC96503665 -Dtests.slow=true -Dtests.locale=in_ID -Dtests.timezone=Asia/Pontianak -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   52.3s | ShardSplitTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:51386 returned non ok status:500, message:Server Error
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([A45CB9EC96503665:25BA37F4E10F5659]:0)
[junit4:junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]    > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4]    > at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:133)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 3677563 T7247 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 52318 T7246 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 3678550 T7338 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 3678550 T7338 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {id=Pulsing41(freqCutoff=8 minBlockSize=10 maxBlockSize=45), _version_=PostingsFormat(name=MockSep)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=in_ID, timezone=Asia/Pontianak
[junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=3,free=176017864,total=334925824
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestReload, TestFastOutputStream, MultiTermTest, ZkNodePropsTest, DocumentBuilderTest, SampleTest, DefaultValueUpdateProcessorTest, TestGroupingSearch, TestAddFieldRealTimeGet, AlternateDirectoryTest, ShardRoutingCustomTest, TestDynamicFieldResource, DocumentAnalysisRequestHandlerTest, FieldMutatingUpdateProcessorTest, TestQueryTypes, TestCopyFieldCollectionResource, TestRemoteStreaming, MoreLikeThisHandlerTest, SyncSliceTest, TestElisionMultitermQuery, PolyFieldTest, TestSolr4Spatial, BasicZkTest, TestFunctionQuery, DateFieldTest, SimpleFacetsTest, TestUniqueKeyFieldResource, SolrTestCaseJ4Test, TestMultiCoreConfBootstrap, XsltUpdateRequestHandlerTest, TestStressReorder, TestPartialUpdateDeduplication, TestRandomDVFaceting, StatelessScriptUpdateProcessorFactoryTest, CurrencyFieldOpenExchangeTest, SortByFunctionTest, EchoParamsTest, TermsComponentTest, MBeansHandlerTest, TestStressLucene, WordBreakSolrSpellCheckerTest, TestSolrJ, LeaderElectionIntegrationTest, DistanceFunctionTest, TestCloudManagedSchemaAddField, SuggesterTest, FastVectorHighlighterTest, SuggesterWFSTTest, ConvertedLegacyTest, DOMUtilTest, SolrCoreTest, TestLFUCache, TestSurroundQueryParser, UpdateParamsTest, QueryEqualityTest, RegexBoostProcessorTest, SolrIndexConfigTest, TestSolrIndexConfig, BadCopyFieldTest, TestFieldCollectionResource, CacheHeaderTest, OpenExchangeRatesOrgProviderTest, IndexReaderFactoryTest, IndexSchemaRuntimeFieldTest, TestPseudoReturnFields, CollectionsAPIDistributedZkTest, TestPostingsSolrHighlighter, TestAtomicUpdateErrorCases, TestValueSourceCache, CoreAdminHandlerTest, TestTrie, TestIndexSearcher, OpenCloseCoreStressTest, ReturnFieldsTest, QueryParsingTest, TermVectorComponentDistributedTest, SpellPossibilityIteratorTest, CachingDirectoryFactoryTest, TestJmxIntegration, TestMergePolicyConfig, TestSearchPerf, TestIBSimilarityFactory, TestComponentsName, TestCharFilters, TestSolrDeletionPolicy1, CSVRequestHandlerTest, TestRTGBase, TestPHPSerializedResponseWriter, TestSchemaSimilarityResource, FullSolrCloudDistribCmdsTest, TestPluginEnable, DocValuesMultiTest, SignatureUpdateProcessorFactoryTest, SimplePostToolTest, ResourceLoaderTest, ZkCLITest, CopyFieldTest, TestSolrXml, DateMathParserTest, TestStressVersions, FieldAnalysisRequestHandlerTest, TestLMDirichletSimilarityFactory, TestBinaryField, TestReversedWildcardFilterFactory, UnloadDistributedZkTest, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, TestNumberUtils, UUIDFieldTest, TestStressRecovery, QueryResultKeyTest, DirectSolrConnectionTest, PingRequestHandlerTest, StandardRequestHandlerTest, SystemInfoHandlerTest, ZkControllerTest, RequiredFieldsTest, JsonLoaderTest, DirectSolrSpellCheckerTest, ScriptEngineTest, CircularListTest, TestSerializedLuceneMatchVersion, TestFiltering, TestFastLRUCache, ClusterStateTest, TestLuceneMatchVersion, SpatialFilterTest, ExternalFileFieldSortTest, XmlUpdateRequestHandlerTest, TestPerFieldSimilarity, ChaosMonkeyShardSplitTest, TestJmxMonitoredMap, BadComponentTest, TestSolrQueryParserResource, SearchHandlerTest, SolrPluginUtilsTest, TestRecovery, TestShardHandlerFactory, ZkSolrClientTest, BasicDistributedZk2Test, SolrIndexSplitterTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, OverseerTest, ShardRoutingTest, RecoveryZkTest, TestReplicationHandler, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, SliceStateUpdateTest, TestDistributedSearch, AutoCommitTest, TestDistributedGrouping, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, HardAutoCommitTest, DistributedTermsComponentTest, TestRangeQuery, TestCoreContainer, StatsComponentTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, BasicFunctionalityTest, TestLazyCores, TestBadConfig, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, TestCSVLoader, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestUpdate, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, PrimitiveFieldTypeTest, TestOmitPositions, FileBasedSpellCheckerTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, TestSolrQueryParser, TestArbitraryIndexDir, LoggingHandlerTest, IndexSchemaTest, TestCollationField, UpdateRequestProcessorFactoryTest, JSONWriterTest, TestCSVResponseWriter, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestBinaryResponseWriter, HighlighterConfigTest, SOLR749Test, TestPropInject, TestQuerySenderListener, TestPropInjectDefaults, TestQuerySenderNoQuery, ResponseLogComponentTest, TestSolrDeletionPolicy2, TestDocSet, NumericFieldsTest, MinimalSchemaTest, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TimeZoneUtilsTest, URLClassifyProcessorTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, TestSolrXMLSerializer, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestLRUCache, TestUtils, TestDocumentBuilder, SliceStateTest, FileUtilsTest, AliasIntegrationTest, ShardSplitTest]
[junit4:junit4] Completed in 53.38s, 1 test, 1 error <<< FAILURES!

[...truncated 55 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 2 errors, 18 ignored (6 assumptions)

Total time: 106 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe< at >lucene.apache.org
For additional commands, e-mail: dev-help< at >lucene.apache.org

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 5730 - Failure!

Policeman Jenkins Server — 2013-05-23T12:47:54

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/5730/
Java: 32bit/jdk1.6.0_45 -client -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 230 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
at __randomizedtesting.SeedInfo.seed([B296600366B56EC4:3370EE1B11EA0EF8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 9095 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4:junit4]   2> 662568 T2014 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_ozv/vs
[junit4:junit4]   2> 662571 T2014 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1369312517869
[junit4:junit4]   2> 662572 T2014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 662573 T2015 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 662673 T2014 oasc.ZkTestServer.run start zk server on port:59838
[junit4:junit4]   2> 662675 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 662774 T2021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >6c9b94 name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 662775 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 662775 T2014 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 662779 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 662781 T2023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >169b1ef name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 662781 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 662781 T2014 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 662784 T2014 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 662785 T2014 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 662787 T2014 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 662789 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 662789 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 662792 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 662793 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 662795 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 662795 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 662797 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 662798 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 662799 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 662800 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 662802 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 662802 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 662804 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 662805 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 662807 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 662807 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 662809 T2014 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 662810 T2014 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 662881 T2014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 662884 T2014 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:42419
[junit4:junit4]   2> 662884 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 662885 T2014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 662885 T2014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109
[junit4:junit4]   2> 662886 T2014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/solr.xml
[junit4:junit4]   2> 662886 T2014 oasc.CoreContainer.<init> New CoreContainer 19076921
[junit4:junit4]   2> 662886 T2014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/'
[junit4:junit4]   2> 662887 T2014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/'
[junit4:junit4]   2> 662951 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109
[junit4:junit4]   2> 662952 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/collection1
[junit4:junit4]   2> 662952 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/collection1/conf
[junit4:junit4]   2> 662958 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/collection1/conf/xslt
[junit4:junit4]   2> 662976 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/collection1/lib
[junit4:junit4]   2> 662977 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/collection1/lib/classes
[junit4:junit4]   2> 662979 T2014 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/conf/core.properties, adding to cores
[junit4:junit4]   2> 662981 T2014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 662981 T2014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 662982 T2014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 662982 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 662983 T2014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 662983 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 662983 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 662984 T2014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 662984 T2014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 662985 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 662988 T2014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 662988 T2014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59838/solr
[junit4:junit4]   2> 662989 T2014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 662990 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 662992 T2034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >15b73de name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 662993 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 662994 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 663006 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 663008 T2036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >337db2 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 663008 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 663010 T2014 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 663012 T2014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 663014 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 663016 T2014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 663017 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 663019 T2014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 663023 T2014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 663024 T2014 oasc.Overseer.start Overseer (id=89739265171193859-127.0.0.1:42419__ozv%2Fvs-n_0000000000) starting
[junit4:junit4]   2> 663027 T2014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 663029 T2038 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 663030 T2014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 663031 T2014 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 663032 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 663034 T2037 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 663036 T2014 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 663036 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 663037 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 663039 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 663040 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 663042 T2041 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1dfe6a7 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 663042 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 663043 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 663045 T2014 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 663115 T2014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 663117 T2014 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:46482
[junit4:junit4]   2> 663117 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 663118 T2014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 663118 T2014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342
[junit4:junit4]   2> 663119 T2014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/solr.xml
[junit4:junit4]   2> 663119 T2014 oasc.CoreContainer.<init> New CoreContainer 33456233
[junit4:junit4]   2> 663119 T2014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/'
[junit4:junit4]   2> 663120 T2014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/'
[junit4:junit4]   2> 663198 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342
[junit4:junit4]   2> 663199 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/collection1
[junit4:junit4]   2> 663200 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/collection1/conf
[junit4:junit4]   2> 663205 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/collection1/conf/xslt
[junit4:junit4]   2> 663225 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/collection1/lib
[junit4:junit4]   2> 663226 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/collection1/lib/classes
[junit4:junit4]   2> 663227 T2014 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/conf/core.properties, adding to cores
[junit4:junit4]   2> 663230 T2014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 663230 T2014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 663231 T2014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 663231 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 663231 T2014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 663232 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 663232 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 663233 T2014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 663233 T2014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 663233 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 663236 T2014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 663237 T2014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59838/solr
[junit4:junit4]   2> 663237 T2014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 663238 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 663241 T2052 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1082422 name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 663241 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 663243 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 663246 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 663248 T2054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >ee891a name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 663248 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 663251 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 664254 T2014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 664256 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 664259 T2041 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 664259 T2054 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 664259 T2036 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 664264 T2014 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 664264 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 664265 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 664337 T2014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 664339 T2014 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:40769
[junit4:junit4]   2> 664340 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 664340 T2014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 664341 T2014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564
[junit4:junit4]   2> 664341 T2014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/solr.xml
[junit4:junit4]   2> 664342 T2014 oasc.CoreContainer.<init> New CoreContainer 28306489
[junit4:junit4]   2> 664342 T2014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/'
[junit4:junit4]   2> 664343 T2014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/'
[junit4:junit4]   2> 664405 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564
[junit4:junit4]   2> 664406 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/collection1
[junit4:junit4]   2> 664406 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/collection1/conf
[junit4:junit4]   2> 664411 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/collection1/conf/xslt
[junit4:junit4]   2> 664429 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/collection1/lib
[junit4:junit4]   2> 664431 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/collection1/lib/classes
[junit4:junit4]   2> 664432 T2014 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/conf/core.properties, adding to cores
[junit4:junit4]   2> 664434 T2014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 664435 T2014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 664435 T2014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 664436 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 664436 T2014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 664436 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 664437 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 664437 T2014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 664438 T2014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 664438 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 664441 T2014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 664442 T2014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59838/solr
[junit4:junit4]   2> 664442 T2014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 664443 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664444 T2066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1af45a3 name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664445 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664446 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 664449 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664450 T2068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >808a48 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664450 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664453 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 665456 T2014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 665457 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 665460 T2054 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 665460 T2036 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 665460 T2068 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 665460 T2041 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 665464 T2014 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 665465 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 665465 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 665540 T2014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 665542 T2014 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:45660
[junit4:junit4]   2> 665543 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 665544 T2014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 665544 T2014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765
[junit4:junit4]   2> 665545 T2014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/solr.xml
[junit4:junit4]   2> 665545 T2014 oasc.CoreContainer.<init> New CoreContainer 26417164
[junit4:junit4]   2> 665545 T2014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/'
[junit4:junit4]   2> 665546 T2014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/'
[junit4:junit4]   2> 665610 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765
[junit4:junit4]   2> 665611 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/collection1
[junit4:junit4]   2> 665612 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/collection1/conf
[junit4:junit4]   2> 665618 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/collection1/conf/xslt
[junit4:junit4]   2> 665638 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/collection1/lib
[junit4:junit4]   2> 665639 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/collection1/lib/classes
[junit4:junit4]   2> 665641 T2014 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/conf/core.properties, adding to cores
[junit4:junit4]   2> 665649 T2014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 665649 T2014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 665650 T2014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 665650 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 665651 T2014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 665651 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 665652 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 665652 T2014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 665652 T2014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 665653 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 665656 T2014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 665657 T2014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59838/solr
[junit4:junit4]   2> 665657 T2014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 665658 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 665660 T2080 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >e6969b name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 665675 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 665677 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 665680 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 665681 T2082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >909675 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 665681 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 665685 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 666688 T2014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 666689 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 666692 T2054 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 666692 T2068 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 666692 T2036 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 666693 T2082 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 666692 T2041 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 666697 T2014 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 666697 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 666698 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 666767 T2014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 666769 T2014 oejs.AbstractConnector.doStart Started SelectChannelConnector< at >127.0.0.1:37924
[junit4:junit4]   2> 666770 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 666770 T2014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 666770 T2014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997
[junit4:junit4]   2> 666771 T2014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/solr.xml
[junit4:junit4]   2> 666771 T2014 oasc.CoreContainer.<init> New CoreContainer 4238383
[junit4:junit4]   2> 666772 T2014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/'
[junit4:junit4]   2> 666772 T2014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/'
[junit4:junit4]   2> 666833 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997
[junit4:junit4]   2> 666834 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/collection1
[junit4:junit4]   2> 666835 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/collection1/conf
[junit4:junit4]   2> 666840 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/collection1/conf/xslt
[junit4:junit4]   2> 666858 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/collection1/lib
[junit4:junit4]   2> 666859 T2014 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/collection1/lib/classes
[junit4:junit4]   2> 666860 T2014 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/conf/core.properties, adding to cores
[junit4:junit4]   2> 666862 T2014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 666863 T2014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 666863 T2014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 666864 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 666864 T2014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 666864 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 666865 T2014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 666865 T2014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 666866 T2014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 666866 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 666869 T2014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 666869 T2014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59838/solr
[junit4:junit4]   2> 666870 T2014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 666871 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 666872 T2094 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1f81511 name:ZooKeeperConnection Watcher:127.0.0.1:59838 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 666873 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 666874 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 666877 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 666878 T2096 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >623658 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 666879 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 666904 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 667907 T2014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 667908 T2014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 667911 T2068 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667911 T2054 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667911 T2096 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667911 T2036 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667911 T2041 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667912 T2082 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 667917 T2014 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 667918 T2014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 667919 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 667924 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 667929 T2050 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 667931 T2036 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 667932 T2038 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4:junit4]   2>   "operation":"createcollection",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "maxShardsPerNode":null,
[junit4:junit4]   2>   "createNodeSet":null,
[junit4:junit4]   2>   "name":"nodes_used_collection",
[junit4:junit4]   2>   "replicationFactor":"2"}
[junit4:junit4]   2> 667932 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 667933 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 667933 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 667942 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 667943 T2091 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 667943 T2047 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 667943 T2077 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 667944 T2047 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 667944 T2032 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 667945 T2047 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 667943 T2091 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 667945 T2047 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 667945 T2032 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 667944 T2077 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 667947 T2032 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 667947 T2047 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 667946 T2091 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 667948 T2047 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 667947 T2032 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 667947 T2077 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 667948 T2091 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 667950 T2032 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 667950 T2047 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 667949 T2077 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 667950 T2091 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 667950 T2032 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 667952 T2077 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 667951 T2047 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/'
[junit4:junit4]   2> 667953 T2091 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 667956 T2032 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 667956 T2091 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/'
[junit4:junit4]   2> 667956 T2077 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 667957 T2032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/'
[junit4:junit4]   2> 667959 T2077 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/'
[junit4:junit4]   2> 668014 T2077 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 668017 T2047 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 668031 T2091 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 668066 T2032 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 668102 T2077 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 668103 T2077 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 668114 T2077 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
[junit4:junit4]   2> 668128 T2047 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 668129 T2047 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 668134 T2047 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
[junit4:junit4]   2> 668149 T2091 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 668150 T2091 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 668165 T2091 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
[junit4:junit4]   2> 668167 T2032 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 668168 T2032 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 668195 T2032 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
[junit4:junit4]   2> 669003 T2077 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 669016 T2077 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 669026 T2077 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 669036 T2077 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669050 T2077 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669054 T2077 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669055 T2077 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669055 T2077 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669056 T2077 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669057 T2077 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669057 T2077 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669057 T2077 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669058 T2077 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data/
[junit4:junit4]   2> 669058 T2077 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 669059 T2077 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669069 T2077 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data
[junit4:junit4]   2> 669070 T2077 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data/index/
[junit4:junit4]   2> 669070 T2077 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 669071 T2077 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data/index
[junit4:junit4]   2> 669073 T2077 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a692b2 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1a163e0),segFN=segments_1,generation=1}
[junit4:junit4]   2> 669074 T2077 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 669076 T2077 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 669077 T2077 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 669078 T2077 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 669079 T2077 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 669080 T2077 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 669080 T2077 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 669087 T2077 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 669087 T2077 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 669088 T2077 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 669090 T2077 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 669092 T2077 oass.SolrIndexSearcher.<init> Opening Searcher< at >198e297 main
[junit4:junit4]   2> 669093 T2077 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/nodes_used_collection_shard1_replica2/data/tlog
[junit4:junit4]   2> 669094 T2077 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 669094 T2077 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 669107 T2102 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher< at >198e297 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 669109 T2077 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
[junit4:junit4]   2> 669171 T2047 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 669183 T2032 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 669187 T2047 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 669190 T2032 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 669192 T2047 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 669192 T2091 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 669193 T2032 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 669207 T2091 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 669209 T2047 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669209 T2032 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669211 T2091 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 669213 T2047 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669214 T2032 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669218 T2047 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669219 T2047 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669220 T2047 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669219 T2032 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669220 T2047 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669221 T2032 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669222 T2032 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669222 T2047 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669223 T2047 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669223 T2032 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669222 T2091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669223 T2047 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669224 T2032 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669225 T2032 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669225 T2047 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/
[junit4:junit4]   2> 669226 T2032 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669226 T2047 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 669227 T2032 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data/
[junit4:junit4]   2> 669227 T2047 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669227 T2091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 669227 T2032 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 669228 T2047 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data
[junit4:junit4]   2> 669229 T2032 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669229 T2047 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index/
[junit4:junit4]   2> 669230 T2032 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data
[junit4:junit4]   2> 669231 T2032 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data/index/
[junit4:junit4]   2> 669230 T2047 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 669232 T2091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669231 T2032 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 669232 T2047 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index
[junit4:junit4]   2> 669233 T2032 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data/index
[junit4:junit4]   2> 669233 T2091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669234 T2091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669245 T2047 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a48637 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1f8a578),segFN=segments_1,generation=1}
[junit4:junit4]   2> 669245 T2032 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >18e74bc lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1aa869b),segFN=segments_1,generation=1}
[junit4:junit4]   2> 669246 T2032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 669245 T2047 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 669245 T2091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 669248 T2091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 669249 T2091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 669249 T2032 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 669249 T2047 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 669249 T2032 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 669249 T2091 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669250 T2047 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 669251 T2091 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/
[junit4:junit4]   2> 669251 T2032 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 669252 T2091 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 669251 T2047 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 669253 T2091 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 669253 T2032 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 669254 T2091 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data
[junit4:junit4]   2> 669253 T2047 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 669254 T2032 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 669255 T2032 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 669254 T2091 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index/
[junit4:junit4]   2> 669256 T2032 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 669255 T2047 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 669257 T2032 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 669257 T2047 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 669256 T2091 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 669258 T2047 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 669258 T2032 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 669259 T2047 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 669259 T2091 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index
[junit4:junit4]   2> 669260 T2047 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 669261 T2091 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >fc5a40 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >773caf),segFN=segments_1,generation=1}
[junit4:junit4]   2> 669262 T2091 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 669262 T2047 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 669262 T2032 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 669264 T2091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 669264 T2091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 669265 T2091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 669265 T2091 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 669266 T2047 oass.SolrIndexSearcher.<init> Opening Searcher< at >6f9b68 main
[junit4:junit4]   2> 669266 T2032 oass.SolrIndexSearcher.<init> Opening Searcher< at >18fe216 main
[junit4:junit4]   2> 669266 T2091 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 669267 T2091 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 669267 T2032 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/nodes_used_collection_shard2_replica2/data/tlog
[junit4:junit4]   2> 669266 T2047 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/tlog
[junit4:junit4]   2> 669268 T2032 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 669267 T2091 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 669269 T2032 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 669269 T2047 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 669270 T2091 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 669270 T2047 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 669271 T2091 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 669274 T2104 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher< at >18fe216 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 669274 T2091 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 669276 T2103 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher< at >6f9b68 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 669278 T2032 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
[junit4:junit4]   2> 669280 T2091 oass.SolrIndexSearcher.<init> Opening Searcher< at >f1cf77 main
[junit4:junit4]   2> 669280 T2047 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
[junit4:junit4]   2> 669281 T2091 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/tlog
[junit4:junit4]   2> 669282 T2091 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 669282 T2091 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 669286 T2105 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher< at >f1cf77 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 669288 T2091 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
[junit4:junit4]   2> 670549 T2037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 670550 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:45660__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:45660/_ozv/vs"}
[junit4:junit4]   2> 670550 T2037 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with numShards 2
[junit4:junit4]   2> 670552 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42419__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42419/_ozv/vs"}
[junit4:junit4]   2> 670554 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46482__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46482/_ozv/vs"}
[junit4:junit4]   2> 670556 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":null,
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"down",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:37924__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:37924/_ozv/vs"}
[junit4:junit4]   2> 670559 T2068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 670559 T2036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 670559 T2041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 670559 T2096 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 670559 T2054 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 670559 T2082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 671111 T2077 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 671112 T2077 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:45660/_ozv/vs collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 671113 T2077 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
[junit4:junit4]   2> 671117 T2077 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 671119 T2077 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=180000
[junit4:junit4]   2> 671281 T2032 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 671282 T2032 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:42419/_ozv/vs collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 671282 T2047 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 671283 T2047 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:46482/_ozv/vs collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 671283 T2032 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4]   2> 671283 T2047 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4]   2> 671288 T2032 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 671289 T2091 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 671290 T2091 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:37924/_ozv/vs collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 671291 T2032 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 671291 T2032 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 671292 T2032 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 671292 T2032 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:42419/_ozv/vs START replicas=[http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1/] nUpdates=100
[junit4:junit4]   2> 671292 T2032 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:42419/_ozv/vs DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 671293 T2032 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 671293 T2032 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 671294 T2032 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 671294 T2032 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
[junit4:junit4]   2> ASYNC  NEW_CORE C6369 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6369_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=down, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 671298 T2048 C6369 P46482 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/_ozv/vs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 671621 T2077 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 671621 T2077 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 671622 T2077 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 671622 T2077 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:45660/_ozv/vs START replicas=[http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1/] nUpdates=100
[junit4:junit4]   2> 671622 T2077 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:45660/_ozv/vs DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 671623 T2077 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 671623 T2077 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 671623 T2077 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 671624 T2077 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
[junit4:junit4]   2> ASYNC  NEW_CORE C6370 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6370_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=down, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 671630 T2092 C6370 P37924 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/_ozv/vs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 672065 T2037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 672078 T2054 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672078 T2096 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672078 T2082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672078 T2036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672078 T2041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672078 T2068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 672100 T2032 oasc.ZkController.register We are http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/ and leader is http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 672101 T2032 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica2 baseURL=http://127.0.0.1:42419/_ozv/vs
[junit4:junit4]   2> 672101 T2032 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 672101 T2032 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=active
[junit4:junit4]   2> 672103 T2032 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 672104 T2032 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=4160 
[junit4:junit4]   2> 672129 T2077 oasc.ZkController.register We are http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/ and leader is http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 672129 T2077 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica2 baseURL=http://127.0.0.1:45660/_ozv/vs
[junit4:junit4]   2> 672130 T2077 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 672130 T2077 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=active
[junit4:junit4]   2> 672132 T2077 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 672133 T2077 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=4190 
[junit4:junit4]   2> 672289 T2047 oasc.ZkController.register We are http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1/ and leader is http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 672290 T2047 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica1 baseURL=http://127.0.0.1:46482/_ozv/vs
[junit4:junit4]   2> 672290 T2047 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 672291 T2047 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 672291 T2108 C6369 P46482 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard2_replica1 recoveringAfterStartup=true
[junit4:junit4]   2> 672292 T2108 C6369 P46482 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 672292 T2047 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 672293 T2091 oasc.ZkController.register We are http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1/ and leader is http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 672292 T2108 C6369 P46482 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=recovering
[junit4:junit4]   2> 672293 T2091 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica1 baseURL=http://127.0.0.1:37924/_ozv/vs
[junit4:junit4]   2> 672294 T2091 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 672294 T2091 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 672293 T2047 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=4350 
[junit4:junit4]   2> 672295 T2109 C6370 P37924 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard1_replica1 recoveringAfterStartup=true
[junit4:junit4]   2> 672294 T2108 C6369 P46482 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 672295 T2109 C6370 P37924 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 672295 T2091 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 672296 T2109 C6370 P37924 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=recovering
[junit4:junit4]   2> 672297 T2091 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=4354 
[junit4:junit4]   2> 672297 T2109 C6370 P37924 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 672297 T2038 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: nodes_used_collection
[junit4:junit4]   2> 672300 T2030 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:46482__ozv%2Fvs_nodes_used_collection_shard2_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 672301 T2075 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:37924__ozv%2Fvs_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 672305 T2054 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
[junit4:junit4]   2> 672306 T2036 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 672306 T2038 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=4160},core=nodes_used_collection_shard2_replica2},null={responseHeader={status=0,QTime=4190},core=nodes_used_collection_shard1_replica2},null={responseHeader={status=0,QTime=4350},core=nodes_used_collection_shard2_replica1},null={responseHeader={status=0,QTime=4354},core=nodes_used_collection_shard1_replica1}}}
[junit4:junit4]   2> 672310 T2050 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2} status=0 QTime=4382 
[junit4:junit4]   2> 672311 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 672312 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 672326 T2111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >1d92036 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 672327 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 672328 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 672330 T2014 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 672332 T2014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 672333 T2113 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager< at >a80067 name:ZooKeeperConnection Watcher:127.0.0.1:59838/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 672333 T2014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 672334 T2014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 672338 T2092 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=5&name=awholynewcollection_0&replicationFactor=5&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 672340 T2036 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 672340 T2038 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4:junit4]   2>   "operation":"createcollection",
[junit4:junit4]   2>   "numShards":"4",
[junit4:junit4]   2>   "maxShardsPerNode":"5",
[junit4:junit4]   2>   "createNodeSet":null,
[junit4:junit4]   2>   "name":"awholynewcollection_0",
[junit4:junit4]   2>   "replicationFactor":"5"}
[junit4:junit4]   2> 672341 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica1 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 672341 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica2 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 672341 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica3 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 672341 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica4 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 672342 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica5 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 672342 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica1 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 672342 T2062 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard1_replica3
[junit4:junit4]   2> 672342 T2089 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard1_replica1
[junit4:junit4]   2> 672343 T2089 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672342 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica2 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 672343 T2031 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard1_replica5
[junit4:junit4]   2> 672343 T2049 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard1_replica4
[junit4:junit4]   2> 672343 T2090 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard2_replica1
[junit4:junit4]   2> 672345 T2090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672343 T2076 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard1_replica2
[junit4:junit4]   2> 672346 T2076 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672343 T2062 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672346 T2090 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672345 T2049 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672344 T2078 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2
[junit4:junit4]   2> 672349 T2049 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672344 T2031 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672344 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica3 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 672344 T2089 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672351 T2031 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672351 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica4 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 672350 T2049 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672349 T2078 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672348 T2090 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672347 T2062 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672347 T2076 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672354 T2090 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672354 T2062 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672353 T2078 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672353 T2049 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672353 T2050 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard2_replica4
[junit4:junit4]   2> 672352 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica5 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 672352 T2064 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard2_replica3
[junit4:junit4]   2> 672358 T2064 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672351 T2031 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672359 T2064 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672351 T2089 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672360 T2031 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672360 T2064 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672361 T2089 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672359 T2029 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard2_replica5
[junit4:junit4]   2> 672362 T2029 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672358 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica1 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 672357 T2050 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672357 T2049 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672356 T2062 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672356 T2078 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672355 T2090 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672354 T2076 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672365 T2078 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672364 T2062 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672364 T2091 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard3_replica1
[junit4:junit4]   2> 672364 T2050 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672367 T2078 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672363 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica2 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 672363 T2029 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 672368 T2029 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672362 T2064 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 672362 T2089 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672361 T2031 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672369 T2064 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672369 T2029 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672368 T2077 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard3_replica2
[junit4:junit4]   2> 672371 T2077 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672368 T2078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/'
[junit4:junit4]   2> 672372 T2029 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672368 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica3 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 672368 T2062 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672367 T2050 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 672367 T2090 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672366 T2091 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672374 T2050 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672366 T2076 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 672365 T2049 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672375 T2090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard2_replica1/'
[junit4:junit4]   2> 672374 T2091 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672374 T2062 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard1_replica3/'
[junit4:junit4]   2> 672373 T2061 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard3_replica3
[junit4:junit4]   2> 672373 T2029 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard2_replica5/'
[junit4:junit4]   2> 672372 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica4 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 672372 T2077 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672379 T2077 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672371 T2031 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672371 T2064 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672370 T2089 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672380 T2047 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard3_replica4
[junit4:junit4]   2> 672381 T2047 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672379 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica5 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 672379 T2049 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard1_replica4/'
[junit4:junit4]   2> 672383 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica1 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:37924__ozv%2Fvs
[junit4:junit4]   2> 672377 T2076 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672377 T2061 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672376 T2091 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672376 T2050 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672395 T2076 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard1_replica2/'
[junit4:junit4]   2> 672395 T2061 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672398 T2061 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672393 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica2 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:45660__ozv%2Fvs
[junit4:junit4]   2> 672384 T2032 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard3_replica5
[junit4:junit4]   2> 672399 T2032 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672383 T2064 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard2_replica3/'
[junit4:junit4]   2> 672383 T2089 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard1_replica1/'
[junit4:junit4]   2> 672382 T2047 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672402 T2047 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672382 T2031 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard1_replica5/'
[junit4:junit4]   2> 672382 T2077 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard3_replica2/'
[junit4:junit4]   2> 672402 T2032 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672404 T2032 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672404 T2047 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard3_replica4/'
[junit4:junit4]   2> 672400 T2130 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard4_replica2
[junit4:junit4]   2> 672406 T2130 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672400 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard3_replica3/'
[junit4:junit4]   2> 672412 T2130 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672399 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica3 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:40769__ozv%2Fvs
[junit4:junit4]   2> 672413 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica4 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:46482__ozv%2Fvs
[junit4:junit4]   2> 672414 T2038 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica5 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:42419__ozv%2Fvs
[junit4:junit4]   2> 672399 T2050 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard2_replica4/'
[junit4:junit4]   2> 672398 T2091 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard3_replica1/'
[junit4:junit4]   2> 672396 T2127 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard4_replica1
[junit4:junit4]   2> 672416 T2127 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672413 T2130 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672412 T2032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard3_replica5/'
[junit4:junit4]   2> 672419 T2127 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672419 T2127 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672417 T2134 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard4_replica5
[junit4:junit4]   2> 672416 T2048 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard4_replica4
[junit4:junit4]   2> 672421 T2048 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672421 T2134 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672416 T2063 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard4_replica3
[junit4:junit4]   2> 672428 T2134 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672428 T2127 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard4_replica1/'
[junit4:junit4]   2> 672430 T2134 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672431 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369312518109/awholynewcollection_0_shard4_replica5/'
[junit4:junit4]   2> 672422 T2048 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672442 T2048 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672443 T2048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/awholynewcollection_0_shard4_replica4/'
[junit4:junit4]   2> 672429 T2130 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard4_replica2/'
[junit4:junit4]   2> 672428 T2063 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 672446 T2063 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 672446 T2063 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 672448 T2063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369312519564/awholynewcollection_0_shard4_replica3/'
[junit4:junit4]   2> 672467 T2078 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672532 T2090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672537 T2062 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672550 T2029 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672561 T2049 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672579 T2031 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672608 T2076 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672652 T2064 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672726 T2134 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672765 T2047 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672767 T2032 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672780 T2063 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672806 T2077 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672808 T2127 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672826 T2048 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672876 T2130 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672880 T2089 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672904 T2090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 672917 T2090 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 672921 T2078 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 672923 T2078 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 672925 T2090 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica1] Schema name=test
[junit4:junit4]   2> 672937 T2078 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica2] Schema name=test
[junit4:junit4]   2> 672951 T2050 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672989 T2031 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 672991 T2029 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 672993 T2031 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 672995 T2061 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 672995 T2029 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673006 T2091 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4]   2> 673008 T2031 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica5] Schema name=test
[junit4:junit4]   2> 673009 T2029 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica5] Schema name=test
[junit4:junit4]   2> 673063 T2076 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673076 T2076 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673089 T2076 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica2] Schema name=test
[junit4:junit4]   2> 673090 T2062 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673108 T2064 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673115 T2062 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673123 T2062 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica3] Schema name=test
[junit4:junit4]   2> 673124 T2064 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673150 T2064 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica3] Schema name=test
[junit4:junit4]   2> 673193 T2049 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673220 T2049 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673242 T2127 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673247 T2049 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica4] Schema name=test
[junit4:junit4]   2> 673248 T2089 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673261 T2127 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673262 T2134 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673263 T2047 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673278 T2089 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673286 T2134 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673286 T2047 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673288 T2127 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica1] Schema name=test
[junit4:junit4]   2> 673292 T2130 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673294 T2130 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673294 T2134 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica5] Schema name=test
[junit4:junit4]   2> 673307 T2047 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica4] Schema name=test
[junit4:junit4]   2> 673311 T2077 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673312 T2077 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673338 T2048 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673339 T2063 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673340 T2089 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica1] Schema name=test
[junit4:junit4]   2> 673342 T2063 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673342 T2048 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673346 T2077 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica2] Schema name=test
[junit4:junit4]   2> 673346 T2032 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673355 T2032 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673362 T2091 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673368 T2130 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica2] Schema name=test
[junit4:junit4]   2> 673372 T2063 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica3] Schema name=test
[junit4:junit4]   2> 673373 T2091 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673375 T2048 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica4] Schema name=test
[junit4:junit4]   2> 673384 T2032 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica5] Schema name=test
[junit4:junit4]   2> 673390 T2091 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica1] Schema name=test
[junit4:junit4]   2> 673474 T2061 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673476 T2061 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673513 T2061 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica3] Schema name=test
[junit4:junit4]   2> 673557 T2050 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673567 T2050 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673609 T2050 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica4] Schema name=test
[junit4:junit4]   2> 673622 T2037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 673632 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:42419__ozv%2Fvs_nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:42419__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:42419/_ozv/vs"}
[junit4:junit4]   2> 673646 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:45660__ozv%2Fvs_nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:45660__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:45660/_ozv/vs"}
[junit4:junit4]   2> 673714 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:46482__ozv%2Fvs_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46482__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46482/_ozv/vs"}
[junit4:junit4]   2> 673760 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:37924__ozv%2Fvs_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"recovering",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:37924__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:37924/_ozv/vs"}
[junit4:junit4]   2> 673787 T2082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2054 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2096 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 673787 T2068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 674302 T2030 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:46482__ozv%2Fvs_nodes_used_collection_shard2_replica1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 674302 T2030 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:46482__ozv%252Fvs_nodes_used_collection_shard2_replica1&state=recovering&nodeName=127.0.0.1:46482__ozv%252Fvs&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard2_replica2&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 674311 T2075 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:37924__ozv%2Fvs_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 674312 T2075 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:37924__ozv%252Fvs_nodes_used_collection_shard1_replica1&state=recovering&nodeName=127.0.0.1:37924__ozv%252Fvs&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard1_replica2&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2011 
[junit4:junit4]   2> ASYNC  NEW_CORE C6371 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6371_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676312 T2108 C6371 P46482 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/ core=nodes_used_collection_shard2_replica1 - recoveringAfterStartup=true
[junit4:junit4]   2> 676313 T2108 C6371 P46482 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:46482/_ozv/vs START replicas=[http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/] nUpdates=100
[junit4:junit4]   2> 676313 T2108 C6371 P46482 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 676314 T2108 C6371 P46482 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676314 T2108 C6371 P46482 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676314 T2108 C6371 P46482 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676315 T2108 C6371 P46482 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 676315 T2108 C6371 P46482 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2/. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676315 T2108 C6371 P46482 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C6372 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6372_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676318 T2109 C6372 P37924 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/ core=nodes_used_collection_shard1_replica1 - recoveringAfterStartup=true
[junit4:junit4]   2> 676318 T2109 C6372 P37924 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:37924/_ozv/vs START replicas=[http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/] nUpdates=100
[junit4:junit4]   2> ASYNC  NEW_CORE C6373 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6373_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676331 T2109 C6373 P37924 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 676332 T2109 C6373 P37924 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676333 T2109 C6373 P37924 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676333 T2109 C6373 P37924 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676333 T2109 C6373 P37924 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 676333 T2109 C6373 P37924 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2/. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676334 T2109 C6373 P37924 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C6374 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >1efb7ce url=http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2 node=127.0.0.1:45660__ozv%2Fvs C6374_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:45660__ozv%2Fvs, base_url=http://127.0.0.1:45660/_ozv/vs, leader=true}
[junit4:junit4]   2> 676378 T2138 C6374 P45660 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_ozv/vs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8 
[junit4:junit4]   2> 676381 T2075 C6374 P45660 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C6375 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6375_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676397 T2142 C6375 P42419 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_ozv/vs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14 
[junit4:junit4]   2> ASYNC  NEW_CORE C6376 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >1efb7ce url=http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2 node=127.0.0.1:45660__ozv%2Fvs C6376_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:45660__ozv%2Fvs, base_url=http://127.0.0.1:45660/_ozv/vs, leader=true}
[junit4:junit4]   2> 676406 T2075 C6376 P45660 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a692b2 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1a163e0),segFN=segments_1,generation=1}
[junit4:junit4]   2> 676407 T2075 C6376 P45660 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 676408 T2075 C6376 P45660 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a692b2 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1a163e0),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a692b2 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1a163e0),segFN=segments_2,generation=2}
[junit4:junit4]   2> 676408 T2075 C6376 P45660 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 676410 T2075 C6376 P45660 oass.SolrIndexSearcher.<init> Opening Searcher< at >1e42cde realtime
[junit4:junit4]   2> 676411 T2075 C6376 P45660 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 676411 T2075 C6376 P45660 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard1_replica2] webapp=/_ozv/vs path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4]   2> ASYNC  NEW_CORE C6377 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6377_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676422 T2030 C6377 P42419 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C6378 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6378_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676422 T2109 C6378 P37924 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676423 T2109 C6378 P37924 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> ASYNC  NEW_CORE C6379 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >1efb7ce url=http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2 node=127.0.0.1:45660__ozv%2Fvs C6379_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:45660__ozv%2Fvs, base_url=http://127.0.0.1:45660/_ozv/vs, leader=true}
[junit4:junit4]   2> 676426 T2138 C6379 P45660 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676427 T2138 C6379 P45660 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_ozv/vs path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 676427 T2109 C6378 P37924 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 676428 T2109 C6378 P37924 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 676428 T2109 C6378 P37924 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> ASYNC  NEW_CORE C6380 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >1efb7ce url=http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2 node=127.0.0.1:45660__ozv%2Fvs C6380_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:45660__ozv%2Fvs, base_url=http://127.0.0.1:45660/_ozv/vs, leader=true}
[junit4:junit4]   2> 676441 T2075 C6380 P45660 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_ozv/vs path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C6381 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6381_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676442 T2109 C6381 P37924 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> ASYNC  NEW_CORE C6382 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6382_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676445 T2030 C6382 P42419 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >18e74bc lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1aa869b),segFN=segments_1,generation=1}
[junit4:junit4]   2> 676445 T2109 C6381 P37924 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index.20130523083531740
[junit4:junit4]   2> 676448 T2109 C6381 P37924 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >1e6f142 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1da5c75) fullCopy=false
[junit4:junit4]   2> ASYNC  NEW_CORE C6383 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6383_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676480 T2030 C6383 P42419 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> ASYNC  NEW_CORE C6384 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore< at >1efb7ce url=http://127.0.0.1:45660/_ozv/vs/nodes_used_collection_shard1_replica2 node=127.0.0.1:45660__ozv%2Fvs C6384_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:45660__ozv%2Fvs, base_url=http://127.0.0.1:45660/_ozv/vs, leader=true}
[junit4:junit4]   2> 676488 T2138 C6384 P45660 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_ozv/vs path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C6385 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6385_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676522 T2109 C6385 P37924 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> ASYNC  NEW_CORE C6386 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6386_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676522 T2030 C6386 P42419 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >18e74bc lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1aa869b),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >18e74bc lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1aa869b),segFN=segments_2,generation=2}
[junit4:junit4]   2> 676523 T2030 C6386 P42419 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 676524 T2030 C6386 P42419 oass.SolrIndexSearcher.<init> Opening Searcher< at >74f2b9 realtime
[junit4:junit4]   2> 676525 T2030 C6386 P42419 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 676525 T2030 C6386 P42419 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard2_replica2] webapp=/_ozv/vs path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 103
[junit4:junit4]   2> ASYNC  NEW_CORE C6387 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6387_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676527 T2108 C6387 P46482 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676527 T2108 C6387 P46482 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 676529 T2030 C6386 P42419 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676529 T2030 C6386 P42419 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_ozv/vs path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 676530 T2108 C6387 P46482 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 676531 T2108 C6387 P46482 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 676531 T2108 C6387 P46482 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 676533 T2142 C6386 P42419 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_ozv/vs path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 676534 T2108 C6387 P46482 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 676535 T2108 C6387 P46482 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index.20130523083531831
[junit4:junit4]   2> 676536 T2108 C6387 P46482 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >13e8e84 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >3d853d) fullCopy=false
[junit4:junit4]   2> ASYNC  NEW_CORE C6388 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6388_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676553 T2109 C6388 P37924 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 676553 T2109 C6388 P37924 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> ASYNC  NEW_CORE C6389 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore< at >1631478 url=http://127.0.0.1:42419/_ozv/vs/nodes_used_collection_shard2_replica2 node=127.0.0.1:42419__ozv%2Fvs C6389_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:42419__ozv%2Fvs, base_url=http://127.0.0.1:42419/_ozv/vs, leader=true}
[junit4:junit4]   2> 676555 T2030 C6389 P42419 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_ozv/vs path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C6390 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6390_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676558 T2108 C6390 P46482 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> ASYNC  NEW_CORE C6391 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6391_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676569 T2109 C6391 P37924 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >fc5a40 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >773caf),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >fc5a40 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >773caf),segFN=segments_2,generation=2}
[junit4:junit4]   2> 676570 T2109 C6391 P37924 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> ASYNC  NEW_CORE C6392 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6392_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676570 T2108 C6392 P46482 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 676571 T2108 C6392 P46482 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676570 T2109 C6391 P37924 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 676571 T2109 C6391 P37924 oass.SolrIndexSearcher.<init> Opening Searcher< at >1f8a691 main
[junit4:junit4]   2> 676591 T2105 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher< at >1f8a691 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> ASYNC  NEW_CORE C6393 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6393_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676592 T2109 C6393 P37924 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index.20130523083531740 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index.20130523083531740;done=true>>]
[junit4:junit4]   2> 676593 T2109 C6393 P37924 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index.20130523083531740
[junit4:junit4]   2> 676593 T2109 C6393 P37924 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/nodes_used_collection_shard1_replica1/data/index.20130523083531740
[junit4:junit4]   2> 676594 T2109 C6393 P37924 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676594 T2109 C6393 P37924 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 676594 T2109 C6393 P37924 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=active
[junit4:junit4]   2> ASYNC  NEW_CORE C6394 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6394_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676602 T2108 C6394 P46482 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a48637 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1f8a578),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >a48637 lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1f8a578),segFN=segments_2,generation=2}
[junit4:junit4]   2> ASYNC  NEW_CORE C6395 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore< at >131f7ef url=http://127.0.0.1:37924/_ozv/vs/nodes_used_collection_shard1_replica1 node=127.0.0.1:37924__ozv%2Fvs C6395_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:37924__ozv%2Fvs, base_url=http://127.0.0.1:37924/_ozv/vs}
[junit4:junit4]   2> 676616 T2109 C6395 P37924 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> ASYNC  NEW_CORE C6396 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore< at >11988c8 url=http://127.0.0.1:46482/_ozv/vs/nodes_used_collection_shard2_replica1 node=127.0.0.1:46482__ozv%2Fvs C6396_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:46482__ozv%2Fvs, base_url=http://127.0.0.1:46482/_ozv/vs}
[junit4:junit4]   2> 676617 T2108 C6396 P46482 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 676617 T2108 C6396 P46482 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 676618 T2108 C6396 P46482 oass.SolrIndexSearcher.<init> Opening Searcher< at >15a2edd main
[junit4:junit4]   2> 676619 T2103 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher< at >15a2edd main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 676628 T2108 C6396 P46482 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index.20130523083531831 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index.20130523083531831;done=true>>]
[junit4:junit4]   2> 676629 T2108 C6396 P46482 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index.20130523083531831
[junit4:junit4]   2> 676629 T2108 C6396 P46482 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369312518342/nodes_used_collection_shard2_replica1/data/index.20130523083531831
[junit4:junit4]   2> 676629 T2108 C6396 P46482 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676630 T2108 C6396 P46482 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676630 T2108 C6396 P46482 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=active
[junit4:junit4]   2> 676634 T2108 C6396 P46482 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 676826 T2037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 676827 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:37924__ozv%2Fvs_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard1",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:37924__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:37924/_ozv/vs"}
[junit4:junit4]   2> 676954 T2037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>   "operation":"state",
[junit4:junit4]   2>   "core_node_name":"127.0.0.1:46482__ozv%2Fvs_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "numShards":"2",
[junit4:junit4]   2>   "shard":"shard2",
[junit4:junit4]   2>   "roles":null,
[junit4:junit4]   2>   "state":"active",
[junit4:junit4]   2>   "shard_state":"active",
[junit4:junit4]   2>   "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2>   "collection":"nodes_used_collection",
[junit4:junit4]   2>   "shard_range":null,
[junit4:junit4]   2>   "node_name":"127.0.0.1:46482__ozv%2Fvs",
[junit4:junit4]   2>   "base_url":"http://127.0.0.1:46482/_ozv/vs"}
[junit4:junit4]   2> 676974 T2054 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2096 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 676974 T2041 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 677400 T2078 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 677536 T2078 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 677558 T2078 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 677601 T2078 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 677657 T2078 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 677707 T2078 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 677709 T2078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 677710 T2078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 677710 T2078 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 677712 T2078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 677712 T2078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 677713 T2078 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 677714 T2078 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data/
[junit4:junit4]   2> 677714 T2078 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 677715 T2078 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 677733 T2078 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data
[junit4:junit4]   2> 677734 T2078 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data/index/
[junit4:junit4]   2> 677735 T2078 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 677735 T2078 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data/index
[junit4:junit4]   2> 677738 T2078 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory< at >18fd51a lockFactory=org.apache.lucene.store.NativeFSLockFactory< at >1390f08),segFN=segments_1,generation=1}
[junit4:junit4]   2> 677738 T2078 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 677741 T2078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 677742 T2078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 677743 T2078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 677744 T2078 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 677745 T2078 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 677745 T2078 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 677746 T2078 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 677747 T2078 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 677747 T2078 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 677767 T2078 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 677814 T2078 oass.SolrIndexSearcher.<init> Opening Searcher< at >1f92596 main
[junit4:junit4]   2> 677815 T2078 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369312520765/awholynewcollection_0_shard2_replica2/data/tlog
[junit4:junit4]   2> 677817 T2078 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 677817 T2078 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 677837 T2147 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard2_replica2] Registered new searcher Searcher< at >1f92596 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 677839 T2078 oasc.ZkController.publish publishing core=awholynewcollection_0_shard2_replica2 state=down
[junit4:junit4]   2> 677973 T2090 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 678010 T2090 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 678052 T2090 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 678120 T2134 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 678153 T2090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 678170 T2134 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 678177 T2134 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 678216 T2090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 678243 T2029 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 678244 T2134 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 678248 T2134 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 678258 T2049 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 678267 T2029 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 678285 T2091 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 678305 T2091 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 678324 T2049 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 678330 T2049 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 678348 T2090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 678350 T2134 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 678351 T2090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 678352 T2090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 678352 T2090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 678353 T2090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 678353 T2090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 678353 T2134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 678355 T2134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 678354 T2090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 678356 T2090 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard2_replica1/data/
[junit4:junit4]   2> 678356 T2090 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer< at >9fa8f
[junit4:junit4]   2> 678357 T2090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 678354 T2029 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 678355 T2134 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 678370 T2090 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369312521997/awholynewcollection_0_shard2_replica1/data
[junit4:junit4]   2> 678370 T2090 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Luc

[...truncated too long message...]

tedZkTest.java:512)
[junit4:junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 939518 T2014 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 276953 T2013 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 940709 T2096 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard1/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 940711 T2096 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 940711 T2096 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 940711 T2096 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 940712 T2096 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard4/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 940713 T2096 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard2/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> 940714 T2096 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard3/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2> 
[junit4:junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ko, timezone=America/Tortola
[junit4:junit4]   2> NOTE: Linux 3.2.0-41-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=1,free=83421152,total=213757952
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ClusterStateTest, ZkNodePropsTest, SampleTest, TestRecovery, TestSolr4Spatial, TestPropInject, SpellCheckComponentTest, TestHashPartitioner, SolrPluginUtilsTest, PingRequestHandlerTest, ShardRoutingCustomTest, TestCollationField, TestSolrQueryParserDefaultOperatorResource, SolrRequestParserTest, TestNumberUtils, SortByFunctionTest, NoCacheHeaderTest, IndexSchemaRuntimeFieldTest, TestArbitraryIndexDir, CurrencyFieldXmlFileTest, PathHierarchyTokenizerFactoryTest, TestUpdate, DateMathParserTest, SolrCmdDistributorTest, FieldAnalysisRequestHandlerTest, TestSuggestSpellingConverter, DocValuesMultiTest, TestSchemaSimilarityResource, StatsComponentTest, TestSystemIdResolver, TestStressReorder, TestDistributedSearch, SolrIndexConfigTest, SignatureUpdateProcessorFactoryTest, TestCoreDiscovery, IndexBasedSpellCheckerTest, URLClassifyProcessorTest, HighlighterTest, JSONWriterTest, TestZkChroot, TestReload, TestFieldTypeResource, SolrCoreCheckLockOnStartupTest, TestBM25SimilarityFactory, FullSolrCloudDistribCmdsTest, TestAddFieldRealTimeGet, TestCSVResponseWriter, IndexReaderFactoryTest, TestSchemaNameResource, TestUniqueKeyFieldResource, TestStressLucene, TestMultiCoreConfBootstrap, TestRangeQuery, ChaosMonkeySafeLeaderTest, FileUtilsTest, BasicDistributedZk2Test, TestRandomFaceting, LeaderElectionIntegrationTest, TestUtils, TestSolrJ, TestRTGBase, TestPartialUpdateDeduplication, CopyFieldTest, DistanceFunctionTest, UpdateParamsTest, NumericFieldsTest, SolrTestCaseJ4Test, TestDynamicFieldCollectionResource, TestPostingsSolrHighlighter, TestWordDelimiterFilterFactory, CollectionsAPIDistributedZkTest]
[junit4:junit4] Completed on J1 in 278.25s, 1 test, 1 failure <<< FAILURES!

[...truncated 461 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:887: There were test failures: 296 suites, 1234 tests, 1 failure, 13 ignored (7 assumptions)

Total time: 43 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe< at >lucene.apache.org
For additional commands, e-mail: dev-help< at >lucene.apache.org