gmane.comp.jakarta.lucene.hadoop.devel

[jira] [Created] (HADOOP-8406) CompressionCodecFactory.CODEC_PROVIDERS iteration is thread-unsafe

Todd Lipcon (JIRA — 2012-05-17T00:56:25

Todd Lipcon created HADOOP-8406:
-----------------------------------

             Summary: CompressionCodecFactory.CODEC_PROVIDERS iteration is thread-unsafe
                 Key: HADOOP-8406
                 URL: https://issues.apache.org/jira/browse/HADOOP-8406
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 2.0.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


CompressionCodecFactory defines CODEC_PROVIDERS as:
{code}
  private static final ServiceLoader<CompressionCodec> CODEC_PROVIDERS =
    ServiceLoader.load(CompressionCodec.class);
{code}
but this is a lazy collection which is thread-unsafe to iterate. We either need to synchronize when we iterate over it, or we need to materialize it during class-loading time by copying to a non-lazy collection

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/Conta

[jira] [Resolved] (HADOOP-8405) ZKFC tests leak ZK instances

Todd Lipcon (JIRA — 2012-05-17T00:40:07

     [ https://issues.apache.org/jira/browse/HADOOP-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-8405.
---------------------------------

       Resolution: Fixed
    Fix Version/s: Auto Failover (HDFS-3042)
     Hadoop Flags: Reviewed

Committed to branch, thanks Eli.
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8405) ZKFC tests leak ZK instances

Todd Lipcon (JIRA — 2012-05-16T22:45:07

Todd Lipcon created HADOOP-8405:
-----------------------------------

             Summary: ZKFC tests leak ZK instances
                 Key: HADOOP-8405
                 URL: https://issues.apache.org/jira/browse/HADOOP-8405
             Project: Hadoop Common
          Issue Type: Bug
          Components: auto-failover, test
    Affects Versions: Auto Failover (HDFS-3042)
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
         Attachments: hadoop-8405.txt

The ZKFC code wasn't previously terminating the ZK connection in all cases where it should (eg after a failed startup or after formatting ZK). This didn't cause a problem for CLI usage, since the process exited afterwards, but caused the test results to get clouded with a lot of "Reconecting to ZK" messages, which make the logs hard to read.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/se

[jira] [Resolved] (HADOOP-8404) etc/hadoop in binary tarball missing hadoop-env.sh

Eli Collins (JIRA — 2012-05-16T22:35:07

     [ https://issues.apache.org/jira/browse/HADOOP-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-8404.
---------------------------------

    Resolution: Duplicate

Dupe of HADOOP-8287
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8404) etc/hadoop in binary tarball missing hadoop-env.sh

Todd Lipcon (JIRA — 2012-05-16T22:09:07

Todd Lipcon created HADOOP-8404:
-----------------------------------

             Summary: etc/hadoop in binary tarball missing hadoop-env.sh
                 Key: HADOOP-8404
                 URL: https://issues.apache.org/jira/browse/HADOOP-8404
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 2.0.0
            Reporter: Todd Lipcon


todd< at >todd-w510:~/releases/hadoop-2.0.0-alpha$ ls etc/hadoop/
core-site.xml               hdfs-site.xml            httpfs-signature.secret  slaves                  yarn-env.sh
hadoop-metrics2.properties  httpfs-env.sh            httpfs-site.xml          ssl-client.xml.example  yarn-site.xml
hadoop-metrics.properties   httpfs-log4j.properties  log4j.properties         ssl-server.xml.example


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atla

Re: [VOTE] Hadoop-1.0.3-rc1

Matt Foley — 2012-05-16T19:19:50

I'm pleased to announce that as of last night, the 1.0.3 vote passed with
  +1 binding - 4 votes
  +1 non-binding - 1 vote
   -1  - no votes

Thanks to everyone who tried it out.
I'm in process of publishing it and will announce on hadoop general shortly.

--Matt
Release Manager, Hadoop 1.0


On Tue, May 8, 2012 at 6:25 PM, Matt Foley <mattf-1oDqGaOF3Lkdnm+yROfE0A< at >public.gmane.org> wrote:

[jira] [Created] (HADOOP-8403) bump up POMs version to 2.0.1-SNAPSHOT

Alejandro Abdelnur (JIRA — 2012-05-16T18:27:07

Alejandro Abdelnur created HADOOP-8403:
------------------------------------------

             Summary: bump up POMs version to 2.0.1-SNAPSHOT
                 Key: HADOOP-8403
                 URL: https://issues.apache.org/jira/browse/HADOOP-8403
             Project: Hadoop Common
          Issue Type: Task
          Components: build
    Affects Versions: 2.0.1
            Reporter: Alejandro Abdelnur
            Assignee: Alejandro Abdelnur
             Fix For: 2.0.1




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8402) Add support for generating pdf clover report to 1.1 release

Suresh Srinivas (JIRA — 2012-05-16T17:51:08

Suresh Srinivas created HADOOP-8402:
---------------------------------------

             Summary: Add support for generating pdf clover report to 1.1 release
                 Key: HADOOP-8402
                 URL: https://issues.apache.org/jira/browse/HADOOP-8402
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 1.0.0
            Reporter: Suresh Srinivas
            Priority: Minor


Add support for generating clover PDF report.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Jenkins build is back to normal : Hadoop-Common-0.23-Build #254

Apache Jenkins Server — 2012-05-16T09:20:04

See <https://builds.apache.org/job/Hadoop-Common-0.23-Build/254/changes>

Build failed in Jenkins: Hadoop-Common-trunk #408

Apache Jenkins Server — 2012-05-16T09:12:30

See <https://builds.apache.org/job/Hadoop-Common-trunk/408/changes>

Changes:

[todd] Move changes from 2.0.0-alpha to 2.0.1-alpha in CHANGES.txt for those that did not make rc1

[tucu] MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu)

[tucu] HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu)

[eli] HDFS-3197. Incorrect class comments in a few tests. Contributed by Andy Isaacson

[tgraves] MAPREDUCE-4238. mavenize data_join. (tgraves)

[eli] HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs. Contributed by Eli Collins

[eli] HADOOP-8398. Cleanup BlockLocation. Contributed by Eli Collins

[eli] HDFS-3419. Cleanup LocatedBlock. Contributed by Eli Collins

[eli] HDFS-3416. Cleanup DatanodeID and DatanodeRegistration constructors used by testing. Contributed by Eli Collins

[eli] HDFS-3417. Rename BalancerDatan

[jira] [Created] (HADOOP-8401) Investigate use of JobObject to spawn tasks on Windows

Bikas Saha (JIRA — 2012-05-15T23:13:08

Bikas Saha created HADOOP-8401:
----------------------------------

             Summary: Investigate use of JobObject to spawn tasks on Windows
                 Key: HADOOP-8401
                 URL: https://issues.apache.org/jira/browse/HADOOP-8401
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 1.0.0
            Reporter: Bikas Saha




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Hadoop-1.0.3-rc1

Devaraj Das — 2012-05-15T21:25:47

Hey Luke, as you know, the security stuff hasn't been tested with IBM's JDK. I'll submit a patch for 8251 but I wouldn't block 1.0.3 for this.
Thanks
Devaraj

On May 15, 2012, at 2:20 PM, Luke Lu wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Luke Lu — 2012-05-15T21:21:53

Sorry, meant for Devaraj only :)

On Tue, May 15, 2012 at 2:20 PM, Luke Lu <llu-S4k2aNtRk8rQT0dZR+AlfA< at >public.gmane.org> wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Luke Lu — 2012-05-15T21:20:10

Hey Devaraj,

Thanks for testing IBM JDK. Can you do the one line fix for IBM JDK as
well for HADOOP-8251, as Todd only fixed Sun JDK. It's not blocker for
1.0.3. But much easier for you to submit a patch. I already have 14
patches in the queue waiting for our legal approval. Doing a code scan
for another one line patch would be inefficient :)

Thanks,

__Luke

On Tue, May 15, 2012 at 12:05 PM, Devaraj Das <ddas-RYNwJFaOa9CEK/hMebVsMw< at >public.gmane.org> wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Suresh Srinivas — 2012-05-15T19:53:36

I download all the artifacts and verified checksums for all the tar bundle,
rpms, debs. Ran on my machine, using tar bundle, TestDFSIO tests,
TestMapRed, mrbench and nnbench.

+1 for the 1.0.3-rc1 release.


On Tue, May 8, 2012 at 6:25 PM, Matt Foley <mattf-1oDqGaOF3Lkdnm+yROfE0A< at >public.gmane.org> wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Matt Foley — 2012-05-15T19:06:35

[Sorry I accidentally hit "Reply" instead of "Reply All" on thread below.
 Adding back common-dev< at > ]

I'm planning to do a first turn on 1.1 next week.  Because so much new
stuff went into it, I'm anticipating that there will need to be some
stabilization before I can call a vote on an RC.
--Matt

On Tue, May 15, 2012 at 11:28 AM, Konstantin Boudnik <cos-1oDqGaOF3Lkdnm+yROfE0A< at >public.gmane.org> wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Devaraj Das — 2012-05-15T19:05:58

+1 (binding)
I downloaded the release, built (non-secure) with IBM's JDK [IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr10-20111207_96808 (JIT enabled, AOT enabled)]. Ran the sleep job successfully on a single node cluster. 

On May 8, 2012, at 6:25 PM, Matt Foley wrote:

Re: Hadoop - Distributed sorting

samir das mohapatra — 2012-05-15T18:35:44

Hi
  Steps to do this:
1) Map: It will only define the key value for each number
 2) Combiner : To sort locally  over chunk of dataset .
 3) Reducer: It will sort after over whole chunk globally-------------->
OUT PUT as sorted

Note: set combiner and reducer as Same class.

Example:
  Let us assume that our data set (integers) is constrained between 100 to
200 and we have 5 files each containing 1000 random integers between 100
and 200 (so a total of 5000 integers between 100 and 200). We read each
file into a Map and then in the Reduce phase, we produce a final Map which
contains the count of all the integers. Now if we sort all the integers
from the final Map and output it
into a list data structure in the form of <Integer, Count> then we have
sorted all the data (see figure below). Aside : In Java, you don’t even
have to come up with the data-structure that I am talking about, if you
just use a TreeMap<http://java.sun.com/javase/6/docs/api/index.html?java/util/TreeMap.html>in
the final Reduce phase, th

Re: Hadoop - Distributed sorting

2012-05-15T18:01:58

Check out Pentaho's howto's when dealing with Hadoop or NoSQL or anything big
data related. http://wiki.pentaho.com/display/BAD/How+To%27s


madhu_sushmi wrote:

Re: [VOTE] Hadoop-1.0.3-rc1

Arpit Gupta — 2012-05-15T17:03:35

+1 (non binding)

ran system tests on secure and non secure deploys and found no regression issues.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On May 8, 2012, at 6:25 PM, Matt Foley wrote:

[jira] [Created] (HADOOP-8400) All commands warn "Kerberos krb5 configuration not found" when security is not enabled

Eli Collins (JIRA — 2012-05-15T16:39:38

Eli Collins created HADOOP-8400:
-----------------------------------

             Summary: All commands warn "Kerberos krb5 configuration not found" when security is not enabled
                 Key: HADOOP-8400
                 URL: https://issues.apache.org/jira/browse/HADOOP-8400
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
    Affects Versions: 2.0.0
            Reporter: Eli Collins
            Assignee: Alejandro Abdelnur


Post HADOOP-8086 I get "Kerberos krb5 configuration not found, setting default realm to empty" warnings when running Hadoop commands even though I don't have kerb enabled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira