gmane.comp.jakarta.lucene.hadoop.devel

[jira] Commented: (HADOOP-4649) Improve abstraction for spill indices

Jothi Padmanabhan (JIRA — 2008-12-02T05:15:44

[ https://issues.apache.org/jira/browse/HADOOP-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652273#action_12652273 ] Jothi Padmanabhan commented on HADOOP-4649: ------------------------------------------- Currently, the IFile*Streams do exactly what you want for SpillRecords - namely create/validate a checksum at the end of the file. So, to avoid duplication of code, we could just use them here. If at a later stage, if we find that the CRC algorithm for IFIle is affecting the performance for SpillRecord or we change these streams when modifying the IFile layout, we could revisit this, no? I am fine with the existing patch but would like to avoid duplication of the code, if we could. I am +1 otherwise.

Maven vs Ivy

Ashish Thusoo — 2008-12-02T04:53:51

Folks, I found that sometime back there was some discussion on which of these to use in order to manage dependencies between sub-projects and hadoop core (I think a lot of it was in the context of Zookeeper). It seemed from that discussion that there was a lean towards using Ivy in order to manage dependencies for subprojects. Is that correct? Is that recommended route for the subprojects? What are the advantages/disadvantages of using one over the other? Thanks, Ashish

[jira] Updated: (HADOOP-4742) Mistake delete replica in hadoop 0.18.1

Wang Xu (JIRA — 2008-12-02T04:57:44

[ https://issues.apache.org/jira/browse/HADOOP-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Xu updated HADOOP-4742: ---------------------------- Attachment: HADOOP-4742.diff

[jira] Updated: (HADOOP-4742) Mistake delete replica in hadoop 0.18.1

Wang Xu (JIRA — 2008-12-02T04:57:44

[jira] Commented: (HADOOP-4716) testRestartWithLostTracker frequently times out

Amar Kamat (JIRA — 2008-12-02T04:37:44

[ https://issues.apache.org/jira/browse/HADOOP-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652270#action_12652270 ] Amar Kamat commented on HADOOP-4716: ------------------------------------ Also the following tweaks causes the test _TestJobTrackerRestartWithLostTracker_ to pass consistently. - _timeout_ : Set DEFAULT_READ_TIMEOUT = 3 sec (default = 3min) and DEFAULT_CONNECT_TIMEOUT = 100 msecs (default = 3 secs) in {{ReduceTask.java}}. - _freshness_ : clearing the _known output_ list in {{ReduceTask.java}} upon a restart.

[jira] Commented: (HADOOP-4278) TestDatanodeDeath failed occasionally

Tsz Wo (Nicholas), SZE (JIRA — 2008-12-02T04:33:44

[ https://issues.apache.org/jira/browse/HADOOP-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652268#action_12652268 ] Tsz Wo (Nicholas), SZE commented on HADOOP-4278: ------------------------------------------------ primaryDeath_0.18.patch does not make sense unless the DFSClient.processDatanodeError(..) codes in HADOOP-1700 are checked in to 0.18. Dhruba, should the DFSClient.processDatanodeError(..) in HADOOP-1700 codes should be committed to 0.18?

[jira] Commented: (HADOOP-4716) testRestartWithLostTracker frequently times out

Amar Kamat (JIRA — 2008-12-02T04:29:44

[ https://issues.apache.org/jira/browse/HADOOP-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652267#action_12652267 ] Amar Kamat commented on HADOOP-4716: ------------------------------------ Also, the reducers build up a list of known map outputs from the map completion events obtained from the task-tracker. Upon restart (HADOOP-3245), the reducer doesn't clear this list. The problem arises (on a very small cluster e.g. test cases) when a map completes and this completion event is not flushed to the history file (in buffer) and the tracker on which it ran gets lost. In such a case the (new) JobTracker has no idea about the map and since the reducer's list is stale, it takes some time to figure out that the map location is bad and use the other (re-executed by the _new_ JobTracker) one. Note than on a large cluster, this should not be an issue as, upon failure, new maps from different host will be tried and after sometime the new location for such (dangling) maps will be pass ed by the JobTracker. We have 2 choices - stale data : This can help in cases where a tracker has not yet joined but the data (map's output) is still valid/available. The drawback being the case where a tracker is lost and data becomes unavailable. In such a case the location will be retried again and again until the (newly re-executed) map's output is pulled from some other tracker. Here the time will be wasted in pulling map output from a lost tracker/node and waiting for the (dangling) maps (from that node) to be re-executed. - fresh data : This can help in cases where few trackers go down. The drawback being that trackers that are up and ready to serve the map output will be ignored since they are yet to join. Here the time will be wasted in waiting for the tracker to _formally_ join back.

[jira] Updated: (HADOOP-4035) Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory

Vinod K V (JIRA — 2008-12-02T04:25:44

[ https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated HADOOP-4035: ------------------------------ Attachment: HADOOP-4035-20081202.txt Attaching new patch. I had done changes to capacity-scheduler.xml.template earlier itself, somehow that got missed in the patch, adding it now. The testcase TestCapacityScheduler.testHighMemoryJobWithInvalidRequirements was failing intermittently because of timing issues with JobInitializationPoller. Fixed that now by using the ControlledJobIntializationPoller. Incorporated the rest of the review comments too.

[jira] Commented: (HADOOP-4445) Wrong number of running map/reduce tasks are displayed in queue information.

Sreekanth Ramakrishnan (JIRA — 2008-12-02T04:21:45

[ https://issues.apache.org/jira/browse/HADOOP-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652265#action_12652265 ] Sreekanth Ramakrishnan commented on HADOOP-4445: ------------------------------------------------ Checked with 80 jobs in a queue, the maximum time taken for getting running and pending tasks was 2 milliseconds(inclusive of scheduler.getJobs()) by refreshing the job tracker and queue details page every 15 seconds. So if we have 10 queues, we would have a maximum delay of 20 milliseconds when user requests for information. This can be further reduced if we get in [HADOOP-4576|https://issues.apache.org/jira/browse/HADOOP-4576] as then we would count tasks in running job queue and then subtract the running job number from total job number to get waiting job count.

[jira] Updated: (HADOOP-4726) documentation typos: "the the"

Edward J. Yoon (JIRA — 2008-12-02T02:49:44

[ https://issues.apache.org/jira/browse/HADOOP-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon updated HADOOP-4726: ----------------------------------- Attachment: HADOOP-4726_v04.patch Thanks for your review. I fixed them. ---- [root< at >udanax hadoop-0.18]# patch -p0 < ~/Desktop/HADOOP-4726_v04.patch ... + patching file src/mapred/org/apache/hadoop/mapred/ReduceTask.java ... [root< at >udanax hadoop-0.18]# svn diff > a.patch [root< at >udanax hadoop-0.18]# cat a.patch Index: src/docs/src/documentation/content/xdocs/streaming.xml =================================================================== --- src/docs/src/documentation/content/xdocs/streaming.xml (revision 722337) +++ src/docs/src/documentation/content/xdocs/streaming.xml (working copy) < at >< at > -48,11 +48,11 < at >< at > ....

[jira] Commented: (HADOOP-4635) Memory leak ?

Pete Wyckoff (JIRA — 2008-12-02T02:11:44

[ https://issues.apache.org/jira/browse/HADOOP-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652235#action_12652235 ] Pete Wyckoff commented on HADOOP-4635: -------------------------------------- failure in Hudson not due to this patch which is fuse-dfs c code only.

[jira] Commented: (HADOOP-4726) documentation typos: "the the"

Tsz Wo (Nicholas), SZE (JIRA — 2008-12-02T01:41:44

[ https://issues.apache.org/jira/browse/HADOOP-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652231#action_12652231 ] Tsz Wo (Nicholas), SZE commented on HADOOP-4726: ------------------------------------------------ - ReduceTask.java is missing in HADOOP-4726_v02.patch. - Seems there are some EOL changes in streaming.xml. It confuses svn. Tried to apply HADOOP-4726.patch (or HADOOP-4726_v02.patch) and than did "svn diff > a.patch". The resulted a.patch contained the entire streaming.xml file.

[jira] Updated: (HADOOP-4713) librecordio does not scale to large records

Chris Douglas (JIRA — 2008-12-02T01:39:44

[ https://issues.apache.org/jira/browse/HADOOP-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-4713: ---------------------------------- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I just committed this. Thanks, Christian

[jira] Updated: (HADOOP-4746) Job output directory should be normalized

Hairong Kuang (JIRA — 2008-12-02T01:27:44

[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HADOOP-4746: ---------------------------------- Attachment: outDir1.patch This patch has two changes: 1. make FileOutputFormat#setOutputPath to throw an IOException; 2. The output dir could be in a file system which is different from the one defined in jobconf; So use Path.getFileSystem to get the file system handle.

[jira] Issue Comment Edited: (HADOOP-4747) Reuse FileStatus in FsShell where possible

David Phillips (JIRA — 2008-12-02T01:05:44

[ https://issues.apache.org/jira/browse/HADOOP-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652225#action_12652225 ] electrum edited comment on HADOOP-4747 at 12/1/08 5:04 PM: ----------------------------------------------------------------- Change main ls() function and shellListStatus() to take a FileStatus rather than a Path. shellListStatus() can return immediately if the FileStatus is not a directory. This has significant speedups for -ls when globbing: hadoop fs -ls /foo/ (fast) hadoop fs -ls /foo/*.dat (previously made a status call for every file) was (Author: electrum): Change main ls() function and shellListStatus() to take a FileStatus rather than a Path. shellListStatus() can return immediately if the FileStatus is a directory. This has significant speedups for -ls when globbing: hadoop fs -ls /foo/ (fast) hadoop fs -ls /foo/*.dat (previously made a status call for every file)

[jira] Updated: (HADOOP-4747) Reuse FileStatus in FsShell where possible

David Phillips (JIRA — 2008-12-02T01:05:44

[jira] Updated: (HADOOP-4747) Reuse FileStatus in FsShell where possible

David Phillips (JIRA — 2008-12-02T01:03:44

[ https://issues.apache.org/jira/browse/HADOOP-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Phillips updated HADOOP-4747: ----------------------------------- Attachment: hadoop-fsshell-reuse-status.patch Change main ls() function and shellListStatus() to take a FileStatus rather than a Path. shellListStatus() can return immediately if the FileStatus is a directory. This has significant speedups for -ls when globbing: hadoop fs -ls /foo/ (fast) hadoop fs -ls /foo/*.dat (previously made a status call for every file)

[jira] Created: (HADOOP-4747) Reuse FileStatus in FsShell where possible

David Phillips (JIRA — 2008-12-02T00:55:44

Reuse FileStatus in FsShell where possible ------------------------------------------ Key: HADOOP-4747 URL: https://issues.apache.org/jira/browse/HADOOP-4747 Project: Hadoop Core Issue Type: Bug Components: fs Affects Versions: 0.19.0 Reporter: David Phillips Priority: Minor FsShell should reuse FileStatus objects instead of converting to a Path and making extra calls to the backend FS (which can be slow and expensive).

[jira] Updated: (HADOOP-4726) documentation typos: "the the"

Edward J. Yoon (JIRA — 2008-12-02T00:35:44

[ https://issues.apache.org/jira/browse/HADOOP-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon updated HADOOP-4726: ----------------------------------- Attachment: HADOOP-4726_v02.patch Oh, sorry. the previous patch is clearly wrong. please try this patch. ---- [root< at >udanax hadoop-0.18]# patch -p0 < HADOOP-4726_v02.patch patching file conf/hadoop-default.xml patching file docs/jdiff/hadoop_0.17.0.xml patching file docs/jdiff/hadoop_0.18.1.xml patching file docs/jdiff/hadoop_0.18.2.xml patching file src/docs/src/documentation/content/xdocs/cluster_setup.xml patching file src/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml patching file src/docs/src/documentation/content/xdocs/mapred_tutorial.xml patching file src/docs/src/documentation/content/xdocs/streaming.xml [root< at >udanax hadoop-0.18]#

[jira] Updated: (HADOOP-4278) TestDatanodeDeath failed occasionally

Tsz Wo (Nicholas), SZE (JIRA — 2008-12-02T00:33:44

[ https://issues.apache.org/jira/browse/HADOOP-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-4278: ------------------------------------------- Attachment: primaryDeath_0.18.patch primaryDeath_0.18.patch: for 0.18

[jira] Commented: (HADOOP-4746) Job output directory should be normalized

Arun C Murthy (JIRA — 2008-12-02T00:13:44

[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652214#action_12652214 ] Arun C Murthy commented on HADOOP-4746: --------------------------------------- The patch is fine, though ignoring the exception bothers me. Should we atleast log that exception, if not abort?