http://blog.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50418 EC2 scripts should configure Hadoop to use all available disks on large instances --------------------------------------------------------------------------------- Key: HADOOP-4745 URL: https://issues.apache.org/jira/browse/HADOOP-4745 Project: Hadoop Core Issue Type: Improvement Components: contrib/ec2 Affects Versions: 0.19.0 Reporter: Tom White Assignee: Tom White Fix For: 0.20.0 The Hadoop configuration on EC2 currently always uses a single disk, even when more are available (http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-storage.html). Performance is significantly boosted by using all the available disks, so we should configure Hadoop to use them automatically. Tom White (JIRA 2008-12-01T14:50:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50411 Hi,all Hadoop Beijing Meeting has successfully concluded on Nov 23. Thank you all for your attention. According to the agreements reached in this meeting, we have finished setting up the hadoop-in-china nonprofit website:www.hadooper.cn. We wish we can form a powerful hadoop community in china. Take a look at this website and if you are interested in contributing for the hadooper-in-china community, please drop me an email. We have uploaded the Hadoop Beijing Meeting pictures, slides and videos on this www.hadooper.cn website, which are also available on the hadoop-in-china google group (http://groups.google.com/group/hadooper_cn). please let me know if you have any suggestions. Thanks! heyongqiang 2008-12-01 heyongqiang 2008-12-01T12:00:13 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50403 Wrong resolution of hostname and port -------------------------------------- Key: HADOOP-4744 URL: https://issues.apache.org/jira/browse/HADOOP-4744 Project: Hadoop Core Issue Type: Bug Components: mapred Affects Versions: 0.20.0 Reporter: Amareshwari Sriramadasu Priority: Critical Fix For: 0.20.0 I noticed the following for one of the hosts in a cluster: 1. machines.jsp page resolves the http address as just "http://hostname" (which doesn't work). It doesnt put the port number for the host. Even if I add the port number manually in the URI, the task tracker page does not come up. 2. All the tasks(both maps and reduces) which ran on the machine ran successfully. But tasklogs cannot be viewed, because port-number is not resolved. ( same problem as in (1)). 3. The reducers waiting for maps ran on that machine fail with connection failed errors saying the hostname is 'null'. Amareshwari Sriramadasu (JIRA 2008-12-01T10:29:45 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50370 [ https://issues.apache.org/jira/browse/HADOOP-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651770#action_12651770 ] Chris Douglas commented on HADOOP-1650: --------------------------------------- {quote} In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false. Any reason behind it? {quote} Deveraj ran into some GC issues in the last attempted migration (above) and some previous experiences with direct buffers left me wary. For the initial checkin, it made more sense to track down any stability problems we run into without the direct buffer noise. We can certainly revisit it when considering other optimizations and I certainly wouldn't be against making it configurable. Chris Douglas (JIRA 2008-11-30T01:04:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50365 [ https://issues.apache.org/jira/browse/HADOOP-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651752#action_12651752 ] Owen O'Malley commented on HADOOP-1230: --------------------------------------- The point is to simplify the code. Clearly they can retain a reference to it, but it will create lots of questions over whether they stay valid. Having the context object passed into the map and reduce methods makes it very easy and clear. Owen O'Malley (JIRA 2008-11-29T21:00:47 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50363 JobTracker TaskInitialization failure on cygwin ----------------------------------------------- Key: HADOOP-4743 URL: https://issues.apache.org/jira/browse/HADOOP-4743 Project: Hadoop Core Issue Type: Bug Environment: Hadoop 0.19.0 on Cygwin (Windows Vista Ultimate) and Ubuntu Linux 8.10 cluster Reporter: Masahiko Higashiyama I wanted to test new release and have been trying to queue new job. But it didn't work. This error is found in JobTracker's log. ======================================== 2008-11-29 21:39:00,276 ERROR org.apache.hadoop.mapred.EagerTaskInitializationListener: Job initialization failed: java.util.regex.PatternSyntaxException: Illegal/unsupported escape sequence near index 47 Leibnitz_[0-9]+_job_200811292137_0001_leibnitz\hadoop_\Qwordcount\E+ ^ at java.util.regex.Pattern.error(Unknown Source) at java.util.regex.Pattern.escape(Unknown Source) at java.util.rege Masahiko Higashiyama (JIRA 2008-11-29T18:30:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50355 Mistake delete replica in hadoop 0.18.1 --------------------------------------- Key: HADOOP-4742 URL: https://issues.apache.org/jira/browse/HADOOP-4742 Project: Hadoop Core Issue Type: Bug Components: dfs Affects Versions: 0.18.1 Environment: CentOS 5.2, JDK 1.6, 16 Datanodes and 1 Namenodes, each has 8GB Memory and a 4-core CPU, connected by GigabyteEthernet Reporter: Wang Xu We recently deployed a 0.18.1 cluster and did some test. And we found if we corrupt a block, the namenode will find it and replicate it as soon as a client read that block. However, the namenode will delete a health block (the source of the above replication operation) at the same time, (I think this issue may affect all 0.18 tree.) Having did some trace, I find in FSNamesystem.addStoredBlock(), it will check the number of replications after add the block to blocksMap: | NumberReplicas num = countNodes(storedBlock); | int numLiveRep Wang Xu (JIRA 2008-11-29T07:50:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50346 Concrete implementation of MultiFileInputFormat ------------------------------------------------ Key: HADOOP-4741 URL: https://issues.apache.org/jira/browse/HADOOP-4741 Project: Hadoop Core Issue Type: New Feature Components: mapred Affects Versions: 0.20.0 Reporter: Enis Soztutar Assignee: Enis Soztutar There has been a demand for a concrete implementation for MultiFileInputFormat. We should include one as a library. Enis Soztutar (JIRA 2008-11-28T16:23:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50345 See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/673/changes ------------------------------------------ [...truncated 2338 lines...] A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/Counters.java A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/LzoTextInputFormat.java A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/TaskUmbilicalProtocol.java A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/LimitTasksPerJobTaskScheduler.java A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/MRConstants.java A zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-trunk/workspace/trunk/src/mapred/org/apache/hadoop/mapred/TextInputFormat.java A zonestorage/hudson/home Apache Hudson Server 2008-11-28T14:54:24 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50334 Add visual cue to denote if a job is running or not in scheduler UI page ------------------------------------------------------------------------ Key: HADOOP-4740 URL: https://issues.apache.org/jira/browse/HADOOP-4740 Project: Hadoop Core Issue Type: Improvement Components: mapred Affects Versions: 0.19.0 Reporter: Sreekanth Ramakrishnan Assignee: Sreekanth Ramakrishnan Priority: Minor add visual cue to denote if a job is running or not in scheduler UI page. Sreekanth Ramakrishnan (JIRA 2008-11-28T11:09:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50325 Hi list, We recently deployed a 0.18.1 cluster and did some test. And we found if we corrupt a block, the namenode will find it and replicate it as soon as a client read that block. However, the namenode will delete a health block (the source of the above replication operation) at the same time, (I think this issue may affect all 0.18 tree.) Having did some trace, I find in FSNamesystem.addStoredBlock(), it will check the number of replications after add the block to blocksMap: | NumberReplicas num = countNodes(storedBlock); | int numLiveReplicas = num.liveReplicas(); | int numCurrentReplica = numLiveReplicas | + pendingReplications.getNumReplicas(block); which means all the live replicas and pending replications will be counted. But in the end of FSNamesystem.blockReceived(), which calls the addStoredBlock(), it will call addStoredBlock() first, then reduce the pendingReplications count. | // | // Modify the blocks->datanode map and node's map. | // | addStoredBlock(blo Wang Xu 2008-11-28T06:09:15 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50319 Minor enhancements to some sections of the Map/Reduce tutorial -------------------------------------------------------------- Key: HADOOP-4739 URL: https://issues.apache.org/jira/browse/HADOOP-4739 Project: Hadoop Core Issue Type: Improvement Affects Versions: 0.19.0 Reporter: Vivek Ratan Assignee: Vivek Ratan Priority: Trivial I found some grammatical/spelling mistakes in the following sections of the Hadoop Map/Reduce tutorial: 'Submitting Jobs to a Queue', 'Debugging', and 'Skipping Bad Records'. The writeups for these sections can also be enhanced a bit. Vivek Ratan (JIRA 2008-11-28T05:14:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50316 [ https://issues.apache.org/jira/browse/HADOOP-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651449#action_12651449 ] David Yu commented on HADOOP-1650: ---------------------------------- In your patch(converting jetty5 to 6), I noticed that SelectChannelConnector.setUseDirectBuffers(boolean) is set to false. Any reason behind it? Generally its better to use direct buffers as they are cheaper to send. -dyu David Yu (JIRA 2008-11-28T01:56:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50309 See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/672/changes Apache Hudson Server 2008-11-27T16:59:34 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50290 saveVersion.sh could write a package-info.java that cannot be compiled when used with git ----------------------------------------------------------------------------------------- Key: HADOOP-4738 URL: https://issues.apache.org/jira/browse/HADOOP-4738 Project: Hadoop Core Issue Type: Bug Components: build Reporter: Hemanth Yamijala When git is being used, the saveVersion.sh script generates a revision based on git log as follows: {{revision=`git log -1 --pretty=oneline`}} This revision is used to generate the HadoopVersionAnnotation in the package-info file. If the revision string contains quotes this will result in a compile time error. Hemanth Yamijala (JIRA 2008-11-27T09:57:47 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50286 [ https://issues.apache.org/jira/browse/HADOOP-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651303#action_12651303 ] Sharad Agarwal commented on HADOOP-1230: ---------------------------------------- Is there a need to explicitly pass Context object in map and cleanup methods? These are expected to be called after setup. Would it be better to set Context in a protected member variable in setup? map and cleanup are expected to use the same reference of Context, right? I think this will simplify the code as we would require to pass Context to only 2 methods out of 4. So may not even require to create a shorthand Mapper#Context Going forward also if we add more methods, it will keep the method signatures simple. Sharad Agarwal (JIRA 2008-11-27T09:51:45 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50278 Job end notification should give status as "KILLED" when job gets killed. ------------------------------------------------------------------------- Key: HADOOP-4737 URL: https://issues.apache.org/jira/browse/HADOOP-4737 Project: Hadoop Core Issue Type: Bug Components: mapred Affects Versions: 0.15.2 Reporter: Ruchir Shah Currently in Job End notification, there are only 2 statuses : SUCCEEDED or FAILED. But as we now have job end status as Killed also, job end notification should give status as "KILLED" when job gets killed. Ruchir Shah (JIRA 2008-11-27T08:07:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50277 Hi, all Korea Hadoop Community hosts half-day Hadoop Tutorial Workshop on November 28(Friday) in Seoul, South Korea. You can check and register the workshop in our website. http://www.hadoop.or.kr/?document_srl=1945 Time: Friday, November 28, 14:00 ~ 18:00 Location: Seoul National University School of Dentistry main building 121 Free and open event (but limited by 100 persons) Agenda - Hadoop Overview - Hadoop Installation & Management - Managing a Hadoop Cluster - MapReduce Programming - Advanced MapReduce Programming Look forward to seeing you there! Jason Jaesun Han 2008-11-27T06:54:47 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50260 Add a link to Hive under "Related Projects" on http://hadoop.apache.org/core/ ----------------------------------------------------------------------------- Key: HADOOP-4736 URL: https://issues.apache.org/jira/browse/HADOOP-4736 Project: Hadoop Core Issue Type: New Feature Components: documentation Reporter: Jeff Hammerbacher Priority: Minor We should link to Hive from http://hadoop.apache.org/core/ and make a Hive home page with forrest (see https://issues.apache.org/jira/browse/HIVE-81) Jeff Hammerbacher (JIRA 2008-11-26T23:46:44 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50254 NameNode reporting 0 size for originally non-empty files -------------------------------------------------------- Key: HADOOP-4735 URL: https://issues.apache.org/jira/browse/HADOOP-4735 Project: Hadoop Core Issue Type: Bug Components: dfs Affects Versions: 0.17.2 Reporter: Christian Kunz NameNode reports 0 size for a handful of files that were non-empty originally. The corresponding blocks on the DataNodes are non-empty. NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully. Christian Kunz (JIRA 2008-11-26T22:36:46 http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/50249 Some lease recovery codes in 0.19 or trunk should also be committed in 0.18. ---------------------------------------------------------------------------- Key: HADOOP-4734 URL: https://issues.apache.org/jira/browse/HADOOP-4734 Project: Hadoop Core Issue Type: Bug Components: dfs Affects Versions: 0.18.2 Reporter: Tsz Wo (Nicholas), SZE Fix For: 0.18.3 In HADOOP-1700, there are codes for detecting corrupted blocks. In HADOOP-4257, there are bug fixes for lease recovery. Tsz Wo (Nicholas), SZE (JIRA 2008-11-26T21:55:44 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.jakarta.lucene.hadoop.devel