gmane.comp.java.hadoop.pig.devel

Re: A major addition to Pig. Working with spatial data

Ahmed Eldawy — 2013-05-24T04:40:37

Hi all,
  Thanks for your help. I've started the project with a minimal
functionality as a start. It's currently hosted in github. It is licensed
under the Apache public license to make it easier to merge with Pig.
Currently it has only a very few functions. I implemented a function from
different types of functions (e.g., Aggregate and create). I'll keep adding
functions and any contributions to the project are welcome. As a beginning,
I need an ANT build file that runs the tests, compiles and generates a jar
file. I'm not familiar with ANT so any help in this is encouraged.
Here's the project home page
https://github.com/aseldawy/pigeon


If you have any comments or suggestion please contact me.


Best regards,
Ahmed Eldawy


On Mon, May 6, 2013 at 3:09 PM, Jonathan Coveney <jcoveney-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Subscription: PIG patch available

2013-05-24T01:00:59

Issue Subscription
Filter: PIG patch available (19 issues)

Subscriber: pigdaily
        
Key         Summary
PIG-3335    TestErrorHandling.tesNegative7 fails on MR2
            https://issues.apache.org/jira/browse/PIG-3335
PIG-3334    Fix Windows piggybank unit test failures
            https://issues.apache.org/jira/browse/PIG-3334
PIG-3333    Fix remaining Windows core unit test failures
            https://issues.apache.org/jira/browse/PIG-3333
PIG-3328    DataBags created with an initial list of tuples don't get registered as spillable
            https://issues.apache.org/jira/browse/PIG-3328
PIG-3318    AVRO: 'default value' not honored when merging schemas on load with AvroStorage
            https://issues.apache.org/jira/browse/PIG-3318
PIG-3295    Casting from bytearray failing after Union (even when each field is from a single Loader)
            https://issues.apache.org/jira/browse/PIG-3295
PIG-3285    Jobs using HBaseStorage fail to ship dependency jars
            https://issues.apache.org/j

[Commented] (PIG-3331) Default values not stored in avro file when using specific schemas during store in AvroStorage

Viraj Bhat (JIRA — 2013-05-23T21:43:21

    [ https://issues.apache.org/jira/browse/PIG-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665713#comment-13665713 ] 

Viraj Bhat commented on PIG-3331:
---------------------------------

Patch posted on the review board.
https://reviews.apache.org/r/11355/
Viraj
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Updated] (PIG-3331) Default values not stored in avro file when using specific schemas during store in AvroStorage

Viraj Bhat (JIRA — 2013-05-23T21:29:25

     [ https://issues.apache.org/jira/browse/PIG-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viraj Bhat updated PIG-3331:
----------------------------

    Attachment: numbers.txt

Input text file with numbers
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Updated] (PIG-3331) Default values not stored in avro file when using specific schemas during store in AvroStorage

Viraj Bhat (JIRA — 2013-05-23T21:29:24

     [ https://issues.apache.org/jira/browse/PIG-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viraj Bhat updated PIG-3331:
----------------------------

    Attachment: expected_DefaultSchemaWrite.avro

ExpectedAvro file with Default Schema
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Mark Wagner (JIRA — 2013-05-23T20:41:21

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665637#comment-13665637 ] 

Mark Wagner commented on PIG-3329:
----------------------------------

Ah, my load was broken. The workaround seems right in that case. Thanks, Allan
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Allan Avendaño (JIRA — 2013-05-23T20:23:20

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665611#comment-13665611 ] 

Allan Avendaño commented on PIG-3329:
-------------------------------------

Hi Mark,

Could you post your script?

I guess, the output of FILTER is empty. 

                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Assigned] (PIG-3329) RANK operator failed when working with SPLIT

Mark Wagner (JIRA — 2013-05-23T19:37:20

     [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Wagner reassigned PIG-3329:
--------------------------------

    Assignee: Allan Avendaño
    

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Mark Wagner (JIRA — 2013-05-23T19:35:22

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665532#comment-13665532 ] 

Mark Wagner commented on PIG-3329:
----------------------------------

Allan/Johnny: I took a look at this and got an exception using a filter also:
{noformat}
Caused by: java.lang.RuntimeException: Error to read counters into Rank operation counterSize 0
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.saveCounters(JobControlCompiler.java:386)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.updateMROpPlan(JobControlCompiler.java:332)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:380)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
        ... 11 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.mapred.Counters.getShortName(Counters.jav

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Allan Avendaño (JIRA — 2013-05-23T18:45:31

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665478#comment-13665478 ] 

Allan Avendaño commented on PIG-3329:
-------------------------------------

Hi to everybody,

I'm working on this, Redis. In the meanwhile, a workaround is through a FILTER BY and then a RANK operator. 

In fact, Johnny, the RANK BY provides a ranking by considering the values of column a. Instead, RANK assigns a sequential number to each tuple. 
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3310) ImplicitSplitInserter does not generate new uids for nested schema fields, leading to miscomputations

Koji Noguchi (JIRA — 2013-05-23T17:53:27

    [ https://issues.apache.org/jira/browse/PIG-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665415#comment-13665415 ] 

Koji Noguchi commented on PIG-3310:
-----------------------------------

I also don't have a good understanding on these, but the change looks reasonable to me.  [~daijy], original uid reassignment was added in PIG-1705 for the self-join.  Can you take a look?
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Johnny Zhang (JIRA — 2013-05-23T17:45:20

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665403#comment-13665403 ] 

Johnny Zhang commented on PIG-3329:
-----------------------------------

The issue doesn't happen when using RANK BY, so another workaround is using "g = RANK e BY a;". 
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Johnny Zhang (JIRA — 2013-05-23T17:43:23

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665400#comment-13665400 ] 

Johnny Zhang commented on PIG-3329:
-----------------------------------

Redis, I actually can reproduce the issue with queries
{noformat}
d = load 'input1' as (a:int, b:int, c:int);
SPLIT d into e if a > 0, f if a < 0;
g = RANK e;
{noformat}

the error I got is from JobControlCompiler, it complains missing OperationID in MROperPlan. I guess the reason is when compile from physical plan to MROperPlan, the operators order is wrong, but not sure. I don't have a fix yet....

The workaround for now is using FILTER BY
{noformat}
d = load 'input1' as (a:int, b:int, c:int);
e = filter d by a > 0;
g = RANK e;
{noformat}

I post comment when I have new find. Thanks.
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: htt

[Commented] (PIG-3329) RANK operator failed when working with SPLIT

Redis Liu (JIRA — 2013-05-23T11:08:20

    [ https://issues.apache.org/jira/browse/PIG-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665071#comment-13665071 ] 

Redis Liu commented on PIG-3329:
--------------------------------

Anybody working on this?
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Subscription: PIG patch available

2013-05-23T01:01:59

Issue Subscription
Filter: PIG patch available (19 issues)

Subscriber: pigdaily
        
Key         Summary
PIG-3335    TestErrorHandling.tesNegative7 fails on MR2
            https://issues.apache.org/jira/browse/PIG-3335
PIG-3334    Fix Windows piggybank unit test failures
            https://issues.apache.org/jira/browse/PIG-3334
PIG-3333    Fix remaining Windows core unit test failures
            https://issues.apache.org/jira/browse/PIG-3333
PIG-3328    DataBags created with an initial list of tuples don't get registered as spillable
            https://issues.apache.org/jira/browse/PIG-3328
PIG-3318    AVRO: 'default value' not honored when merging schemas on load with AvroStorage
            https://issues.apache.org/jira/browse/PIG-3318
PIG-3295    Casting from bytearray failing after Union (even when each field is from a single Loader)
            https://issues.apache.org/jira/browse/PIG-3295
PIG-3285    Jobs using HBaseStorage fail to ship dependency jars
            https://issues.apache.org/j

Re: Review Request: PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

Viraj Bhat — 2013-05-23T00:12:14

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11135/
-----------------------------------------------------------

(Updated May 23, 2013, 12:12 a.m.)


Review request for pig.


Summary (updated)
-----------------

PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values


Description
-------

Default values are not honoured when merging default schema


This addresses bug PIG-3318.
    https://issues.apache.org/jira/browse/PIG-3318


Diffs
-----

  http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java 1484564 
  http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java 1484564 
  http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/

[Created] (PIG-3337) Fix remaining Window e2e tests

Daniel Dai (JIRA — 2013-05-22T22:10:20

Daniel Dai created PIG-3337:
-------------------------------

             Summary: Fix remaining Window e2e tests
                 Key: PIG-3337
                 URL: https://issues.apache.org/jira/browse/PIG-3337
             Project: Pig
          Issue Type: Sub-task
          Components: e2e harness
            Reporter: Daniel Dai
            Assignee: Daniel Dai
             Fix For: 0.12
         Attachments: PIG-3337-1.patch



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Updated] (PIG-3337) Fix remaining Window e2e tests

Daniel Dai (JIRA — 2013-05-22T22:10:21

     [ https://issues.apache.org/jira/browse/PIG-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-3337:
----------------------------

    Attachment: PIG-3337-1.patch
    


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Updated] (PIG-3322) AVRO: AvroStorage give NPE on reading file with union as top level schema

Viraj Bhat (JIRA — 2013-05-22T21:22:22

     [ https://issues.apache.org/jira/browse/PIG-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viraj Bhat updated PIG-3322:
----------------------------

    Fix Version/s: 0.12
    

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Commented] (PIG-3322) AVRO: AvroStorage give NPE on reading file with union as top level schema

Viraj Bhat (JIRA — 2013-05-22T21:22:21

    [ https://issues.apache.org/jira/browse/PIG-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664543#comment-13664543 ] 

Viraj Bhat commented on PIG-3322:
---------------------------------

Review board
https://reviews.apache.org/r/11333/
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Updated] (PIG-3322) AVRO: AvroStorage give NPE on reading file with union as top level schema

Viraj Bhat (JIRA — 2013-05-22T21:10:21

     [ https://issues.apache.org/jira/browse/PIG-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viraj Bhat updated PIG-3322:
----------------------------

    Attachment: expected_testLoadAvrowithNulls.txt

Expected File generated from the testcase
                

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira