gmane.comp.python.bio.devel

UniProt GOA parser

Iddo Friedberg — 2013-05-08T23:28:17

A new uniprot-GOA parser is available for you to poke around:

https://github.com/idoerg/biopython/tree/uniprot-goa/Bio/UniProtGOA

More on Uniprot-GOA: http://www.ebi.ac.uk/GOA

There are three file formats: GAF (gene association file) , GPA (gene
product association) and GPI (gene product information) explained here:
http://www.ebi.ac.uk/GOA/downloads

Input GAF files can be very large, due to the growth of uniprot GOA. If you
would like to test in a timely fashion, I suggest you get historical files,
which are smaller. Once you get to the > 40 version numbers, the runtime
for the example code in UniProtGOA.py goes over 2 minutes (on my i5
machine).

Old GAF files are available here:
ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/old/UNIPROT/

Current GPI and GPA files are not very large.

Thanks to Peter for his help on this.

Best,

Iddo

Progress with ticket 3336

Nate Sutton — 2013-05-07T21:12:59

Hi,

Here is a progress follow up to http://lists.open-bio.org/pipermail/biopython-dev/2013-April/010548.html .  I have added a commit to the github branch that adds an option to create claude branch lines using linecollection.  The linecollection objects are stored in a tuple before adding them to the plot.  It’s in Bio/Phylo/_utils.py.  Is this what the last bullet point was requesting in https://redmine.open-bio.org/issues/3336 ?  

Thanks!

Nate

P. S.  I used a tuple to store the linecollection objects instead of a list because that was mentioned in the ticket but if that looks like it should be different let me know.  Also, I got some global variables to work with the code but I was only able to do that after declaring them as globals twice.  If there are suggestions on how to code that differently let me know.

_______________________________________________
Biopython-dev mailing list
Biopython-dev< at >lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/biopython-dev

uniprot-GOA parse

Iddo Friedberg — 2013-05-07T16:24:00

hi,

As promised, I have written a uniprot-goa parser. Very skeletal, has
iterators for reading the three uniprot-GOA file types, a write function,
and a couple of usage examples.

No github write access, so attaching.

Cheers,

Iddo

Biopython Phylo Proposal

Yanbo Ye — 2013-05-03T02:19:07

Hi everyone,

I forget to post my gsoc proposal page here. Any comment?
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/yeyanbo/1#

Thanks,

Yanbo

Uniprot-GOA parser

Iddo Friedberg — 2013-05-02T16:12:12

Does anybody have a GOA parser in the works? Currently writing a simple
parser for GAF, GPA and GPI formats. Can contribute if there is interest.

More on GOA: http://www.ebi.ac.uk/GOA

Cheers,

Iddo

gsoc phylo project questions

Yanbo Ye — 2013-04-30T07:20:59

Hi Eric,

Again, thanks for your comment. It might be better to discuss here.
https://github.com/lijax/gsoc/commit/e969c82a5a0aef45bba1277ce01d6dbee03e6a84#commitcomment-3096321

I have changed my proposal and timeline based on your advice. I think I was
too optimistic that I didn't consider about the compatibility with existing
code or other potential problem that may exist. After careful
consideration, I removed one task from the goal list to make the time more
relaxed, the tree
comparison<http://www.biopython.org/wiki/Phylo_cookbook#Comparing_trees>(seems
I miss understood this). I might be able to complete all of them. But it's
better to make it as an extra task, to make sure this coding experience is
not a burden.

According to your comment:

1. I didn't know PyCogent and DendroPy. I'll refer to them for useful
solutions.
2. For distance-based tree and consensus tree, I think there is no need to
use NumPy. And for consensus tree, my original plan is to implement a
binary class to count the clade with the same leaves for performance. As
you suggest, I'll implement a class with the same API and improve the
performance later, so that I can pay more attention to the Strict and Adam
Consensus algorithms.
3. I didn't find the distance matrix method for MSA on Phylo Cookbook page,
only from existing tree.
4. For parsimony tree search, I have already know how several heuristic
search algorithms work. Do I need to implement them all?
5. I'm not clear about the radial layout and Felsenstein's Equal Daylight
algorithm. Isn't this algorithm one way of showing the radial layout? I'm
sorry that I'm not familiar with this layout. Can you give some figure
examples and references?

Best,

Yanbo

[GSoC2013] Codon alignment and analysis in Biopython

Saket Choudhary — 2013-04-28T14:13:43

Hi All,

I am a Fourth Year Undergraduate studying Chemical Engineering at IIT
Bombay, India.

As a warmup, I have been working upon NGS analysis this sem. Besides
playing around with NGS pipelines, scripting one on my own, I have
lately been plying around with the PSSM approach for BWA .
https://github.com/pkerpedjiev/bwa-pssm/ . So far my contributions
to this project have been only at testing level, I havent really
contributed to the code as such, just reported some bugs.

I have been using BioPython for my Computational Biology Course that I
am doing this semester. I have also been using BIoPython for NGS
analysis, modifying fastq formats and all. I recently contributed to
the BWA wrapper for BioPython
(https://github.com/biopython/biopython/pull/167). I am also working
on the samtools wrapper.

I was a Google Summer of Code 2012 Student for
Connexions(http://cnx.org) and contributed towards SlideImporter
Module(Pyramid/Python): That allows the user to upload SlideShows to
SlideShare and Google Drive and makes a CNXML module out of it.
(https://github.com/oerpub/oerpub.rhaptoslabs.slideimporter). I am
still working for the same organisation and contribute regularly
(https://github.com/oerpub/oerpub.rhaptoslabs.swordpushweb/, Checkout
branches starting with 'saket')

I have deep interest in Biology and have done Molecular and Cell
Biology and the computational aspects of it. BioPython is like the
Best amalgamation for me : Bio + Python

I would like to participate in GSoC 2013 as a student . I would like
to contribute to " Codon Alignment" . Having worked on bwa-pssm
codebase for a while has made me acquainted with alignment algorithms
and I think I can contribute to this.

I have written the first draft of my proposal at
https://docs.google.com/document/d/1xlRSJ74POxi51xn7Web08ACZt4r7KSFIrzRAawTL2ds/edit#heading=h.j7p6z4a3ssy3

Being a first draft it may have a lot of ambiguity , say I would be
trying to implement a function that is already existing in Biopython
or I would have misinterpreted the problem completely. You can post
your comments on the doc itself and I will try to improve upon it.

Thanks

Saket
https://github.com/saketkc

[Biopython - Bug #3430] (New) Error parsingPubMedCentral XML files

2013-04-27T19:46:51

Issue #3430 has been reported by Paulo Nuin.

----------------------------------------
Bug #3430: Error parsing PubMedCentral XML files
https://redmine.open-bio.org/issues/3430

Author: Paulo Nuin
Status: New
Priority: Normal
Assignee: Biopython Dev Mailing List
Category: Main Distribution
Target version: 
URL: 


It seems that there is an error parsing locally downloaded PubMedCentral xml (extension nxml) files. Using the code 

< at >
from Bio import Entrez
handle = open('nihms83342.nxml')
records = Entrez.parse(handle)
for record in records:
    print record
< at >

the following error occurs (copied from iPython), even though the XML header contains the declaration


---------------------------------------------------------------------------
NotXMLError                               Traceback (most recent call last)
<ipython-input-5-e78d8d3c3888> in <module>()
      2 handle = open('nihms83342.nxml')
      3 records = Entrez.parse(handle)
----> 4 for record in records:
      5     print record

/Library/Python/2.7/site-packages/Bio/Entrez/Parser.pyc in parse(self, handle)
    229                         # We did not see the initial <!xml declaration, so
    230                         # probably the input data is not in XML format.
--> 231                         raise NotXMLError("XML declaration not found")
    232                 self.parser.Parse("", True)
    233                 self.parser = None

NotXMLError: Failed to parse the XML data (XML declaration not found). Please make sure that the input data are in XML format.

The XML file in question is attached.


----------------------------------------
You have received this notification because this email was added to the New Issue Alert plugin

Lazy-loading parsers,was: Biopython GSoC 2013 applications via NESCent

Peter Cock — 2013-04-27T11:20:57

Hi Zhigang,

It isn't too late to write up a proposal for GSoC 2013, but please
also introduce yourself on the NESCent Phyloinformatics
Summer of Code community on Google Plus:
https://plus.google.com/communities/105828320619238393015

The GSoC program is a great chance to spend a few months
focussed just on one programming project - which can be
really fun. However, the fact that you're interested in making
contributions outside of GSoC is great.

I wrote some more about the lazy-loading sequence parsers
and indexing idea on the biopython-dev mailing list last month:
http://lists.open-bio.org/pipermail/biopython-dev/2013-March/010469.html

However, lazy-parsing can also be done separately from the
indexing. This is something I was trying in my experimental
SAM/BAM parser mentioned on this thread:
http://lists.open-bio.org/pipermail/biopython-dev/2013-April/010492.html

The basic idea here was that the raw data for each record was
loaded into memory as a (bytes) string, but not all of it was
parsed into the individual fields right away. For example, the
tags get turned into a dictionary only if the user tried to use
the tag values. Similarly for many of the BAM fields, the binary
string was only decoded if needed.

I once tried something similar with the FASTQ parser. I wrote
a subclass to preserve the normal SeqRecord interface, but
only decode the ASCII encoded quality scores into a list of
integers if needed. This worked but that attempt did not seem
to make things any faster.

An example where I think there would be clear benefits to a
lazy parsing approach is EMBL/GenBank files where parsing
the features could be delayed (both the complex feature
location, and their dictionary of annotations).

However, for this to be a successful GSoC project, you
would need to have a good understanding of Python and
how our existing parsers work to have a realistic chance
of completing it. I should be quite a technically exciting
project, with the hope of being able to show big speedups
via benchmarks.

Does that help? Is there a particular file format you'd be
interested in - perhaps something you are already using
in your projects or work?

Regards,

Peter

Fw: GSoC Application

阮铮 — 2013-04-27T08:18:56

Sorry I forget to introduce myself.
My name is Zheng Ruan, currently a first year graduate students at the University of Georiga. I have a background in biology and now pursue a PhD in Bioinformatics. I taught myself many programming language and finally python became my favorite. I use biopython a lot for everyday sequence manipulation and format conversion. For example, I use Bio.Phylo class for reroot tree and visualization functionality in the sever http://bioinformatics.publichealth.uga.edu/SpeciesTreeAnalysis/index.php, which has just been accepted by NAR. I really hope to contribute the the Open Source Project and GSoC seems to be a good start point. I think the codon alignment proposal suitable to me because it requires medium programming skills and I've already finished the Computational Molecular Evolution by Zihen
 g Yang and also have a phylogenetics course this spring semester. Thanks for all.


Best,
Zheng Ruan


------------------ Original ------------------
From:  "Zheng Ruan"<rz1991< at >foxmail.com>;
Date:  Apr 27, 2013
To:  "biopython-dev"<biopython-dev< at >lists.open-bio.org>; 

Subject:  [Biopython-dev] GSoC Application



Hi Biopython developers,

I hope it's not too late to inform you my interest in Biopython GSoC, but I talked to Eric Talevich several weeks before that I would like to apply the Codon Alignment proposal. As a simple trial, I've already got a simple implementation of codon alignment script as can be found in https://github.com/zruan/CodonAlignment. It doesn't account for frame shift and mismatches between protein sequences and nucleotide sequences. But as least it works if the protein sequences can be exactly translated by nucleotide sequences. Hopefully I will finish my proposal this week. Is there any suggestions?


Sincerely,
Ruan

_______________________________________________
Biopython-dev mailing list
Biopython-dev< at >lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/biopython-dev

GSoC Application

阮铮 — 2013-04-27T02:12:11

Hi Biopython developers,

I hope it's not too late to inform you my interest in Biopython GSoC, but I talked to Eric Talevich several weeks before that I would like to apply the Codon Alignment proposal. As a simple trial, I've already got a simple implementation of codon alignment script as can be found in https://github.com/zruan/CodonAlignment. It doesn't account for frame shift and mismatches between protein sequences and nucleotide sequences. But as least it works if the protein sequences can be exactly translated by nucleotide sequences. Hopefully I will finish my proposal this week. Is there any suggestions?


Sincerely,
Ruan

_______________________________________________
Biopython-dev mailing list
Biopython-dev< at >lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/biopython-dev

Self Introduction

Yanbo Ye — 2013-04-27T03:16:22

Hi guys,

My name is Yanbo Ye. I'm a third-year master student in
bioinformatics(mainly phylogenetics) in China. My master project is about
the genome evolution of baculovirus and a bioinformatics tool development.
The tool is a java program named
BlastGraph<https://github.com/bigwiv/BlastGraph>that can be used for
sequence clustering, genome tree construction and gene
gain and loss estimation. This involves a range of phylogenetics
algorithms commonly used.

So based on my experice, I'd like to contribute to the Phylo package of
Biopython for some unimplemented methods during the Google Summer of Code
this year, which will be under the direction of Eric Talevich.

I have been using python and Biopython for nearly a year to do some
sequence file parsing and tree manipulation. But I must say I'm still new
to python about the design convention or any popular python packages, etc.
Anyway, I'm a fast learner and would like to improve my python programming
skills through this project. I'd like to discuss with you during this
project and hope you could help me.

Thanks,

Yanbo

Biopython GSoC 2013 applications via NESCent

Peter Cock — 2013-04-24T19:19:48

To all the Biopythoneers,

For the last few years Biopython has participated in the
Google Summer of Code (GSoC) program under the umbrella
of the Open Bioinformatics Foundation (OBF):
https://developers.google.com/open-source/soc/
https://github.com/OBF/GSoC

Unfortunately like quite a few previously accepted organisations,
this year the OBF not accepted. Google has kept the total about
the same year on year, so this is probably simply a slot rotation
to get some new organisations involved.

The good news (for those not following the Biopython-dev
mailing list) is we have an alternative option agreed with
the good people at NESCent, as we did back in 2009:

http://biopython.org/wiki/Google_Summer_of_Code
http://informatics.nescent.org/wiki/Phyloinformatics_Summer_of_Code_2013

I'd like to thank Eric for co-ordinating this, and encourage
any interested potential students to sign up to the Biopython
development list and NESCent's Google+ group as soon as
possible (if you haven't done so already):

http://lists.open-bio.org/mailman/listinfo/biopython-dev
https://plus.google.com/communities/105828320619238393015

Google are already accepting student applications, and the
deadline is Friday 3 May.  That doesn't leave very long for
asking feedback and talking to potential mentors - which
is essential for a competitive proposal.

Thank you for your interest,

Peter

Bug: Bio.PDB.DSSP fails on PDB files with no header

Jan Kosinski — 2013-04-23T08:09:30

The following script:

import sys
import Bio
from Bio.PDB.DSSP import DSSP
from Bio.PDB.PDBParser import PDBParser

pdb_filename = sys.argv[1]
p = PDBParser()
structure = p.get_structure('chupacabra', pdb_filename)
model = structure[0]

biodssp = DSSP(model, pdb_filename, dssp="dssp")

will fail on files without HEADER with traceback:
Traceback (most recent call last):
  File "testdssp_fail.py", line 13, in <module>
    biodssp = DSSP(model, pdb_filename, dssp="dssp")
  File
"/home/modorama_dev/modorama/ENV_INI/lib/python2.7/site-packages/Bio/PDB/DSSP.py",
line 200, in __init__
    dssp_dict, dssp_keys = dssp_dict_from_pdb_file(pdb_file, dssp)
  File
"/home/modorama_dev/modorama/ENV_INI/lib/python2.7/site-packages/Bio/PDB/DSSP.py",
line 101, in dssp_dict_from_pdb_file
    out_dict, keys = make_dssp_dict(out_file.name)
  File
"/home/modorama_dev/modorama/ENV_INI/lib/python2.7/site-packages/Bio/PDB/DSSP.py",
line 121, in make_dssp_dict
    if sl[1] == "RESIDUE":
IndexError: list index out of range

This is because in newer dssp versions the header of DSSP output will have
empty lines if the PDB file had no HEADER and this part of make_dssp_dict
function will fail
        for l in handle.readlines():
            sl = l.split()
            if sl[1] == "RESIDUE:

Changing it to sth like:
if len(sl) > 1 and sl[1] == "RESIDUE":
fixes the problem.

Cheers,
Jan

Parsing fastq files with SeqIO.parser(handle)

Alex Leach — 2013-04-19T11:29:28

Dear BioPython Devs,

Probably a strange request, but I was wondering if it might be a good idea
to make the fasta parser raise an error when it is asked to parse
incorrectly formatted files.

I ask, because a while ago, I made a simple command line utility to
convert sequence files to/from various formats, using SeqIO.parser. It's
attached if anyone's interested.

My supervisor's now using it to filter fastq formatted sequences by
length, but keeps forgetting to add a '-format fastq' option. The script
by default assumes fasta formatted sequences, which, like SeqIO.parser is
by design, but the problem is that the parser doesn't mind at all when a
fastq file doesn't contain a single ">" character.

Are there any interfaces to make the fasta parser stricter? This error is
completely silent until picked up by external programs; hmmer, in this
instance. Ideally, an error would be raised much earlier in the process,
especially as the department's NFS servers take ages to retrieve and
convert an IonTorrent dataset. (I've got him using /var/tmp for the
converted files, but he keeps the original fastq's in an NFS home folder,
which is sloooooow).

The department's using BioPython 1.57 btw.

Thanks for your time.
Kind regards,
Alex

p.s. Don't suppose there's any plans to implement any parsers as
C-extensions?

---
Alex Leach. BSc, MRes
Chong & Redeker Labs
Department of Biology
University of York
YO10 5DD
Tel: 07940 480 771
EMAIL DISCLAIMER: http://www.york.ac.uk/docs/disclaimer/email.htm_______________________________________________
Biopython-dev mailing list
Biopython-dev< at >lists.open-bio.org
http://lists.open-bio.org/mailman/listinfo/biopython-dev

[Biopython - Feature #3427] (New) Cache paths inBio.Phylo trees for later use

2013-04-15T14:29:45

Issue #3427 has been reported by Ben Morris.

----------------------------------------
Feature #3427: Cache paths in Bio.Phylo trees for later use
https://redmine.open-bio.org/issues/3427

Author: Ben Morris
Status: New
Priority: Normal
Assignee: 
Category: 
Target version: 
URL: 


I'm doing some analyses using Bio.Phylo in which I need to find many distances between pairs of taxa, and have found that this can be quite slow as currently implemented.

My current solution is to extend Newick.Tree objects and cache the result of the get_path function. This way, after finding the distance between species A and B, finding the distance between A and C doesn't require recomputing the path from A to the root. Example:

<pre>
class CachingTree(bp.Newick.Tree):
    _paths = {}
    def __init__(self, tree):
        self.__dict__.update(tree.__dict__)

    def get_path(self, target, **kwargs):
        if not target in self._paths:
            self._paths[target] = bp.Newick.Tree.get_path(self, target=target,
                                                          **kwargs)
        return self._paths[target]

</pre>


Should this functionality be incorporated into the BaseTree class itself?


----------------------------------------
You have received this notification because this email was added to the New Issue Alert plugin

Level of Python 3 support?

Peter Cock — 2013-04-14T20:10:02

Hi all,

We've have Python 3 coverage on the Travis continuous
integration tests & the nightly builtbot tests for quite some
time now.
http://travis-ci.org/biopython/biopython
http://testing.open-bio.org/biopython/tgrid

There have been a few Python 3 specific issues reported
(which have been fixed), but I think we should probably
go ahead and say we support Python 3.1 or later (with
the exception of some minor C code which has not been
ported).

Does this seem like a good move for the next release?

We've already said that this (Biopython 1.62) will be the
final release to support Python 2.5, which will help make
some bits of the 2/3 compatibility code much simpler.

Regards,

Peter

samtools threaded daemon

Chris Mitchell — 2013-04-11T02:57:41

Hi everyone,

I've been doing a ton of mpileup work recently with samtools so I made a
python daemon to parallelize the process.  Is there any interest in a
generic SamTools package for BioPython?  I know pysam exists, but it'd be
an added dependency as well as not threaded.  In my experience, for
querying a ton of positions threading mpileup is the best way to go (much
faster than -l bed_file in my use cases).  If there's interest, I'll
package it as a general SamTools command line wrapper with the added
bonuses that for certain operations you can input a list and thread those
parts.

Chris

Intro

Andrei Rozanski — 2013-04-10T16:16:02

Hi all,

As a phd student in bioinformatics, I found in Biopython a useful tool.
Actually, I work on NGS, differential gene expression, alternative splicing
and conservationa analysis of transposable elements.
Therefore, would like to introduce myself and say that I would like to
contribute.

Best regards,

Debugging the warning filters in the unit tests

Peter Cock — 2013-04-10T11:17:34

Hi all,

I've suspected for a some time that there is some subtle bug
in run_tests.py or a limitation in the Python warnings module -
although our code tries to reset the filters between tests this
doesn't seem to be working all the time.

The obvious effect of this is getting loads of ResourceWarning
messages under Python 3.3, e.g.
http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.3/builds/130/steps/shell/logs/stdio

Sometimes theses warnings can manifest as real bugs, in
particular on Windows where you cannot delete a file when
there is still an open handle to it which will not be closed
until the garbage collection runs. i.e. Bugs which tend to
only show under Windows with PyPy or Jython. e.g.
https://github.com/biopython/biopython/commit/943fffe2835dca4a996a6a171f026f6374ecaaa9

After fixing most of the handle leaks (which is a good thing
to do anyway), the remainder of being shown are in the
documentation tests (where fixing them can make the
examples harder to follow):
http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.3/builds/140/steps/shell/logs/stdio

I'm hoping someone might have some insight into the
current (not entirely successful) attempts to handle
warning filters in run_tests.py

Note this is complicated because many of the
individual test_XXX.py files manipulate the (global)
warnings filters, either to silence harmless expected
warnings, or to verify that desired warnings appear.

Thanks,

Peter

P.S.

One of the first things I want to do once we drop support
for Python 2.5 (currently we're planning just one more
release support it) is update the doctests to use context
managers for file handles (i.e. use the with statement).
Currently we can't easily do this in plain doctests without
including this line explicitly:

from __future__ import with_statement

We could do that automatically in test_Tutorial.py though
as part of the code which extracts doctest examples form
the LaTeX and runs them.

Abstract for "Biopython Project Update" at BOSC 2013

Peter Cock — 2013-04-09T13:36:27

Hi all,

First a general reminder that the BOSC 2013 abstract deadline is
this coming Friday 12 April, http://www.open-bio.org/wiki/BOSC_2013

Second, we need to prepare an abstract for the traditional Biopython
Project update. Both Brad and I should be there for BOSC 2013,
and the preceding mini hackathon/codefest to which everyone is
welcome: http://www.open-bio.org/wiki/Codefest_2013

Who else is planning to be at BOSC 2013 (and the Codefest), and
is there anyone keen to present the update? I'm happy to give the
talk, but this is a nice chance for someone else to present and give
a more rounded impression of the Biopython developers ;)

According to http://biopython.org/wiki/Documentation#Presentations
the recent Biopython presenters at BOSC have been:

BOSC 2013, Berlin, Germany - ???
BOSC 2012, Long beach, USA - Eric
BOSC 2011, Vienna, Austria - Peter
BOSC 2010, Boston, USA - Brad
BOSC 2009, Stockholm, Sweden - Peter
BOSC 2008, Toronto, Canada - Tiago
BOSC 2007, Vienna, Austria - Peter
BOSC 2006, Fortaleza, Brazil - No one
...

So, any volunteers from our development team? I know from
personal experience that giving a talk can be very useful for
securing travel funding - and we can if need be ask the BOSC
committee to waive the registration fee (in a new initiative for this
year at BOSC).

Thanks,

Peter
(Disclaimer: I'm also on the BOSC committee)