gmane.science.biology.biodas

ensembl das sources

Jonathan Warren — 2012-04-12T11:50:51

http://das.ensembl.org/das/sources now contains a non redundant list of ensembl das sources that points to archived ensembl instances using the new sources document specification and conforms to the 1.6 specification way of specifying data source uris http://www.biodas.org/documents/spec-1.6.html#uris. This document is hosted by the Sanger but not maintained by the ensembl project itself. The DAS data sources it points to are hosted by the ensembl project however.

Any issues/suggestions then let me know.

Cheers

Jonathan.




Jonathan Warren
Senior Developer and DAS coordinator
blog: http://biodasman.wordpress.com/
jw12< at >sanger.ac.uk

DAS workshop outcomes 2012

Jonathan Warren — 2012-03-13T13:34:34

Thanks to all those who attended the DAS workshop this year it was a great year in terms of developer outcomes and we hope the tutorials on day 1 were useful for new users.

There is a section on the workshop page with outcomes of the workshop http://www.biodas.org/wiki/DASWorkshop2012#Outcomes_from_workshop_2012 please feel free to add further links and information there.
Thanks to David for all the great pictures available here http://www.flickr.com/photos/davidmam/sets/72157629110016398/ looks to me like we were having a good time :)

I'd personally like to thank the other tutors many of whom put in extra hours outside of work to make this a successful workshop.

New users who attended the workshop my be interested to know that talks from last years workshop that illustrate how DAS is being used in the wider community are available as video links from here: http://www.biodas.org/wiki/DASWorkshop2011#Day_2

Thanks again

Jonathan.


Jonathan Warren
Senior Developer and DAS coordinator
blog: http://biodasman.wordpress.com/
jw12< at >sanger.ac.uk
Ext: 2314
Telephone: 01223 492314

Re: das developer day topics

Jim Procter — 2012-02-22T21:38:59

ahah - I didn't look in the dev branch.. great work!  (and I see lots of 
it was done last march :)

Jim.

Re: das developer day topics

Andy Jenkinson — 2012-02-22T21:00:33

On 22 Feb 2012, at 15:03, Jim Procter wrote:



OK, for posterity, the ProServer stuff is currently in the dev branch of subversion - there was an authentication framework before, but no HTTP basic/digest and no encryption.

Re: das developer day topics

Jim Procter — 2012-02-22T15:03:36

great :)  I've taken a look at what ProServer currently does, and I 
think that the demo that David Martin, myself and Thomas Down have 
planned will complement this.  After the long running thread from 
December about auth/das, David did some thinking about graceful service 
degradation when authentication fails, and client-neutral methods for 
advertising server authentication mechanisms. At least, that is the plan ;)

Jim.

Re: das developer day topics

Andy Jenkinson — 2012-02-22T11:32:08

Hi Jim,

If people are interested in knowing what has been done so far in proserver for auth/encryption I will run through it. In general, I guess these topics may be a bit more interactive than being talks per se, but I have updated the ones I know about.

Cheers,
Andy

On 22 Feb 2012, at 11:08, Jim Procter wrote:

das developer day topics

Jim Procter — 2012-02-22T11:08:53

Hi Jonathan and everyone.

I noticed that there are several topics listed on the developer day 
under 'developments since last meeting' for which there is no proposer 
(authentication/encryption in proserver is the one that caught my eye).

Who is going to talk about those topics ?

Jim.

Only 10 days left to register for DAS Workshop 2012

Jonathan Warren — 2012-02-02T09:44:52

Only 10 days left to register for DAS Workshop 2012

DAS is currently being used to share annotations on genomes, protein
alignments, structural and interaction information.

If you are interested in sharing biological information the DAS workshop
below may be of interest to you.

Learn of and contribute to current developments in DAS such as: DAS in  
the cloud, DAS for Genotype Data, DAS searching, DAS for collaborative  
annotation projects, DAS alternative formats.

Registration is open for the 2012 DAS workshop (27-29 February) at the
Genome Campus, Hinxton UK. If you are interested in attending, please
find out more by going to http://www.ebi.ac.uk/training/onsite/120227_DAS.html 
  and
register via the web link at the bottom of the page. This workshop will
cater for novice to expert DAS users as each day is optional.

Please register early as places will be limited. Registration closes  
10 February 2012 - 12:00.

If you are interested in giving a 15 minute talk on the second day  
please email Jonathan Warren using jonathan.warren< at >sanger.ac.uk

Many thanks

The Sanger/EBI DAS team.

Jonathan Warren
Senior Developer and DAS coordinator
blog: http://biodasman.wordpress.com/
jw12< at >sanger.ac.uk
Ext: 2314
Telephone: 01223 492314

DAS Workshop Registrations

Jonathan Warren — 2012-01-20T10:41:18

Hi

If you are intending to come to the DAS workshop this year can you  
register soon as we would like to get an idea of numbers for the 3  
respective days (1st day tutorials and short talks, 2nd and 3rd day  
developers hackathon). This will then enable us to start firming up  
the schedule for the 3 days.
If you are not intending to come to the workshop can I ask for some  
feedback that may help us entice you in the future?

Many thanks

Jonathan.

See workshop email for you convenience below:

DAS is currently being used to share annotations on genomes, protein
alignments, structural and interaction information.

If you are interested in sharing biological information the DAS workshop
below may be of interest to you.

Registration is open for the 2012 DAS workshop (27-29 February) at the
Genome Campus, Hinxton UK. If you are interested in attending, please
find out more by going to http://www.ebi.ac.uk/training/onsite/120227_DAS.html 
  and
register via the web link at the bottom of the page. This workshop will
cater for novice to expert DAS users as each day is optional.

Please register early as places will be limited. Registration closes  
10 February 2012 - 12:00.

If you are interested in giving a 15 minute talk on the second day  
please email Jonathan Warren using jonathan.warren< at >sanger.ac.uk

Many thanks

The Sanger/EBI DAS team.



Jonathan Warren
Senior Developer and DAS coordinator
blog: http://biodasman.wordpress.com/
jw12< at >sanger.ac.uk
Ext: 2314
Telephone: 01223 492314

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Jonathan Warren — 2012-01-17T10:44:40

Hi

I'd be against using OAuth because it will probably suffer from the  
same limitations that OpenId suffered from. 1) It will be the in thing  
for a while with support for certain programming languages or  
environments (such as browsers) but will soon go out of fashion when  
something else with a cool sounding name comes along :)
2) It takes us away even more than we already are from the principle  
that the DAS system should be simple.
3) If it's complicated it can be difficult to change your application  
if the environment that you are running your application in changes or  
other factors make you change it (e.g. when the sanger web site no  
longer supported sticky sessions I had to try and rewrite the openID  
code for the registry which turned out to be dependent on normal in  
memory Java sessions, so I had to change code that I didn't really  
understand, thus introducing security holes, so I ended up turning to  
good old username and passwords).
4) Everybody understands username and passwords boxes in an user  
interface whereas if we start going on about OAuth authentication and  
using their facebook accounts people will get frightened off - I'm  
also not sure I won't my gmail or facebook accounts to log me in to  
the DAS registry automatically and start distributing my contact  
lists ;)

My preference and I think most other peoples when we have been on this  
discussion before (3rd time around??) is to use basic https security  
that has been around for years. I also prefer the option of sending  
the username and passwords for every request for a secure data source  
as the server then remains in one state (the registry sends them in  
the headers, MyDAS currently sends them in the xml  - we need to  
decide which to use). This would be in preference to an amazon S3 type  
system where tokens are given for a limited period of time to the  
client/user (the S3 type system would be my second option). These  
would be a good balance between ease of development/understanding and  
security. If specific clients and servers wanted to add extra security  
they could add IP restrictions so they know the client is hosted at a  
specific place.

Cheers

Jonathan.


On 16 Jan 2012, at 23:16, Dan Bolser wrote:


Jonathan Warren
Senior Developer and DAS coordinator
blog: http://biodasman.wordpress.com/
jw12< at >sanger.ac.uk
Ext: 2314
Telephone: 01223 492314

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Dan Bolser — 2012-01-16T23:16:57

Heh... it doesn't work:

https://github.com/dbolser/TwitterBot---nowlistening-Perl-script-for-xmms

Not sure what I'm doing wrong.


On 16 January 2012 21:12, Dan Bolser <dan.bolser< at >gmail.com> wrote:

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Dan Bolser — 2012-01-16T21:12:52

<snip (too mental for me ;-)>


Right, the person who 'owns' the data (i.e. a list of contacts hosted
on a Google account) explicitly grants the third party 'app'
permission to access the data (via the account) in a specified way (as
defined by the Google APIs). That app can then email all your contacts
in the middle of the night while you're sleeping, but you trust that
that won't happen when you click the 'grant' button.

i.e. I (the verified me) can grant Ensembl permission to access my SNP
genotype data from 23andMe (hah), and I trust Ensemble not to do
anything nasty with that data when I log off.

Although it's a bit of a pain to set this up, the point is that
literally thousands of app developers (including me) have done it
before, and there are hundreds of docs. Here is where I started when I
built a command line twitter bot:
https://dev.twitter.com/docs/auth


I'm not trying to say its quick and easy to do and everyone should do
it like this, I just thought I'd provide the above encapsulation,
which hopefully isn't too far from how it could be done.


Cheers,
Dan.

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Andy Jenkinson — 2012-01-16T18:34:51

I agree with this. Security through obscurity should never be presented as "private". We should continue to implement some form of authentication but we should keep it simple to start with.

On 16 Jan 2012, at 17:20, Bernat Gel wrote:

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Andy Jenkinson — 2012-01-16T18:31:52

I rather suspect this is a purely mental exercise, but that's fine for me ;)

On 16 Jan 2012, at 16:44, Jim Procter wrote:

It is similar, and the solutions we are discussing are similar. As I say, there are solutions but they require some coordination.

CORS is what allows Dalliance to avoid having to communicate with DAS servers using a server in the middle (as Ensembl does). I'm saying that if a server is required to do the authentication bit (as I suspect it would be), I hope there is no technical block to having the user's computer send the signed token to the DAS server (i.e. via CORS) rather than the "Dalliance authenticator proxy". It is a slightly different model so I can't say it would definitely work.

CORS really isn't an issue - the server either allows CORS or it doesn't (in which case it is not supported by Dalliance) - it does not interact with the security side of things in any problematic way. You can use HTTP authentication in combination with CORS just fine, but we are discussing OpenID + OAuth. I am assuming that CORS is enabled on the DAS server because if it is not then Dalliance will not work. What I am talking about is the fact that Dalliance is a JS application. That is, all its code is completely editable by anyone who cares to open the Developer console, Firebug etc. There is no advantage to using OpenID+OAuth over HTTP authentication (which is what I have implemented in ProServer already so it works with Dalliance as-is) if the user can just disable the bit of the code
that authenticates you. Thus we would have to ensure that the Dalliance JS only gets access to an OAuth token if an externally verifiable set of conditions is met. This is what, for me, mea!
ns that there must be a server component rather than it all being done in JS.

I think we're talking crossed purposes. CORS is not a way of securing servers, it's just a way of lifting the restriction placed on Javascript by browsers that it can't request URLs outside the domain of the webpage the Javascript came from. An authenticated cross-origin request (in CORS language) simply means a regular HTTP digest or HTTP basic authenticated request that happens to be setting the headers that are needed to get around this restriction. The potential for misuse is not from misusing the CORS request (indeed interfering with CORS is not possible by fiddling with the javascript). A cross-origin model is completely standard for DAS, indeed it is the whole point of having a distributed system. Nor is the misuse I am worried about to do with faking an authentication. It is simply
making a trivial change in a trusted client to manipulate it into accessing secured data by bypassing the need to do the authentication step at all.

OAuth is entirely based upon the notion that the server with the data (e.g. Google Contacts) can trust the application (e.g. the Android Contacts app) to do the right thing with the data. There is no requirement that the person who owns the data, or any other person, has to be present, and the application doesn't have to prove that this will happen. It just has to get the user to agree that the application can be trusted. It's up to us therefore to provide a secure link between OpenID and OAuth.

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Bernat Gel — 2012-01-16T17:20:55

Hi all,

I agree with Manuel (and I think Andy) that without a real demand for
private DAS sources implementing some complex authentication scheme might
be an overkill. However, I think that a key thing with security is that if
it's done, it has to be weel done. Somehow, a false sense of privacy (such
as the cryptic URLs in easyDAS) might be a bad thing if the user is not
aware of its weaknesses and end up unkwowingly sharing data deemed to
remain private.

So, unless a DAS sourceis really private and secure, it should be made
clear that it is public (although maybe difficult to find).

Bernat

On Mon, Jan 16, 2012 at 18:10, Manuel Corpas <mc< at >manuelcorpas.com> wrote:

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Manuel Corpas — 2012-01-16T17:10:29

Hi guys,

why not just trying to do something *really* easy that could be
implemented in a couple of weeks rather than trying to engineer the
perfect solution all at once? We don't really know how much demand
there is for authenticated DAS sources, so we may end up spending a
lot of effort for something that might be underused.

The truth is that currently all DAS sources are public. Some "private
sources" available are those provided by easyDAS with its long
"cryptic" urls. That alone I think is already something. Any
improvement on the ability to keep sources private, whatever small,
would be a step in the right direction.

My two cents.

Manuel

Manuel Corpas, PhD
Tel:      +44.122349.2372
Web:    http://manuelcorpas.com/about/
Twitter: < at >manuelcorpas



On 16 January 2012 16:44, Jim Procter <jprocter< at >compbio.dundee.ac.uk> wrote:

Re: Personal genomics/Deploying a DAS server for Dummies/6 Easy steps

Jim Procter — 2012-01-16T16:44:55

Hi Andy, and anyone else still reading :)

On 14/01/2012 01:03, Andy Jenkinson wrote:
Bother! am I feeding troll(s) again ? ;)
I'm not sure how different this is from the 3D security challenge from 
your bank/credit card company that opens in an iframe after you put your 
credit card details in to a secure shopping site and hit the confirm 
button. Authentication still has to happen in order for a transaction 
(in our case one involving a DAS request/response session) between the 
DAS server and the Ensembl server, before it renders the result in the 
page sent to your browser.

As for dalliance and secure servers:
it won't actually be happening that way - take a look at this: 
http://www.biodalliance.org/cors.html
Interesting angle. I'm not actually sure what happens if the page from a 
server allowed to contact another server via CORS has been edited after 
it arrived in the browser sandbox - but I don't think it is really that 
much of an issue either way.

There are two situations worth considering:
1. an honest user wants to access secure data. In which case, they must 
authenticate via the dalliance host page with servers providing secure 
data in order to grant dalliance access via an authenticated 
cross-origin request from a specified server on the trusted CORS list.
2. a black hat attempts to gain access to secure data. In which case 
they must fake an authentication. We then have to rely on the 
authentication framework detecting cracking attempts.

The case when a (probably) honest user with valid credentials misuses 
the cross-origin request by messing with the javascript on the page is 
no different to having a user in a secure intranet who brings a virus in 
via a physical device or over a VPN. Since DAS is rather limited in its 
scope for exploits, I don't think there will be a problem with someone 
fiddling with the javascript, assuming, of course, the servers hosting 
the authenticated sources actually secured their server properly and 
have un-DOSable data providers at the back end...
understood, and I agree, with the caveat that I'm not even sure how 
familiar we all are with HTTP(s) auth styles, and personally, would 
appreciate it if there was a practical session or tutorial on the ins 
and outs of authenticated DAS for those who need to do it. This could 
cover how different styles of authenticated DAS sources work with the 
existing clients (e.g. do I really have to authenticate against every 
secure das source every time a request is made?!). It could also involve 
a tutorial on setting up a secure DAS source using various schemes. For 
example, a tutorial on SSO systems such as Shibboleth2 as used by the UK 
access management federation could be useful if people needs to give 
specific UK academics access to a DAS source.

Of course - nothing in this email constitutes me volunteering to give 
such a tutorial ....
Jim.

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Andy Jenkinson — 2012-01-14T01:03:40

Not sure I follow. The problem with the above model isn't with Ensembl it's with the DAS servers. DAS servers can't verify the user that is sitting at his machine (i.e. use OpenID) because they don't communicate with his machine. They only communicate, and can thus verify, the client (i.e. using OAuth). It would be better if the DAS server had some way to check the user directly (the client would be like a proxy for the user) but it is not possible with OpenID under Ensembl's current architecture (or at least it wasn't when I was investigating). So that means you have to do one of the below options instead (including having only Ensembl do the authentication).

best way is actually to have the assignment of who can access the data determined at the DAS server (a list of OpenID identities) which clients can simply query for. That way you're still !
s!

Basically yes, I would have thought there'd have to be some server-side Dalliance script doing most of the work, with it hopefully being possible to pass the signed token to the browser so that the user's machine is still the one issuing the DAS request. However, I just saw this, which is a potential solution (for OAuth at least):
http://derek.io/blog/2010/how-to-secure-oauth-in-javascript/
I'm not sure it helps however, as we need to ensure that Dalliance will only issue an authorised request to the DAS server if it has proof that the user has been authenticated and is on the access control list. With JS being editable on the client side, that check cannot be in JS.

I have no objections to participating in principle, but I remain to convinced it is worth the effort (i.e. that HTTP auth is insufficient and that OAuth would be adopted).

Re: Personal genomics/Deploying a DAS server for Dummies/6 Easy steps

Jim Procter — 2012-01-13T13:04:43

Hi. I see the dreaded secure DAS session topic as risen its head again.

On 13/01/2012 00:42, Andy Jenkinson wrote:
But Ensembl *does* need to know who you are in order to request data
that you are allowed access to. In the OAuth model, you would have to
allow Ensembl to access privileged data from the third party DAS server,
and that would be achieved by the Ensembl browser presenting you with an
OpenID login and redirect to subsequent access control pages from the
third party server.
best way is actually to have the assignment of who can access the data determined at the DAS server (a list of OpenID identities) which clients can simply query for. That way you're still s!
et!
Yes. security isn't cheap it seems :)
So the problem here with OAuth is that pure browser clients need a
secure store for authenticating a user's access to a particular resource
? I don't think there is any way around that either, and I think the
onus to provide this is the hosting page of the client - i.e.
dalliance.org would need to be recognised as a peer on a secure DAS
OAuth network (using the servers own keys). Then, users wanting access
to secure data would log in to the DAS source independently via a secure
session on dalliance.org, which would then allow dalliance to browse the
data from the secure server. If someone wants to set up a Dalliance
browser in their own OAuth trust network, then they would need to have
their own Dalliance hosting server and make it known to the other peers
accordingly.
I'd be very much in favour of people who *need* to achieve this spending
some time during the developer days with an invited expert (with special
anti-trolling skills to counter folk like me) in a closed session, in
order to identify components that would enable both browser and
non-browser based clients to work with OAuth, and set up a trial OAuth
DAS source network for testing. There are libraries that support OAuth
(http://oauth.net/code/) both for providers and consumers, but DAS
client libraries will need extension to allow secure negotiation and
signed DAS HTTP interaction, and their APIs will need additional
parameters/methods to allow session management.

Jim.

Re: Personal genomics/Deploying a DAS server for Dummies/6Easy steps

Andy Jenkinson — 2012-01-13T00:42:50

Originally I wanted to use a combination of OpenID and OAuth as an end-to-end solution. However, OpenID is based around the expectation that you are authenticating with a website using a browser - the protocol uses HTTP redirects, and OpenID providers have to have some way of telling you are logged in - cookies, forms etc. Ideally in DAS, it is the DAS server that needs to check that you are who you say you are, not just the client. For a client like Ensembl, your browser simply never communicates with the DAS server so the DAS server can't get you to authenticate with the OpenID provider.

It would be possible instead to have a system whereby you configure the DAS server to authorise a piece of software via OAuth, and have the client take care of making sure only certain users can access that data source. You are putting the onus of deciding which applications to trust onto the owner of the data, which is not ideal for data like personal genomics (although certainly not worse than handing out passwords). Obviously this has major implications for every client that wants to support it, because it's a whole extra suite of functionality they all need to implement. There would need to be a way for the clients to know who gets to decide who can access the source, though it could be implemented via some DAS-specific way easily I imagine. I think if we were to do it, possibly the be
st way is actually to have the assignment of who can access the data determined at the DAS server (a list of OpenID identities) which clients can simply query for. That way you're still set!
ting the permissions on the client. You're still trusting the client to honour the list (which includes not caching it), but at least the clients don't need to maintain the access control list. However they still need to implement OpenID. That's a technical requirement but, in the case of Ensembl which already providers user login facilities, it has other consequences. Convincing people it's worth adopting DAS is one thing, but convincing them that they must also use OpenID for their login system is another. All this is why I say it's possible, but there is a significant investment to get it all up and running. I'm also not sure if pure browser-implemented clients like Dalliance can use this method, both OpenID and OAuth involve signing messages with the application's secret key, and it's
difficult to do that (and store these keys) without a server of some kind. They'd probably be forced into using one. Still, this is my preferred solution if everybody was on board and had !
the resources to do it.

Lastly, neither solution works for daemon-style clients (e.g. command-line analysis applications where the user is not present), again because they can't use OpenID. The catch-all solution is to use X.509 certificates (public/private key cryptography) but it is heavyweight and probably too complicated to provide a good user experience. Truth be told, it has proved difficult to discuss this topic amongst the community because it gets technically very complex.

Cheers,
Andy

DAS2GFF3?

Dan Bolser — 2012-01-12T18:09:48

Hi,

Given a set of features in DAS format, what is the easiest way to
generate GFF3 format? Can most DAS server implementation be persuaded
to serve GFF3 in response to DAS feature requests?


Cheers,
Dan.