Data Integration and Data Retrieval

(1)

Curso práctico de base de datos e integración de información biológica

Data Integration and Data Retrieval

Curso práctico de base de datos e

integración de informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(2)

Overview

• Data and services at the EBI

• Integration models

– EB-Eye

– SRS

– Biomart

• Integrative access

– Web services

• REST

• SOAP

(3)

Curso práctico de base de datos e integración de información biológica Genomes Nucleotides Proteins Structures Other molecules Interactions Experiments Literature Ontologies

EBI databases

(4)

(5)

2007/6/30 Curso práctico de base de datos e integración de información biológica 5

Challenges of Data Integration

• Different types of data (sequence, function, literature etc.)

• Different data formats (FASTA, EMBL, Genbank, tab

delimited etc.)

• Different storage formats (ASCII flatfile, XML, RDBMS)

• No standard formats for common fields (citations,

descriptions, dates etc.)

• Volume and size of data

(6)

(7)

Challenges when using tools in unison

• Manually transfer data from one application to another

• Understand disparate data formats

• Convert file formats where appropriate

• Manage and understand disparate application

(8)

EB-Eye

Curso práctico de base de datos e

integración de informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(9)

EB-Eye Approach to Data Integration

• Parse original data

• Reformat data if necessary

• Create searchable indices

• Extrapolate links between datasets

• Standardise field formats

(10)

Ligand

What is the data available ?

> 20 domains

>137M entries

(11)

Ligand

What is the data available – formats

<XML> . . . </XML> <XML> . . . </XML> <XML> . . . </XML> <XML> . . . </XML> <XML> . . . </XML> <XML> . . . </XML> <XML> . . . </XML> ID : .. PARENT ID : .. RANK : .. ... ID ... AC ... DT ... ID ... AC ... DT ...

(12)

What is the data available – sizes

81M 43M 4.2G

57Gb, >500 files

1G 8.4G

374Gb, >600 files

6.3G 25K

(13)

Points to take into consideration

• Our World

– A large amount of data

– A variety of file formats

– A variety of file sizes

– Data formats are changing

• Our Quest

– Index the data as fast as possible

– Add and configure a new domain easily

– Detect errors in the data

– Being up to date

– No downtime

(14)

Parsing and indexing different formats

Indexer

Lucene API

Db EMBL grammar Taxonomy grammar UniProt grammar . . . Parser (ANTXR) Medline grammar InterPro grammar Dump file grammar

. . . Parser (ANTLR) Uniprot Index Embl Index Taxonomy Index

ID AF030562; SV 1; linear; genomic DNA; STS; FUN; 852 BP. AC AF030562;

DT 04-DEC-1997 (Rel. 53, Created) DT 03-MAR-2000 (Rel. 62, Last updated, Version 2) XX

DE Fusarium venenatum clone VEN-A RAPD band generated using Operon primer

DE OPW-03, sequence tagged site. . . .

Flat files

<MedlineCitation Owner="NLM" Status="MEDLINE"> <PMID>10997935</PMID> <DateCreated> <Year>2000</Year> <Month>10</Month> <Day>04</Day> </DateCreated> … XML files <MedlineCitationSet>

<Title>Clinica chimica acta; international journal of clinical chemistry</Title> <ISOAbbreviation>Clin. Chim. Acta</ISOAbbreviation>

</Journal> . . . . . . ID Creation Date Modification Date issn volume name

ID AF030562; SV 1; linear; genomic DNA; STS; FUN; 852 BP. XX

AC AF030562; XX

DT 04-DEC-1997 (Rel. 53, Created)

DT 03-MAR-2000 (Rel. 62, Last updated, Version 2) XX

DE Fusarium venenatum clone VEN-A RAPD band generated using Operon primer DE OPW-03, sequence tagged site.

XX KW STS. XX

OS Fusarium venenatum

OC Eukaryota; Fungi; Ascomycota; Pezizomycotina; Sordariomycetes; OC Hypocreomycetidae; Hypocreales; mitosporic Hypocreales; Fusarium. XX

RN [1] RP 1-852

RA Yoder W.T., Christianson L.M.;

RT "Species-specific primers resolve members of the section Fusarium. RT Taxonomic status of the edible 'Quorn' fungus re-evaluated"; RL Fungal Genet. Biol. 0:0-0(1997).

XX RN [2] RP 1-852

RA Yoder W.T., Christianson L.M.; RT ;

RL Submitted (21-OCT-1997) to the EMBL/GenBank/DDBJ databases. RL Microbiology, Novo Nordisk Biotech, Inc., 1445 Drew Ave., Davis, CA 95616, RL USA XX FH Key Location/Qualifiers FH FT source 1..852 FT /organism="Fusarium venenatum" FT /strain="ATCC20334“ FT /db_xref="taxon:56646" . . . ID AC

Creation date / Modification date

Description Organism species Organism classes References References <database> <name>IntAct.Experiment</name>

<description>Experimental procedures that allowed to…</description> <release>1.0</release> <release_date>2007-Feb-16</release_date> <entry_count>5697</entry_count> <entries> <entry id="EBI-77680"> … Dump file (XML)

(15)

Divide and Conquer the Indexing

UniProt (>4M entries) Embl (>83M entries) 2 files, ~9.4G >600 files ~ 375G Medline (>16M entries) >500 files ~ 57G XML Taxonomy (>0.37M entries) 1 file, ~81M GO (>0.23M entries) 1 file ~ 27M XML

Others (ArrayExpress Ensembl, Intact, …)

XML dump XML dump XML dump

8 cpu 8 cpu 8 cpu 8 cpu

XML XML XML dump XML dump XML dump Embl Index Uniprot Index Embl Index Taxonomy Index Medline Index ArrayExpress Index Ensembl Index Intact Index XML XML XML dump XML dump XML dump Db

(16)

Let’s put some figures on it

Less than 18 hours to index all the EBI

0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000 60,000,000 70,000,000 80,000,000 e m b ln e w _ s td ip rm a tc h e s e m b ld e le te d u n ip ro t e m b ln e w _ w g s e m b lc d s m e d li n e re le a s e e m b lr e le a s e _ w g s e m b lr e le a s e _ s td 00:00:00 01:12:00 02:24:00 03:36:00 04:48:00 06:00:00 07:12:00 08:24:00 09:36:00 Entries Time (hours)

(17)

Web side story

UniProt Index Embl Index Taxonomy Index Medline Index ArrayExpress Index Ensembl Index Intact Index

Load balancer

(18)

Libraries

• Indexing

– Lucene (

http://lucene.apache.org

)

– ANTLR (

http://www.antlr.org/

)

– ANTXR (

http://javadude.com/tools/antxr/index.html

)

– JGroups (

http://www.jgroups.org

)

• Web

– Tomcat (

http://tomcat.apache.org/

)

(19)

EB-eye

• Global search mechanism

– Searches most of the EBI resources in one go

– Not specific to any resource

• Unified searches of the EBI resources

– Free-text search (unified semantic)

– Basic results display (Google-like)

– Simple cross reference navigation

(20)

EB-eye results summary page

• Organized into categories

called “domains”

• Number of results per domain

• Refine your search

• Expand/Collapse for more

(21)

EB-eye domain result page

• Results for all the resources in a domain

• A domain can contain several resources

• First 3 entries displayed for each resource

• View more entries for a particular resource

• Hierarchy of domains

• Forward search (smaller set of resources)

• Backward search (wider set of resources)

• Refine your search

(22)

EB-eye domain result page (one resource)

• Basic information: ID, name, description…

• Link to the main resource web site

• Additional links

(23)

EB-eye cross-references navigation

• Navigate inside the EB-eye

• References context

• Navigation…

• Using resources explicit references

(24)

EB-eye Advanced Search

• Accessible from all the pages

• Simple search criteria

• Domain specific search

• Domain selection

• Fields selection

(25)

Exercices

• Find about protease but not about protease inhibitors

• proteases are classified into 4 large groups.

Find out about cysteine and/or serine proteases but not

about aspartic or metallo peptidases

• Find all uniprot entries regarding SARS but not those that

are incomplete (are fragments) or a putative (no evidence

they are what they pretend to be) and represent the spike

glycoprotein.

(26)

Queries with SRS

Curso práctico de base de datos e

integración de informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(27)

Overview

• What is SRS?

– Data Integration

– Subentries

– Analysis Tools

– Search Terms

• SRS Web Interface

– Quick Searches

– Library Specific Searching

– Query History

– Linking

– Analysis Tools

– Databank Information and Index Browsing

(28)

What is SRS?

• Central resource for biological data

• Data integration and retrieval system

– more than 200 databases

• Databases are linked via common data

• Data analysis applications server

– more than 150 applications

• Run tool on query results or user supplied data

• Results linked to related data in other databases

(29)

SRS Approach to Data Integration

• Parse original data

• Reformat data if necessary

• Create searchable indices

• Extrapolate links between datasets

• Standardise field formats

(30)

Main page

Tips and Aids

Databank Information

Information

(33)

Databank Information

(34)

Databank Information

(35)

(36)

(37)

(38)

SRS query language: index search

• Index queries can be targeted to specific data sources and to specific

fields in them, or full entries (AllText)

– [database-field:searchterm], e.g, search all proteins in SwissProt with term

"kinase" in the description field

[swissprot-Description:kinase]

– applicable database and field names can be found from the web-interface

– SRS is completely case-insensitive, this applies to search terms, database

names, and field names

• Searchterms can be

– plain strings

– strings with wildcards

– regular expressions

– numeric range queries

• Searchterms can be combined with boolean operators

• Queries return sets of accession numbers or ids

(39)

SRS query language: expressions

• Wildcards: * matches any string, ? matches any character

• Regular expressions are given within slashes /expr/

– syntax similar to regular expressions in python (character classes

[]; grouping (); alternatives |, repetitions *,+,?; beginning and end of

word ^,*)

– Example: pattern /^[OPQ][0-9][A-Z0-9][A-Z0-9][A-Z0-9][0-9]$/

matches UniProt accession numbers

• Numeric ranges x:y (use hash-mark instead of colon

between field name and search term)

– [medline-year#1999:2003]

– boundaries can be excluded with ! in front of the boudary, e.g.,

100:!200 means

≥

100 and <200

(40)

SRS query language: operators

• Boolean operators: and(&), or(|), butnot(!)

– [swissprot-Description:kinase!inhibitor]

• Linking

– [swissprot-Description:kinase] > PATHWAY

• Searching multiple data sources (only field names

common to all selected databases can be used) :

(41)

Query expressions

• Index queries (e.g.,

**[embl-des:kinase!inhib*]**

) and

full databases (e.g.,

pdb

) are query expressions

• Query expressions can be combined with operators and

grouped with parentheses to form more complex query

expression

• The operators are:

– set intersection (&), union (|), and subtraction (!)

– link-operators < and >

(42)

(43)

(44)

(45)

Library Search

Quick search input box

Search types

(46)

Quick Search

UniProtKB Selected

(47)

(48)

Quick Search

UniProtKB selected Search term entered

Group expanded

(49)

Quick Search

Results from multiple databases

(50)

Standard Query Form

UniProtKB selected Click on standard query form

(51)

Standard Query Form

Search options Predefined views Input fields Creating a view

(52)

Standard Query Form

Choosing fields

(53)

(54)

Standard Query Form

(55)

Standard Query Form

(56)

(57)

Standard Query Form

Include sequence Change to list view Choose sequence format

(58)

(59)

Standard Query Form - Subentries

(60)

(61)

(62)

(63)

Standard Query Form - Subentries

(64)

(65)

(66)

Extended Query Form

UniProtKB selected

Click on Extended query form

(67)

Extended Query Form

Special date

field type User chosen

fields

Choose view type

(68)

(69)

(70)

(71)

2007/6/30 Curso práctico de base de datos e integración de información biológica 71 Click link button Choose query Link options

Linking

(72)

Linking

Databases to link to

(73)

(74)

(75)

(76)

(77)

(78)

(79)

2007/6/30 Curso práctico de base de datos e integración de información biológica 79 List of queries Result options View options Direct input of an SRS query

Query History

(80)

2007/6/30 Curso práctico de base de datos e integración de información biológica 80 Choose view Choose query Click to rerun query

Rerun Query

(81)

(82)

2007/6/30 Curso práctico de base de datos e integración de información biológica 82 Click to combine Choose boolean logic to combine Choose multiple queries

Combine Queries

(83)

(84)

2007/6/30 Curso práctico de base de datos e integración de información biológica 84 Choose output to file or browser Choose output format

Save Results

(85)

(86)

Subentries

• Subentries are used when there is repeated structured

information within an entry.

• Need to be able to search inside this structured information

as if they were each isolated entries

(87)

(88)

Subentries

Another example – features

E.g. searching for key

ACT_SITE

and description

(89)

(90)

BioMart

Curso práctico de base de datos e integración de

informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(91)

BioMart

• A collaboration

– European Bioinformatics Institute (EBI)

– Cold Spring Harbor Laboratory (CSHL)

• Aim

– To develop a generic data management system that works well for

biology

(92)

Curso práctico de base de datos e integración de información biológica FK FK FK FK PK

PK

FK

(93)

Deployment

Source data

Transformation

1

(94)

(95)

Deployment

XML XML XML

Configuration

2 Source data

Transformation

1

(96)

(97)

Deployment

Source data

XML XML XML

Configuration

2 Transformation

1 BioMart software

3 Querying

(98)

Datasets, Attributes and Filters

GENE gene_id(PK) gene_stable_id gene_start gene_chrom_end chromosome gene_display_id description

Mart

Dataset

Attribute

Filter

(99)

Data federation

Links

Dataset 1

Exportable

name = uniprot_id attributes = uniprot_ac

Dataset 2

Importable

name = uniprot_id filters = uniprot_ac

(100)

Data federation

Links

Dataset 1

Exportable

name=genomic_region attributes=chr_name, chr_start, chr_end

Dataset 2

Importable

name=genomic_region filters=chr_name (=), chr_start (>=), chr_end (<=)

(101)

Data Flow

JAVA

PERL

Web

Service

Web

GUI

Command

line

Desktop

GUI

XML

Mart

(102)

(103)

Examples

• Retrieve all SNPs for ‘novel’ human G-protein coupled receptor genes (GPCRs

– IPR000276) on chromosome 2.

• Retrieve the sequences of the exons of the human MEFV gene in FASTA

format.

• Retrieve the gene structure (i.e. start and end coordinates of exons) of the

mouse gene ENSMUSG00000042351.

• Retrieve all human disease genes containing transmembrane domains located

between p11.2 and q22.

• The file contains a list of probeset IDs from a microarray experiment using the

Affymetrix array HG-U133 Plus 2.0 (human). Retrieve the 500 bp upstream of

the transcripts matching these probeset IDs.

• Retrieve the sequences 5kb upstream of all human ‘known’ genes between

D1S2806 and D1S464.

• Retrieve all human SNPs that have an ID from The SNP Consortium (TSC),

from chromosome 6 between 15 Mb and 15.2 Mb, with 200 bases flanking

sequence.

• Retrieve the mouse homologues of Homo sapiens genes CASP1, CASP2,

CASP3, and CASP4.

(104)

Distributed Annotation System

Curso práctico de base de datos e integración de

informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(105)

Distributed Annotation System

• Developed in 1999/2000 by Lincoln Stein (CSHL) and

collaborators as a GFF-based web service.

• Biodas.org (

http://www.biodas.org

):

• A client-server system in which a single client integrates

information from multiple servers.

• It allows a single machine to gather up genome annotation

information from multiple distant websites, collate the

(106)

General Feature Format

• Genome annotations, especially structural annotations, are often

stored in GFF (General Feature Format) files.

• GFF is a simple tab-delimited data format.

• Fields are: <seqname> <source> <feature> <start> <end> <score>

<strand> <frame> [attributes] [comments]

(107)

DAS Concept

Reference server

Annotation server B

Annotation server A

Annotation server C

(108)

(109)

DAS Server

• DAS request to retrieve the data sources on a DAS server:

• http://das.ensembl.org/das/dsn

(110)

DAS Server

• DAS request to retrieve features on a segment:

• http://das.ensembl.org/das/ens_36_omim_genes/features?

segment=1:1,1000000

(111)

(112)

Links

• http://www.dasregistry.org/listServices.jsp

• http://www.ebi.ac.uk/das-srv/uniprot/das/aristotle/sequence?segment=P00280

• http://www.ebi.ac.uk/das-srv/uniprot/das/aristotle/features?segment=P00280

• http://www.ebi.ac.uk/dasty/display.html?q=P00280

(113)

Web services

Curso práctico de base de datos e

integración de informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(114)

REST: REpresentational State Transfer

http://www.ebi.ac.uk/Tools/webservices/rest/dbfetch/uniprot/slpi_human

GET, POST

HTML,XML,PNG

RESTful web services

(115)

SOAP: Simple Object Access Protocol

fetchData(uniprot,wap_rat,default,xml)

SOAP services

(116)

(117)

WSDL structure

• <wsdl:definitions>

– <wsdl:types>

• <schema/>

– </wsdl:types>

– <wsdl:message>

– <wsdl:portType>

• <wsdl:operation/>

– </wsdl:portType>

– <wsdl:binding />

– <wsdl:service />

• </wsdl:definitions>

(118)

Messages

• <wsdl:message name="fetchDataRequest">

– <wsdl:part name="query" type="xsd:string"/>

– <wsdl:part name="format" type="xsd:string"/>

– <wsdl:part name="style" type="xsd:string"/>

• </wsdl:message>

• <wsdl:message name="fetchDataResponse">

– <wsdl:part name="fetchDataReturn" type="xsd:string"/>

(119)

PortType

• <wsdl:portType name="WSDBFetchServer">

• <wsdl:operation name="fetchData"

parameterOrder="query format style">

• <wsdl:input message="impl:fetchDataRequest"

name="fetchDataRequest"/>

• <wsdl:output message="impl:fetchDataResponse"

name="fetchDataResponse"/>

• </wsdl:operation>

• </wsdl:portType>

(120)

Binding

• <wsdl:binding name="WSDbfetchSoapBinding"

type="impl:WSDBFetchServer">

• <wsdlsoap:binding style="rpc“

transport="http://schemas.xmlsoap.org/soap/http"/>

• <wsdl:operation name="fetchData">

– <wsdlsoap:operation soapAction=""/>

–

<wsdl:input name="fetchDataRequest">

• <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"

namespace="http://wsdbfetch.ws.jdbfetch.ebi.ac.uk" use="encoded"/>

–

</wsdl:input>

– <wsdl:output name="fetchDataResponse">

• <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"

namespace="http://www.ebi.ac.uk/ws/services/WSDbfetch" use="encoded"/>

– </wsdl:output>

• </wsdl:operation>

• </wsdl:binding>

(121)

Service

• <wsdl:service name="WSDBFetchServerLegacyService">

– <wsdl:port binding="impl:WSDbfetchSoapBinding" name="WSDbfetch">

• <wsdlsoap:address

location="http://www.ebi.ac.uk/ws/services/WSDbfetch"/>

– </wsdl:port>

(122)

Accessing EBI Web services

Curso práctico de base de datos e

integración de informacion biológica

Segovia 4 de Julio de2007

Alberto Labarga – EMBL-EBI

(123)

Overview

• Technologies

– REST

– DAS

– SOAP (rpc/enc, doc/lit)

• Services

– Dbfetch

– EB-Eye

– SRS

– Martservice

(124)

REST: REpresentational State Transfer

http://www.ebi.ac.uk/Tools/webservices/rest/dbfetch/uniprot/slpi_human

GET, POST

HTML,XML,PNG

(125)

Any web page is a web service

(126)

Friendly URL and XML documents

• http://www.ebi.ac.uk/Tools/webservices/rest/dbfetch/uniprot/slpi_human

• http://www.ebi.ac.uk/Tools/webservices/rest/dbfecth/uniprot/slpi_human/xml

• http://www.ebi.ac.uk/Tools/webservices/rest/dbfetch/uniprot/slpi_human/fasta

(127)

Available databases

• EMBL Nucleotide Sequence Database

(

embl

)

• EMBL CDS (

emblcds

)

• EMBL-Contig (

emblcon

)

• EMBL Annotated Contigs (

emblann

)

• EMBL Sequence Version Archive

(

emblsva

)

• Genome Reviews (

genomereviews

)

• Human Genic Bi-Allelic Sequences

Database (

hgvbase

)

• NCBI Reference Sequence

– refseq

,

refseqp

• Integrated Resource of Protein Domains

and Functional Sites (

interpro

)

• Medline (

medline

)

• International Protein Index (

ipi

)

• Uniprotkb (

uniprot

)

• UniProt Reference Clusters

– uniref100,uniref50, uniref90

• UniProtKB Sequence/Annotation

Version Archive (

unisave

)

• UniProtKB Archive (

uniparc

)

• Protein Data Bank (

pdb

)

• Patent Protein Sequences

(128)

REST client

use LWP::UserAgent;

use URI::URL;

my $ua= LWP::UserAgent->new();

my $u=“http://www.ebi.ac.uk/Tools/webservices/rest/dbfetch/uniprot/wap_rat"

my $req = GET URI::URL->new($u);

my $resp = $ua->request($req);

(129)

MartService

• Overview information(GET)

– Marts

– Datasets

– Configuration

• Queries (POST).

(130)

MartService Overview (GET)

• Marts

?type=registry

• Datasets

?type=datasets&mart=mymart

• Configuration

?type=configuration&dataset=mydataset

(131)

Web service

<Query virtualSchemaName="central_server_1">

<Dataset name="

hsapiens_gene_ensembl

" >

<Attribute name="

ensembl_gene_id

"/>

<Attribute name="

ensembl_transcript_id

"/>

<Filter name="

chromosome_name

" value="

1 "/>

<Filter name="

band_end

" value=”

p36.33

"

/

>

<Filter name="

band_start

" value=”

q44

"/>

</Dataset>

<Dataset name="

msd

">

<Attribute name="

pdb_id

"/>

<Attribute name=”

experiment_type

"/>

<Filter name="

experiment_type

" value=”

NMR

"/>

</Dataset>

</Query>

(132)

SRS wgetz

• Usage: wgetz [-ew] [-page wwwPageName] [-id userId] [-uName

uName] [-gName gName] [-permUserId permUserId] [-l libList] [-lib

libName] entry entryName] i2f] dbg] dbr] dbs] dbt] dbw] lo]

[-info libInfo] [-qfrom queryFrom] [-launchFrom launchFrom] [-lfrom

linkFrom] [-from fromLibName] [-to toLibName] [-f fieldList] [-bv

viewEntriesStartN] [-lv viewEntriesChunkSize] [-lset listSetNumber] [-sf

seqFormat] [-q queryString] [-mime mimeType] [-qnum queryNumber]

enum entryNumber] snum sessionNumber] vt] vn viewNumber]

[-view [-viewName] [-rs [-viewRecordSep] [-cs [-viewColumnSep] [-bf

browseIndex] [-blib browseLibs] [-ifile icarusFileType] [-newId] [-appl

appl] [-package package] [-version] [-codeStrings] [-off] [-bioScout

bioScoutId] [-jobName jobName] [-debug] [-ascii] [-ldrName

loaderName] sort sortOn] sortDir sortDir] sqlImage sqlImage]

[-pMan projMan] [-fMan fromMan] [-dlout] [-dlkey dlkey] [-noSession]

'queryExpression'

(133)

SRS wgetz

• http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz

?

• -e+[SWISSPROT-acc:P32234]

• -e+[SWISSPROT-acc:P32234]+-ascii

• -e+[SWISSPROT-acc:P32234]+-vn+1 (get IDs)

• -e+[SWISSPROT-acc:P32234]+-vn+2 (get entries)

• -e+[SWISSPROT-acc:P32234]+-vn+4 (get sequence)

(134)

SRS wgetz

• +-page+cResult+-ascii

(count results)

• +-view+SequenceSimple

(get sequences)

• -permUserId+SEGOVIA2007 (use saved queries and

views)

(135)

Perl client

use LWP::Simple;

my $srs="http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz";

my $id = "SEGOVIA2007";

my $query = "[interpro:*]";

my $view= "FAMILIAS";

#

http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz?-permUserId+SEGOVIA2007+-ascii+-view+FAMILIAS+[interpro-id:]

my $url = “$srs?-permUserId+ $id+-ascii+-view+$view+$query";

print get $url;

(136)

DAS Perl client

use LWP::Simple;

use XML::Simple;

$s="http://www.ebi.ac.uk/das-srv/uniprot/das/aristotle/sequence?segment=P00280"; $f="http://www.ebi.ac.uk/das-srv/uniprot/das/aristotle/features?segment=P00280";

$xml = new XML::Simple;

my $features = $xml->XMLin(get $f);

my $sequence = $xml->XMLin(get $s);

print $sequence->{SEQUENCE}->{content};

my $list = $features->{GFF}->{SEGMENT}->{FEATURE};

foreach $feature (keys %$list){

print "$feature: ";

print "$list->{$feature}->{START} $list->{$feature}->{END}\n";

}

(137)

SOAP: Simple Object Access Protocol

fetchData(uniprot,wap_rat,default,xml)

SOAP services

(138)

fetchData (db, id, format, style)

entry

(139)

wsdbfetch

• fetchData dbName:id <output format> <output style>

• fetchBatch dbName 'id1;id2' <format> <style>

• getSupportedDBs

• getDbFormats dbName

(140)

Perl client

use SOAP::Lite;

my $WSDL='http://www.ebi.ac.uk/Tools/webservices/wsdl/WSDbfetch.wsdl';

my $soap = SOAP::Lite->service($WSDL);

# fetchData dbName:id <format> <style>

my $result = $soap->fetchData(‘uniprot’, ‘default’, ‘raw’);

die $soap->call->faultstring if $soap->call->fault;

(141)

Python

from SOAPpy import WSDL

wsdl ="http://www.ebi.ac.uk/Tools/webservices/wsdl/WSDbfetch.wsdl")

dbfetch = WSDL.Proxy(wsdl)

resultList = dbfetch.fetchData("uniprot:slpi_human", "fasta", "raw")

for result in resultList: print result

Ruby

require 'soap/wsdlDriver'

wsdl = 'http://www.ebi.ac.uk/Tools/webservices/wsdl/WSDbfetch.wsdl'

dbfetchSoap = SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver

puts dbfetchSoap.fetchData("uniprot:slpi_human", "fasta", "default")

(142)

EBI web services (global search)

getResultsIds (db, query)

ids

getResults (db, query)

results

(143)

Web Services API: Metadata

• List available domains

(i.e. available for query in EB-eye)

String[] listDomains()

• List available fields in a domain

(e.g. id, description…)

String[] listFields(String domain)

> listDomains() 2can arrayexpress-experiments arrayexpress-genes arrayexpress-repository biomodels chebi emblcds embldeleted emblnew_con emblnew_standard emblnew_wgs emblrelease_con … > listFields(“msdpdb”) id acc_number name description

(144)

Web Services API: Metadata

• Get referenced domains in a domain or an entry

(i.e. EB-eye domains referenced in a domain or an entry)

String[] getDomainsReferencedInDomain(String domain)

String[] getDomainsReferencedInEntry(String domain,

String entryId)

• List additional databases referenced in a domain

(i.e. non EB-eye domains referenced in a domain)

String[] listAdditionalReferenceFields(String domain)

> getDomainsReferencedInDomain(“msdpdb”) go intenz interpro medlinenew medlinerelease msdpdb taxonomy uniprot > listAdditionalReferenceFields(“msdpdb”) CATH PFAM SCOP

(145)

Web Services API: Basic Full-text Search

• Get number of results for a simple query

int getNumberOfResults(String domain, String query)

• List result IDs for a simple query

String[] getResultsIds(String domain, String query, int

start, int size)

• List result fields values for a simple query

String[][] getResults(String domain, String query,

String[] fields, int start, int size)

> getNumberOfResults(“msdpdb”, “dopamine”) 18 > getResultsIds(“msdpdb”, “dopamine”, 0, 3) 60734 14833 12412

> getResults(“msdpdb”, “dopamine”, [“id”, “acc_number”, “name”, “description”], 0, 3)

"60734" "2a3r" "CRYSTAL STRUCTURE OF HUMAN…" "TRANSFERASE XRAY entry at resolution 2.6" "14833" "5pah" "HUMAN PHENYLALANINE…" "MONOOXYGENASE XRAY entry at resolution 2.1" "12412" "1i15" "DOPAMINE D2 RECEPTOR…" "SIGNALING PROTEIN TMODEL entry at resolution…"

(146)

Web Services API: Entry Search

• Get result fields values for a particular entry

String[] getEntry(String domain, String entryId, String[]

fields)

• Get result fields values for a list a entries

String[][] getEntries(String domain, String[] entryIds,

String[] fields)

> getEntry(“uniprot”, “NR1H2_HUMAN”, [“id”, “acc_number”, “description”])

NR1H2_HUMAN P55055

Oxysterols receptor LXR-beta (Liver X receptor beta) (Nuclear orphanreceptor LXR-beta)…

> getEntries(“uniprot”, [“NR1H2_HUMAN”, “Q8BP65_MOUSE”, “Q8AXU8_CHICK”], > [“id”, “acc_number”, “description”])

"NR1H2_HUMAN" "P55055" "Oxysterols receptor LXR-beta (Liver X receptor beta) (Nuclear..." "Q8BP65_MOUSE" "Q8BP65" "8 days embryo whole body cDNA, RIKEN full-length enriched... " "Q8AXU8_CHICK" "Q8AXU8" "Liver X receptor alpha."

(147)

Web Services API: Cross-references Search

• Get referenced entries for a domain in a particular entry

String[] getReferencedEntries(String domain, String

entryId, String referencedDomain)

> getReferencedEntries (“uniprot”, “NR1H2_HUMAN”, “medlinerelease”)

7926814 12819202 15489334

(148)

Eb-Eye perl client

use SOAP::Lite;

my $nameSpace = 'http://webservice.ebinocle.ebi.ac.uk';

my $endPoint = 'http://www.ebi.ac.uk/ebisearch/service.ebi';

my $search = SOAP::Lite-> uri($nameSpace)-> proxy($endPoint);

my $result = $search->getResultsIds("uniprot", "kinase", 0, 100);

my @ids = $result->valueof('//getResultsIdsResponse//out//string');

(149)

EBI web services (analysis tools)

jobid

getResults (jobid)

results available

checkStatus (jobid)

status

run(params, data)

poll (jobid, type)

result file

(150)

Perl

• use SOAP::Lite;

• my $WSDL = 'http://www.ebi.ac.uk/Tools/webservices/wsdl/WSFasta.wsdl';

• my $fasta_client = SOAP::Lite->service($WSDL);

• my %params=();

• $params{'program'}='fasta3';

• $params{'database'}='uniprot';

• $params{'email'}='[email protected]';

• $params{‘async'}= 1;

• $data={type=>"sequence",

• content=>"MRCSISLVLGLLALEVALARNLQEHVFNSVQSMCSDDSFSEDTECI"};

• # $data={type=>"sequence",

• # content=>“uniprot:slpi_human"};

• my $jobid = $fasta_client >runFasta(

• SOAP::Data->name('params')->type(map=>\%params),

• SOAP::Data->name( content => [$data]));

(151)

Perl client (cont.)

# set a loop for checking job submission status

# RUNNING, NOT_FOUND, ERROR, DONE

my $status = $fasta_client ->checkStatus($jobid);

while (status eq "RUNNING") {

sleep 10; $status = $fasta_client->checkStatus($jobid);

}

# when job is done, poll for the results

my $result = $fasta_client ->poll($jobid) if ($status eq "DONE") ;

(152)

Perl client (cont.)

my $results = $fasta_client->getResults($jobid);

die $soap->call->faultstring if $soap->call->fault;

for $result (@$results){

# tooloutput, toolxml, toolaln, toolpng

$res=

$fasta_client->poll($jobid,$result->{type});

write_file($outfile.".".$result->{ext},$res);

(153)

REST client

use LWP::UserAgent;

use HTTP::Request;

use HTTP::Request::Common;

use URI:URL;

$URL="http://www.ebi.ac.uk/Tools/webservices/rest/submit";

my %params=();

$params{'tool'}="iprscan";

$params{'sequence'}="uniprot:slpi_human";

$params{'seqtype'}="P";

$params{'email'}="[email protected]";

@args=%{\%params};

$ua= LWP::UserAgent->new();

$submit_url= URI::URL->new($URL);

$resp = $ua->request( POST $submit_url

, 'Content' => \@args);

print $resp->content();

(154)

Curso práctico de base de datos e integración de información biológica Ensembl

EnsemblEnsembl Ensembl ((((MartserviceMartserviceMartservice))))Martservice

Ensembl Ensembl Ensembl Ensembl ((((MartserviceMartserviceMartserviceMartservice))))

Data Retrieval Data RetrievalData Retrieval Data Retrieval ((((WSDbfetchWSDbfetchWSDbfetchWSDbfetch))))

Data Retrieval Data Retrieval Data Retrieval Data Retrieval ((((WSDbfetchWSDbfetchWSDbfetch))))WSDbfetch

Expression Profiler Expression Profiler Expression Profiler Expression Profiler

Expression Profiler Expression ProfilerExpression Profiler Expression Profiler

Protein families, Protein families, Protein families, Protein families, motifs and domains motifs and domains motifs and domains motifs and domains

((((WSInterProScanWSInterProScanWSInterProScanWSInterProScan))))

Protein families, Protein families, Protein families, Protein families, motifs and domains motifs and domains motifs and domains motifs and domains

((((WSInterProScanWSInterProScanWSInterProScan))))WSInterProScan

Protein structure & function: Protein structure & function: Protein structure & function: Protein structure & function:

(MSD API) (MSD API) (MSD API) (MSD API)

Protein structure & function: Protein structure & function: Protein structure & function: Protein structure & function:

(MSD API) (MSD API) (MSD API) (MSD API)

Literature and text mining Literature and text mining Literature and text mining Literature and text mining

((((WhatizitWhatizitWhatizitWhatizit, , , , CiteXploreCiteXploreCiteXploreCiteXplore))))

Literature and text mining Literature and text mining Literature and text mining Literature and text mining

((((WhatizitWhatizitWhatizitWhatizit, , , CiteXplore, CiteXploreCiteXplore))))CiteXplore

Sequence homology Sequence homology Sequence homology Sequence homology WSWUBlast WSFasta WSMPsrch WSScanPS Wise, PromoterWise, ScanWise Sequence homology Sequence homology Sequence homology Sequence homology WSWUBlast WSFasta WSMPsrch WSScanPS Wise, PromoterWise, ScanWise Sequence alignment Sequence alignment Sequence alignment Sequence alignment WSClustalW WSMuscle WSTCoffee Sequence alignment Sequence alignment Sequence alignment Sequence alignment WSClustalW WSMuscle WSTCoffee Sequence analysis Sequence analysis Sequence analysis Sequence analysis (WSEmboss) Sequence analysis Sequence analysisSequence analysis Sequence analysis

(WSEmboss)

Ontologies ( Ontologies (Ontologies (

Ontologies (OntologyLookupOntologyLookupOntologyLookupOntologyLookup))))

Ontologies ( Ontologies ( Ontologies (

Ontologies (OntologyLookupOntologyLookupOntologyLookup))))OntologyLookup

DaliLite, Maxsprout DaliLite, Maxsprout Integr8 API Integr8 API Integr8 API Integr8 API Integr8 API Integr8 APIIntegr8 API Integr8 API

Uniprot UniprotUniprot Uniprot APIAPIAPIAPI

Uniprot Uniprot Uniprot Uniprot APIAPIAPIAPI

Available Services

ID mapping ( ID mapping (ID mapping (

ID mapping (PicrPicrPicrPicr, , , , MartserviceMartserviceMartserviceMartservice))))

ID mapping ( ID mapping ( ID mapping (

ID mapping (PicrPicrPicr, Picr, , , MartserviceMartserviceMartservice))))Martservice

Search (EB-Eye)

ChEBI ChEBI ChEBI ChEBI APIAPIAPIAPI

ChEBI ChEBIChEBI ChEBI APIAPIAPIAPI

(155)

(156)

(157)

Documentation

(158)