DISCOVERING SEMANTICSIN MULTIMEDIACONTENT USING WIKIPEDIA
ANGELA FOGAROLLI AND MARCO RONCHETTI
∗
Abstrat. Semanti-based informationretrievalisanarea ofongoingwork. Inthispaper wepresenta solutionforgiving
semantisupporttomultimediaontentinformationretrievalinane-Learningenvironmentwhereveryoftena largenumberof
multimediaobjetsandinformationsouresareusedinombination. SemantisupportisgiventhroughintelligentuseofWikipedia
inombinationwithstatistialInformationExtrationtehniques.
Keywords: ontentretrievalandltering: searhoversemi-struturalWebsoures,multimedia,Wikipedia,e-Learning
1. Introdution. Nowadays,organizationshavetodealwith informationoverloading. Theyneedaway
to organize and store their ontent and being able to easily retrieve it when neessary. Our objetive is to
provideasystemforindexing andretrievingontentbasedonthesemantiprovidebyWikipedia. Retrieving
thedesiredontentanbediultduetothethehighspeiallyoftermsin asearhtask.
Inourwork,weareaddressingtheproblemofaessingdierentkindsofunstruturedorsemi-strutured
informationsourestakingadvantagesofthesemantiprovidedbypubliavailableresouressuhasWikipedia.
Furthermoreusingtheapproahwewilldesribeinsetion4wewouldliketoautomatizethetaskofannotating
aorpusanddisoverrelationsbetweenannotations. Nextwewill useannotationinombinationwith textual
informationretrievalfordeterminingthesearhontextandbasedonitwewillbeabletogivesearhsuggestions
and perform query expansion. Using annotation in information retrieval is not a new idea [6, 4℄ even in
ombination withontologies[3℄, it hasbeenwidely used in videoand imageretrievalgenerating alsoasoial
phenomenalikefolksonomy[15,13℄. What isnewis theuseof domainindependentpubli available semanti
toautomatiallydesribeontentin dierentkindofmedia.
We are applying our approah in the e-Learning ontext, speially enhaned streaming video letures
(see [8, 5℄) beause of the peuliarity in this senario of ombining dierent kinds of unstrutured or
semi-struturedsouresofinformation. E-Learningpresentsmanyproblematisinommonwiththebusinesssenario
in termsofontent lassiationforits amountof informationto lassifyandfor thedierentontextswhere
aspei information anbe relevant. Our target repository ollets dierent kinds of media (video, audio,
presentationslides, text douments),whih anbesearhedandpresentedin ombination. Foreah reorded
event(e.g: leture,seminar,talk,meeting)weprovidenotonlythevideobutalsorelatedmaterials,whihan
onsistof presentation slides,doumentsorWeb sites thespeakerpointsto. All theresouresaretemporally
synhronizedwiththevideo.
An exampleof howamultimedia presentationof thiskindlookslikeanbefoundin gure1.1;thevideo
withthespeakerappearstogetherwithpresentationslidesoradditionalnotes. Videoandslidesaresynhronized
andanbenavigatedbymeansofatemporalbarorbyslidetitles.
Weansummarizethefollowingvestateoftheartapproahesto multimediaindexingandnavigation:
1. Useofmetadatatobrowsekeyframes.
2. Usetextfromspeeh,using transript-basedsearh.
3. Mathing keyframesvs. queryingofimages. Keyframesextratedasshotrepresentativesareused for
retrieval. Itrequires userto loateimages/otherkeyframes,frombrowsingorothersearh.
4. Useof semantifeatures. They arebased uponpre-proessing videoorkeyframesto detetfeatures.
Featuresanberelatedto ontologies.
5. Usevideo/imageobjetsasqueries.
We onentrate onpt. (2) and partially onpt. (3), we use thetext-from-speeh tehniqueombinedwith a
textualanalysisofthespeehandtheeventrelatedmaterialusingWikipediainsteadofontologies.
InWikipedia,theoneptoflassandinstanearenotseparatedasintheontologialsense,duetothefat
thatitisnotonstrainedtoaformalmodel,forthereasonofwhihitisnotpossibletoformalizereasoningon
theWikipediaontentdiretly.
The use of Wikipedia url as suggested in[10℄ for onept identiation ould guarantee interoperability
betweendomainontologies, whiletheextensiveongoingresearheortforextrating anontologialviewfrom
∗
University of Trento, Dept. of Information and Communiation Teh., Via Sommarive 14, 38050 Trento, Italy
Fig.1.1.Anexampleofmultimediapresentation
WikipediaouldsoonleadtothereationofanontologialviewbasedonWikipediawhihouldbethereferene
formanydomains.
Theombinationofinformation extratedfrom videoand relatedmaterial givesaompletepiture ofan
event,sinein thereal worldthesumofall themediausedbyaspeakerismeantto fullydesribetheevent's
topisto failitateknowledgetransfertotheaudiene.
Inthispaperwereportabouthowweprovidesemantisupportandunsupervisedannotationofmultimedia
materialbasedoninformationextratedfrom Wikipedia,ratherthantheusageof SemantiWebtehnologies
(speially withoutontologies). Ourapproahisdomain independent,and intheory itouldalso be applied
todierentuseaseswherethereisaneedforlusteringorannotationofaorpus.
The struture of this paper is organizedas follows: in the nextsetion we desribe the ontext and the
motivation of our work; setion 4 gives an overview of our approah. In setion 5 we apply the approah
desribed in the previoussetion to ouruse ase. Finally, we disuss the diretions we are planningto take
regardingfurther work.
2. Semantis in the Web. In theWeb, someolletionsof data ontainingsemanti annotations(e.g.
UniProt: http://www.ebi.a.uk/swissprot/index.html,Eademy:http://www.eademy.om)arenow
ava-ilableandthereis atrendto semantiallyenablemoreandmoreWebontent. Eventhoughthis trendis
per-eivable,thereisstillahugeamountofmaterialonwhihthesetehnologieshavenotbeenapplied. Onelimiting
fatorforafasteradoptionofSemantiWebtehnologies,isthediultytondready-to-useoneptualizations
forannotatingexisting materialandmakingitSemantiWebompatible.
relationshipsbetweenentitiesinoneormoredomainontologies. Weexperieneddiultiesinndingontologies
whihoveravarietyofdomains,sinee-Leturesanoveranunpreditableamountofdomains(e.g. omputer
siene,history,meteorology,geography,math...). Inaddition,the termsexpressedin e-Leturesare usually
individuals of an ontology (e.g. the term 'Colletion' in aJava Programming lass ould be modelled as an
instane of adata ontainer lass in a JavaProgramming ontology) and ndingpopulated ontologies with a
wide overage of individuals to date is a big hallenge, and usually requires the involvementor aknowledge
engineer.
Our requirement wasto nd a broad, domain-independent olletionof individual terms (as opposed to
onepts) whih are onneted by relations. To the best of our knowledge, the most omplete olletion of
thiskindisWikipedia. Wikipediaisafreelyavailableenylopediawhih isonstantlygrowinginsize andin
fame thanks to the opyleft liense that allows the ontent to be opied, modied and redistributed aslong
as there is an aknowledgment of the author and the new ontent is published under the same liense (see
http://www.wikipedia.org). Wikipedia ontains a lassiation of topis, organized with an hierarhy of
ategoriesand withrelationshipsbetweenelements. Theadvantageof usingit isthat the soialollaborative
network around it makes its ontent always up to date and it overs in details a huge amount of topis in
dierentdomainsandlanguages.Inadditionitalsotakesintoaountthedierentpossiblemeaningsofaterm
throughadisambiguationpage.
What we anextrat using Wikipedia are the relationshipsbetween topis. Aording to Obrst's
deni-tions in [11℄, Wikipedia notonly oers weak semanti information, suh asparent-hildrelationships, but it
also ontains lexiographirelationships thatone the domain of interest is determinedan oer medium
semanti. InWikipediawedonothavestrongsemanti,i.e. weannotdesribereal-worldrelationshipssuh
asaar hasaminimumoffour wheels aswith theusageofan ontology. Weanonlydeduethat onepts
areonnetedwithoutknowinghow;weantellthat oneoneptin one ategoryisrelatedto otheronepts
whih arelinkedinthedesriptionoftheoneptitself.
InWikipedia,theoneptoflassandinstanearenotseparatedasintheontologialsense,duetothefat
thatitisnotonstrainedtoaformalmodel,forthereasonofwhihitisnotpossibletoformalizereasoningon
theWikipediaontentdiretly. Thereareprojets(seesetion3)that trytoembedsemantiinsideWikipedia
extendingthe Wikisoftware used to write Wikipediapages [16℄, and someothers(e.g. www.dbpedia.org[2℄)
whih provideanRDF representationofWikipedia,tomakeitsontentmahine-interpretable.
WeuseWikipediaas ataxonomytoobtainlexiographirelationshipsand inombinationwithstatistial
information extration we andedue related onepts to the termsextrated from our orpus. Inaddition,
sineour orpusoversa representationof a partof the real worldwealso usethe orpus itself astraining
data"fordomaindisambiguationin Wikipedia.
There is alot of work aboutextrating semantis (some isreportedin setion3) from Wikipedia ontent
to build an ontologial representation. We are therefore ondent that even though for now we an extrat
information without a high semanti value in the feature, at the light and with the ombination of other
researheortintheareawewillbeabletoinreasethepowerofourapproahintermsofexibility,extension
andauray.
3. RelatedWork. Wikipediaontainsavastamountofinformation,thereforetherehavebeenmainlytwo
approahesfor exploringitsontentandmakeitmahinereadable. Therst approah onsistsin embedding
semanti notationsin itsontent [16, 7℄; while the other onedeals with information extration basedon the
understandingofhowtheWikipediaontentisstrutured:[1,14,17,12,19℄.
TheSemantiWikipediaprojet[16℄isaninitiativethatinvitesWikipediaauthorstoaddsemantitagsto
theirartilesinordertomakethemmahineinterpretable.ThewikisoftwarebehindWikipedia(MediaWiki[7℄),
itselfenablesauthorstorepresentstruturedinformationinanattribute-valuenotation,whihisrenderedinside
awikipagebymeansofanassoiatedtemplate.
Theseondmain streamof Wikipediarelatedwork isonautomatially extratknowledge fromthe
Wiki-pediaontentasin[1, 14,17,12,19℄.
DBpedia [1℄is a ommunity eort to extrat strutured information from Wikipedia and to make this
informationavailable ontheWeb. DBpediaoerssophistiatedqueriesagainstWikipediaandtoother linked
datasets on the Web. The DBpedia datasetdesribes 1,950,000 things", inluding at least 80,000 persons,
onsists of around 103 million RDF triples. DBpedia extrats [2℄ RDF triples from Wikipedia informations
presentedinthepagetemplatessuhasinfoboxesandhyperlinks.
Yago [14℄ is a knowledge base whih extends the relationships of DBpedia extending the standard RDF
notation. At Deember2007,Yagoontainedover1.7 millionentities (likepersons, organizations,ities, et.)
A YAGO-query onsists of multiple lines (onditions). Eah line ontains oneentity, a relation and another
entity.
DBpediaor Yagoould replaedWikipediaasasoureof knowledge in oursemantidisoveryapproah,
althoughatthetimeofthiswritingtheseknowledgebasesontainonlyentities(suhaspersonandplaes)and
notabstratoneptsastheonewehaveine-Learningmaterial. Inadditionwedon'tknowaprioriwithwhih
propertiesatermaanbesearhed,soinourdomain replaingWikipidiafree-textwouldnotbebeneial.
ISOLDE [17℄ is a system for deriving a domain ontologies using named-entity tagger on a orpus and
ombining the extrated information with Wikipedia and Wiktionary. The results shows that this kind of
approahworksbetterwithsemi-struture informationsuhasditionaries.
KYLIN [19℄ is another projet whih aim is automatially omplete the information presented in the
Wikipediainfoboxesanalyzingdisambiguatedtext andlinksin Wikipediapages.
Ponzettoetal.[12℄intheirworkhaveexploredinformationextrationonWikipediaforreatingataxonomy
ontainingalargeamountofsubsumptions,i. e. is-arelations.
4. A Semanti Disovery Approah . In this setionweexplain the proessof extrating semantis
frommultimedia ontent. Wetestedtheapproahheredesribedone-Leturepresentations. An e-letureisa
multimedia presentationusually omposed ofavideowithfousonthespeaker,presentationslidesand other
textualdoumentswhihanbeidentifybythepresenterasrelatedsoureofinformation. Dierentmediain
apresentationareusedfordrawingabetterpitureoftheontenttobeommuniated.
Hene its of fundamental importane to take into aount the dierent modalities of the media. In
partiular, we investigate textualmodality analyzingthe fullontentof therelated materialsuh asslides or
doumentsand theauditory modalitytranslating itin textual whih representsthemost promisingaspet of
thedataweanproess. Forthisreasonweapplyautomatispeehreognition(ASR)tothevideosoundtraks
and subsequently we are interested in the STT translation (speeh to text) to provide for data that anbe
analyzedinombinationwiththeothertextualresouressuhasslides,notes andotherdouments.
The reason of fousing on auditory and textual modality instead of visual modality is intrinsi on the
natureofpresentations. Unlikeotherdomainssuh asmovieornews,invideopresentationstheimages inthe
keyframesare moreorless still, usually the speakerand part of his/herpresentationis aptured. The sene
almost neverhanges, the transitions being related to ahangeof fous(from slide to blakboard and bak)
orto thehange of slide. So, in e-Letures, eventhough lowlevelfeature reognitionsuh asteahergesture
andfaialprosodymightgiveinformationaboutimportaneofertainpassages,wedeidednottoattakthis
issuebeausewebelieveitwouldonlybringus aminor addedvaluein omparisonto theknowledgeweould
retrieveexploringtheauditoryandtextualmodality.
Duetotheseonsiderations,wefousourresearhworkonmakingbetteruseofspeehandtextualontent.
Furthermore,relatingtheextratedspeehandtextualontentwiththerightdomainknowledgeouldprovide
anothermodetotaklethesemantigapallowingmoreeetivelassiationandsearhesonthevideoontent.
Ingure4.1wegiveagraphialillustrationoftheproessofextratingsemantiannotationfrommultimedia
ontent. As input the system reeives a e-Leture presentation and based on the Wikipedia knowledge it
automatiallygeneratessomedesriptiveslabelsforthemultimediaontent.
In the next paragraph we will desribe in details how this proess of automati semanti disovering is
takingplae.
Theexplanation of ourmethod an besplit into twoparts. In therst partdealswith theextration of
ontentfrommultimedialeturematerialswithoutanyregardsaboutsemantis;whileintheseondpartwego
indepth inthepassageswhihinvolvedisoveringthesemantisbehindtheontentpreviouslyextratedfrom
themedia.
So, in orderto disoverthe semantipresentin aorpuswersthave toextrat andidentify termsfrom
it. Onewehavethelist ofthe wordsontainedin eahunit ofthe olletion,weanlink them throughthe
relationshipswewilldeterminethroughWikipedia. InpartiulartherstpartisaboutInformationExtration
Fig.4.1. GeneralOverviewoftheapproah
ThetwostepsareindependentinthesensethatInformationExtrationanbearriedoutindierentways
whileour Wikipediamodule ouldstill beusedto nd relationsbetweenterms. Wegiveherean explanation
oftherstpartonlyforthesakeofontextualization.
4.1. Information Extration. Inthissetionwewillextratandmodelthemultimediaontentthough
theanalysisandombinationofthetextualandauditorymodality. Usinganout-of-the-boxspeehreognition
toolwetranslatevideospeehintotext,andweombineitwiththetextextratedfrompresentationslidesand
othertextualinformationsoures.
Seondly, we extrats terms for the resulting textual resoures whih represent the entire ontent of the
multimedia material. We performed Information Extration(IE) by using Luene (http://luene.apahe.
org/), a state of the art tool whih provides Java-based indexing and searh tehnology using a statistial
approah.
Luenehadbeenusedintheprojetasasearhengineforqueryinganunstruturede-Learningrepository,
butsineitalsoprovidesbasiAPIsforanalyzingtext,weexploitedLuenealsoforextratinginformationfrom
ourorpus. Ingeneraltermextrationtoolsusingastatistialapproahbasially lookforrepeatedsequenes
oflexialitems.
Consequently we store the extrated information in a Luene index that we later use for information
retrievalandfor extratingthemostimportanttermsoutoftheentire e-Letureontent. Wealso exploreda
linguistiapproahbasedonNaturalLanguageProessing(NLP)usingotherstateofthearttoolsin thearea
suhasGATE(http://gate.a.uk/)andIBMUIMA (http://www.researh.ibm.om/UIMA/[18℄), butthe
approah wasnotsuited for ourusease sineit islanguageand grammar dependent. Infat,in e-Learning
thematerialanbeindierentlanguages,andsometimesmorethanonelanguageanbeombinedinthesame
event. Forinstane, in some asespresentationslides are written in English while thespeeh is deliveredin
another language(e.g. in Italian). As aonsequene, the work neededto adapt alinguisti approah to our
needswasexessive.
Moreover, story tellingdoesnot play an importantrole in e-Learningatleast not in the disiplines we
onsideredand this makes it diult to loate and lassify atomi elements in text into predened
ate-gories for Entity Reognition. For these reasons we hose a statistial approah and we alulated a term
In order to alulate the term vetor we had to store our multimedia material into an index. Many
doumentsanreferto thesameevent. Forinstane, wehaveat leasttwomain information souresforeah
event: presentationslidesandvideotransript.
Followingourmultimodalview,wemodeledtheentireeventsmultimediamaterialintoasingledoument
inaindex. InthiswaythetermvetoralulatedthroughLueneisfatual.
Forimprovingtheperformaneofthistaskweareurrentlyworkingontheindexingphraseandinpartiular
in the pre-proessing task (e.g. leaning the text from Italian or English stop-words and applying dierent
language stemmers as a lter, building ategorizer for improving the quality of the raking of the extrated
letureterms.).
4.2. Semanti Extration using Wikipedia. Inthis setion we explainhowweenhane information
retrievalbasedonthereognitionofthemostimportanttopis andtherelationsbetweenthemin theontent
ofmultimedialeture material.
Understandingwhatamediaisaboutbeforeenteringinitsdetailsofthesearhresults,whihwouldmean
wathingavideopresentationorreadingtherelatedmaterial,isoneofthemailgoalintheareaofMultimedia
Information Retrieval,andit ouldbeveryusefulwhen theamountof themultimedia materialgrowsinsize.
Having awayfor ategorizingor understanding themain oneptsin theontentwill help in managinglarge
multimediarepositories.
Furthermore,e-Learningusers(typiallystudents)donothavearihunderstandingofthedomainorofhow
onetopi isonnetedto others. For this reasonatool, whih hasthegoalofenablingaess to information,
should givean overview of the material ontentfor helping end users to ahievemore eetive searhes and
aquirethe neededknowledge. Forexample,auserlooking forthetermColletion in aJavaprogramming
lassmustndoutaboutthe dierenttypesofColletionsuh asHashMap, Mapand Set sinethese
terms an also be found in the leture material, and they all mean Colletion. Understanding relationships
betweentermsin ourorpuspermitsalsousto automatially disovertheimportanttopisofaneventwhih
anbeusedforunsupervisedlassiationofthematerial.
Our startingpoint forthe seond phaseof the semantidisoveryproess isthe list of termswhih were
extrated from multimedia material (video, slides, and douments) during the Information Extrationphase
desribedin theprevioussetion.
InWikipedia, welook upthemostimportantextrated termsfrom ourorpus. Thegoalofthis phaseis
to ndaWikipediadenition pageforeveryimportanttermand totrytoextrat relationstoothertermsby
examiningthehypertextuallinksinthepage. Thisisdonebyproessinglinksinthepage. Therefore,theterm
ofinterestisfoundinWikipedia,andallthelinks initspageareanalyzed.
Pre-requirementsfordesribingtheproessofextratingsemantisfromtheWikipedia,arethefourabstrat
oneptdesribedbelowthatwewillbementionedagainintheseondpartofthesetionforgivingadesription
oftheproessitself.
1. Wikipedialookup.
For eah extrated term we searh for pagesin Wikipediawhih ontain theterm in their name. In
Wikipedia every page is named by astring omposed of topi name and topi domain. After that,
we olletthelinks foreverypage. Thesearh is made ona loal opyof the English versionof the
Wikipedia database,but weould alsoreah thesameresult bydownloadingand parsingWikipedia
Webpages. Wehosetomaintainaopyofthedatabasetoinreasethespeedofthetask.
2. Stronglinkdenition.
We dene a link to be strong if the page it points to has a link bak to the starting page. For
instane,Romeand Italy arestrongly linkedsinethe pageon Romesaysthat it is theapital
of Italy, while thepageon Italyreports that Romeis theapital ofthe state. A minortown loated
in Italywill insteadhaveaweaklink with Italy,sinein its pageit willbedelared that thetown
is in Italy but in the page for Italythe minor town will most likelynot be mentioned. In our ase,
stronglinksareandidatesfortopisrelatedtothesearhedterm,andtheywillbeusedforgivinguser
suggestionsin queryexpansionandintheproessofsummarygenerationofWikipediadenitions.
3. Domaindisambiguation.
A word anhave multiple Wikipediadenitions beauseit anassumedierent meanings(senses)in
to evaluate theaordaneof thesemantiexpressed in thedisambiguatedtermswith theone of the
eventandwendoutthatthisistrueforthemajorityof theases.
4. AnnotationthroughWikipediadenition summarization.
Inthislaststepweusetheextratedstronglinksforeveryimportantwordofaneventtoautomatially
generate a summary of the word denition in Wikipedia.The summary is generated taking all the
sentenesfrom theWikipediadenitionpagein whih astronglink ispresent;usuallyftyperentof
theontentoforiginal denitionis seleted. Thesummaryis thenusedforexpressing themeaningof
theimportantterm. Inotherwords,weannotate theleturethroughWikipediaterminology,and for
eahtermwekeepabriefdenition.
Agraphialrepresentationoftheproessisgivenin gure4.2.
Fig.4.2. Semantiannotationsgenerationproess
Thegureshows thatby givinge-Leturesasinputand throughthefourstepsof thesemantidisovery
approah we an enrih e-Leture with semanti metadata. It follows a desription of the semanti
disov-eryproess omposed of four logialsteps: Wikipedia denition lookup, Domain disambiguation,Strong link
Analyses,andSummarization.
Suppose one of the terms extrated from the e-Leture material is Colletion, whih is in the list of
theextrated keywords. Considerasimplied list of extratedkeywords(notethe preseneof wordsin more
thanone language!) follows: Elemento, Map,Tipo,Objet, Method, Interfae, omputersiene, Colletion,
Oggetto.
TherststepofthealgorithmistosearhforallthepageswhihontainthetermColletionintheirname.
A searh in the Wikipediadatabasewill nd arelativelylargenumber of pagesthat satisfy thisrequirement
dueto thedierentmeaningsthewordanhave. Consequentlyweretrievethelinksofeveryfoundpage. We
thenusethelinks in thedisambiguationstep. Forinstane, in theaseofhavingatermnamed Colletion
intheleturematerial, aWikipediaqueryforthewordColletionreturnstheWikipediadisambiguationpage
Colletion,whih pointsto otherpagessuhasColletion(horse),Colletion(museum),Colletion(JoeSample
album),Colletion(ageny),Colletion(omputing),Colletionlass.
forautomatisummarizationoftheontentofWikipediaentries. Sointhisstepweextratthestronglinksfrom
alltheWikipediapagesseletedduringtherststep. Forexamplethestronglinks forColletion(omputing)
whih isone ofthe pages seletedduring the rststep, are: objet-oriented, lass, map, tree, set, array, list.
WedothesamewithallthepageslistedintheWikipediadisambiguationpageforolletion.
Thethirdstepresolvesdomain ambiguity. WeautomatiallyidentifytherightWikipediadenitionbased
onthedomaindenedbythemultimedia. Soweseletonepageamongalltheonesweretrievedduringstepone
whihisinonordanewiththedomainofinterest. Thedisambiguationfuntiononsidersforeveryandidate
pageitsstronglinks givenbysteptwo. Inpartiularitlooksfororrespondenesamongthelinksnamesand
the keywords extrated from the orpus. Thepage whih has the largestnumberof links in orrespondene
withtheorpustermswill beonsidered tobetheorretoneanditwill beused asthedisambiguatedterm.
Supposing that Colletion(omputing) is the Wikipedia artileis thepage that responds to this requirement
then everytime Colletion will be mention in the leture material it will be assoiated with the meaning of
theWikipediaartileColletion(omputing). Thetermthathasbeendisambiguatedhasthesamemeaningin
Wikipediaandin theorpus. Theresultofthisstepistheidentiationofthedisambiguatedtermswiththeir
links. InotherwordsthisstepomparesthestronglinksofeveryandidateWikipediadenitionwiththeterm
vetoroftheletureinexam.
Duringthelast stepwereateasummary forthemostimportantwordsinthe leture. Eahtermin the
leturehas aorrespondingWikipediadenition andbasedontherankingin thetermvetorweanidentify
the n most importantterms in the e-Leture. The summary reation uses the extrated strong links of the
mostimportantterms. Foreveryoneofthem,wedownloadandparsetheorrespondentWikipedia. Fromthe
Wikipediapageweseletonlythesenteneswhihontainastronglinkand thetermitself. Theombination
ofextratedsentenespermitstogenerateareasonablywellwrittenartilesummarization.
5. Appliations. Inthis setionwedesribe someappliationswhere our approah anbe used. Many
other appliations are under onsiderations. In partiular we applied the semanti disoveryapproah into
NEEDLE [8℄- Next gEneration sEarh engine for Digital LibrariEs -. NEEDLE is an e-learning appliation
whih aimsat indexing, searhing and presentingstrutured andunstrutured multimedia data. Thesystem
providesawaytosearhe-learningmaterialsthroughaweb-basedsearhinterfae.
Thee-Learningmaterialsonsistofvideoleturesandorrespondingaudiotraks,PowerPointpresentation
slideset. Theappliation'smainobjetiveistopresentthestruturedandunstruturedmultimediae-Learning
materials. TheusersouldquerytheNEEDLE systemtosearhformaterialsoftheirinterest.
Thedatai.e. videoleturesandslides,forNEEDLEomefromtheLODE[5℄system. LODEisweb-based
appliation for presenting thevideo letures synhronizedwith presentationslides. The audioontentof the
videoleturesfromtheLODE systemistransribedusingspeehreognitionandspeehproessingtools. The
textontentofthetransriptionandPowerPointslidesareindexedandsearhedusingNEEDLE system.
Most of the ommerial searh engines only oer text based searh and few also provide image searh.
However, there is still a need for searhing video, audio, graphis et. Commerial video hosting sites like
YouTubethatoersearhforvideoatuallyperformsthesearhonlyonthemeta-data(textontentdesribing
thevideo) attahed with thevideo. Theydo notsearhthe video/audioontent. We oertextual searh on
audio ontent using the transript in ombination with the ontent of the douments whih ome with the
videosuh aspresentation slides; moreoverweenanhed thesearhtask with searh suggestionbased onthe
identiationofrelationshipbetweentopisinWikipediaandweautomatiallyextratalsothroughWikipedia
labelsfordesribingthemostimportleture'stopis.
Weansummarizewiththefollowingpointsthefeatures whereourapproahouldontribute:
•
Searh SuggestionandQueryExpansionWikipediais usedfornding topisrelatedto thesearhedone. In oursearhuserinterfaeweshow
thehitsforthesearhedstringandabunhoflinkstosomerelatedtopiswhihhaveaorrespondene
inourrepository. Alikononeofthelinkwillinitiateasearhfortheourrenesofthelinktermin
thelearningmaterial. This isdonebyviewing allthestronglinks retrievedthroughWikipediawhih
termappearalsointheeventmaterial,inthiswayweansuggestdierentsearhtermsortopisthat
areonnetedtotherstsearhedone.
•
AutomatiAnnotationIn this way there are no more ambiguities in the meaning of a term used for annotation. Another
advantage of the strong link identiation in ombination with the term vetor extrated for every
eventisthepossibilitytoautomatially desribethemostimportantoneptsoftheevent.
•
AutomatiSummarizationThesemantidisoveryapproahdesribed in theprevioussetion broughtus tothe individuationof
thestronglinksforeahtopi. Basedonthemweangenerateforeaheventannotation(topi)abrief
summarizationof the desriptionof the topi in Wikipedia. A lik on oneof theevent annotation
will display thesummary plusthe retrievedhits for that term. In oursearh userinterfaefor eah
event(leture,seminar, meeting) we show thesix most importantwordsand the relatedsummarized
Wikipediadenitions.
Ingure 5.1 the possible features derived by theimplementation of the semanti disoveryapproah
de-sribedaboveareshown. Figure5.1isasreenshotofNEEDLE whereeveryletureitisrstsummarizedby
meansofalistofimportanttopis. Sotheuserlookingat theimportanttopislistanunderstandifaleture
inthehitsisrelevantornotforeahsearhandinases/heangoindepthlookingatthedetailshitsinsidethe
letureitself. Everyhitintheletureisomposedofabrieftextualdesriptionanfourpresentationmodalities.
Theresultanbeanalyzedwathingpartofthevideowherethehithavebeenfound,listeningonlytotheaudio,
onlywathingtheassoiatedslideorhavingaombineview,wherevideoandslidearetime-synhronized.
Fig.5.1.Anappliationof semantisextrationthroughWikipedia
Ontherightmostpartofgure5.1apop-upwindowforwithadesriptionofoneoftheimportantterms
oftheletureanbeviewedbylikingontheterm. ThedesriptionisatuallythesummaryoftheWikipedia
artile whih refers to the term itself. In this way there is no ambiguity with the meaning of a term used
for desribing the multimedia ontent. Below theinput eld designed for running new searhes there is the
implementation of the related topis feature. Based onthe last usersearh, the system advies the user for
6. Future diretions. One of the ativities we plan for the near future is related to extending the
Wikipedia module to support various languages. The multilanguagesupport onsists in reognizing relations
betweentermsin theorpuswhiharenotin English. Asarststep, well look atthelinks to theother
in-stanesofWikipediaindierentlanguages. Inmostases,pagesintheWikipediainstaneinonelanguagehave
linkstopagesinmanyotherWikipediainstanesinotherlanguages.Sinetheselinkswerereatedmanuallyby
thepageauthors,inmostasesthereisnoambiguityinthetranslation. Inasealinktothetargetlanguageof
interestisnotpresent,weanresorttofreelyavailable,albeitlesstrustableexternalsouresfortranslatingfrom
andto English. TheWikipediaproessdesribedintheprevioussetionwillnothangebut writinglanguage
dependentproessingmodulessuhaslanguagespeistemmersshouldbeaddedtoenabletheomparisonof
therelatedWikipediaontentfoundinEnglishwiththetermsontainedin themultimediaontentrepository.
Consequentlywehavesheduledanevaluationofthepresentedapproahforannotatingalargeamountoftext
resouresandauserbasedevaluationtoassessiftheintrodutionofsemantimultimediainformationretrieval
isatuallybringinganadvantagetothestudent. Wewillarryoutastudentsperformaneevaluationonsome
topis presented in thee-Learning repository andwewill ompare theresultswith theones wegatheredlast
yearusingatextbasedsearhthatwasnotsemantiallyenhaned[9℄.
7. Conlusion. Inthis paperwedesribed an approah to semantially annotate the ontent of an
un-struturedmultimediarepository. Theannotationhasbeendoneombiningthetermsextratedfromtheorpus
withlexiographirelationshipsfromWikipedia. Wikipediahasbeenusedasanalternativetoontologies. The
ontent annotatedin this waypermitsto keeptrak ofthe relationsbetween annotations. Theapproahhas
beenusedforgivingsearh suggestionsin multimedia informationretrieval,in multimedia annotationandfor
givingabriefdesriptionof thetopis of themultimedia event. Ourapproahis domain independent, and it
ould in theory also be applied to dierent use ases where there is a need for lustering or annotationof a
orpus.
REFERENCES
[1℄ K.Aberer,K.-S.Choi,N.F.Noy,D.Allemang,K.-I.Lee,L.J.B.Nixon,J.Golbek,P.Mika,D.Maynard,
R.Mizoguhi,G.Shreiber,andP.Cudré-Mauroux,eds.,DBpedia:ANuleusforaWebofOpenData,vol.4825
ofLetureNotesinComputerSiene,Springer,2007.
[2℄ S. Auer and J. Lehmann, What have innsbruk and leipzig in ommon? extrating semantis from wiki ontent., in
Proeedingsofthe4thEuropeanSemantiWebConferene,ESWC2007,2007.
[3℄ M.Bertini,A.D.Bimbo,andC.Torniai,Enhanedontologiesforvideoannotationandretrieval,inMIR'05:Proeedings
ofthe7thACMSIGMMinternationalworkshoponMultimediainformationretrieval,NewYork,NY,USA,2005,ACM,
pp.8996.
[4℄ K.Bontheva, D.Maynard,H.Cunningham, andH.Saggion,Usinghuman language tehnology forautomati
an-notationandindexingofdigitallibraryontent.,in6thEuropeanConfereneonResearhandAdvanedTehnologyfor
DigitalLibraries(ECDL'2002),Rome,September2002.
[5℄ M.DolzaniandM.Ronhetti,Videostreamingovertheinternettosupportlearning:thelodesystem,inWITTransations
onInformatisandCommuniationTehnologies,vol.34,2005,pp.6165.
[6℄ M.Dowman, V.Tablan, H.Cunningham, andB. Popov,Web-assisted annotation, semantiindexingand searh of
televisionandradionews,inProeedingsofthe14thInternationalWorldWideWebConferene,Chiba,Japan,2005.
[7℄ A.Ebersbah,M.Glaser,andR.Heigl,Wiki: WebCollaboration,Springer,November2005.
[8℄ A.Fogarolli, G.Riardi,andM.Ronhetti,Searhinginformationina olletionofvideo-letures,inProeedings
ofWorldConfereneonEduationalMultimedia,HypermediaandTeleommuniations2007,Vanouver,Canada,June
2007,AACE,pp.14501459.
[9℄ A. Fogarolli andM. Ronhetti,Casestudy: Evaluationof a tool for searhing insidea olletion of multimodal
e-letures,inProeedings ofWorld Confereneon Eduational Multimedia,Hypermediaand Teleommuniations 2007,
Vanouver,Canada,June2007,AACE,pp.38933900.
[10℄ M. Hepp, K. Siorpaes, and D. Bahlehner, Harvesting wiki onsensus: Using wikipedia entries as voabulary for
knowledgemanagement.,InternetComputing,11(2007),pp.5465.
[11℄ L.Obrst,Ontologiesforsemantiallyinteroperablesystems,inCIKM'03:Proeedingsofthetwelfthinternationalonferene
onInformationandknowledgemanagement,NewYork,NY,USA,2003,ACMPress,pp.366369.
[12℄ S.PonzettoandM.Strube,Derivinga largesaletaxonomyfromwikipedia,inProeedingsofthe22ndNational
Con-fereneonArtiialIntelligene(AAAI-07),Vanouver,B.C.,July2007.
[13℄ J.Porter,Folksonomies: Auser-drivenapproahtoorganizingontent,UserInterfaeEngineering,(2005).
[14℄ F.Suhanek,G.Kasnei,andG.Weikum,Yago: Alargeontologyfromwikipediaandwordnet,ResearhReport
MPI-I-2007-5-003,Max-Plank-InstitutfürInformatik,Stuhlsatzenhausweg85,66123Saarbrüken,Germany,2007.
[15℄ J.T.Tennis,Soialtaggingandthenextstepsforindexing,in17thSIG/CRClassiationResearhWorkshop,2006.
[17℄ N.WeberandP.Buitelaar,Web-basedontologylearningwithisolde,inPro.ofISWC2006WorkshoponWebContent
MiningwithHumanLanguageTehnologies,2006.
[18℄ C.WeltyandJ.Murdok,Towardsknowledgeaquisitionfrominformationextration,TheSemantiWeb-ISWC2006,
(2006),pp.709722.
[19℄ F.WuandD.Weld,Autonomouslysemantifyingwikipedia,inACMSixteenthConfereneonInformationandKnowledge
Management(CIKM-07),Lisbon,Portugal,November2007.
Editedby: DominikFlejter,TomaszKazmarek,MarekKowalkiewiz
Reeived: November23rd,2007
Aepted: January15th,2008
