Whenmovingtoanenterpriseintegrationstrategy,largeorganizationsoftenface a proliferation ofsoftwaretoolsthatarebuilttosolveidenticalproblems.Fewof these toolsworktogether,muchlessworkacrossproblemdomains toprovidean integratedsolution.
Data profiling,datamodeling, datatransformation,dataquality,and business intelligence toolsplayakeyroleindataintegration.Integrationcanbecomea mature, manageableprocess ifthesetoolsareenabled toworkacrossproblem domains.
Theconsequences oftheinabilitytomanagemetadataaremanyandsevere:
v Changesthataremadetosourcesystemsaredifficult tomanageandcannot matchthepace ofbusinesschange.
v Datacannotbeanalyzedacrossdepartmentsandprocesses.
v
Metadatacannotbeshared amongproductswithoutmanuallyretypingthe metadata.
v Withoutbusiness-leveldefinitions,metadatacannotprovidecontextfor information.
v Documentationisout-of-dateorincomplete, hamperingchange management andmakingit hardertotrainnewusers.
v Effortstoestablishaneffectivedatastewardshipprogramfailbecauseofa lack ofstandardizationandfamiliaritywiththedata.
v Establishinganaudittrailforintegrationinitiativesisvirtuallyimpossible.
Themetadataservices componentsofIBMInformationServercreatea fully integratedsuite,eliminatingtheneedtomanuallytransportmetadatabetween applicationsand providea standalonemetadatamanagementapplication.
Metadata services introduction
Metadataservicesare partoftheplatformonwhichIBMInformationServeris built.Byusingmetadataservices,youcanaccessdataandachievedataintegration tasks suchasanalysis,modeling, cleansing,and transformation.
Themajor metadataservices componentsofIBMInformationServerare WebSphereBusinessGlossary, WebSphereMetadataServer,and WebSphere MetaBrokers andbridges
WebSphere Business Glossary
WebSphereBusinessGlossary isa Web-basedapplication thatprovidesa
business-orientedview intothedataintegrationenvironment.ByusingWebSphere Business Glossary,youcanviewandupdatebusinessdescriptionsandaccess technicalmetadata.
Metadataisbestmanagedbythosewhounderstandthemeaningand importance of theinformationassetstothebusiness.Designedforcollaborativeauthoring,
WebSphereBusinessGlossary givesuserstheabilitytoshareinsightsand experiences aboutdata.Itprovides userswith thefollowinginformationabout dataresources:
v Businessmeaninganddescriptionsof data v Stewardshipofdataandprocesses
v Standardbusinesshierarchies v
Approvedterms
WebSphereBusinessGlossary isorganized andsearchable accordingtothe semantics thataredefinedbya controlledvocabulary,whichyoucancreateby usingtheWebconsole.
WebSphere Metadata Server
WebSphereMetadataServerprovidesa varietyof servicestoothercomponentsof IBM InformationServer:
WebSphereMetadataServerprovidesa commonrepositorywithfacilitiesthatare capable ofsourcing,sharing,storing,andreconcilinga comprehensivespectrumof metadataincludingbusinessmetadataandtechnicalmetadata.
Businessmetadata
Business metadataprovides businesscontextfor informationtechnology assetsandaddsbusinessmeaningtotheartifactsthatarecreatedand managedbyotherITapplications.Businessmetadataincludescontrolled vocabularies, taxonomies,stewardship,examples,and businessdefinitions.
Technicalmetadata
Technicalmetadataprovidesdetailsaboutsourceandtargetsystems, their tableand fieldstructures,attributes,derivations,anddependencies.
Technicalmetadataalsoincludesdetailsaboutprofiling,quality,andETL processes,projects,andusers.
WebSphere MetaBrokers and bridges
WebSphereMetaBrokersand bridgesprovidesemanticmodelmappingtechnology thatallows metadatatobe sharedamongapplicationsforallproductsthatare usedinthedataintegrationlifecycle:
v Datamodeling orcasetools v
Businessintelligenceapplications v Datamartsanddatawarehouses v Enterpriseapplications
v Dataintegrationtools
Customerswhousethesecomponentscanestablishcommondatadefinitions acrossbusinessandITfunctions.
v Driveconsistencythroughoutthedataintegrationlifecycle v Deliverbusiness-orientedandIT-orientedreporting
v Provideenterprisevisibilityforchangemanagement
v Easilyextendtonew,existing,andhomegrownmetadatasources
Scenarios for metadata management
AcomprehensivemetadatamanagementcapabilityprovidesusersofIBM InformationServer withacommon waytodealwithdescriptive information surrounding theuseofdata.Thefollowingscenariosdescribeusesofthis capability.
Web-based education:Profilingyourcustomer
AWeb-based,for-profiteducationproviderneededtoretainmorestudents.
Business managersneededtoanalyzethestudentlifecyclefromapplication tograduation anddirectrecruitingeffortsatindividualswiththebest chance ofsuccess.
Tomeet thisbusinessimperative,thecompanydesignedanddelivereda businessintelligencesolutionthatusesadatawarehousethatcontains a single viewofstudentinformationthatispopulatedfromoperational systems. TheITorganizationusesWebSphereMetadataServerto coordinate metadatathroughouttheproject.Other toolsthatwereused includedEmbarcaderoERStudiofordatamodeling andBrioforBusiness Intelligence.
Theoverallprojecttimewasreducedbyprovidingmetadataconsistency and accuracyacrosseverytool. Thebusinessusersnowhavetrustworthy metadataabouttheinformationintheirBrioreports.WebSphereBusiness Glossary providedbusinessdefinitionstoWebSphereMetadataServer.The netresultismoreconfidentdecision-makingaboutstudentsandbetter student-retentioninitiatives.
Financial Services:Measuringlevelsofservice
Thedatawarehousing divisionofamajorfinancial servicesprovider neededtoprovideinternalcustomerswithcriticalenterprise-widedata aboutlevelsofservicethatarespecified bysignedservicelevelagreements (SLAs).Thedatawarehousinggroup alsoneededtoprovidebusiness definitionsof eachfield,includingmetricsthatdetailedactualversus promisedlevelsofservice.
TheorganizationusesIBMInformationServertocreatean enterprisedata warehouseand datamartstosatisfyeachSLA. Thedivisionusedmetadata services withinWebSphereInformationAnalyzer,WebSphereQualityStage, and WebSphereDataStagetocollaborateina multiuserenvironment.The datawarehousing groupwasalsoable toprovideHTMLreportsthat outlinedthestatisticsthatare associatedwiththeloadingofthedatamart tosatisfytheSLA.
Thedivisionmetitsservice-levelagreementsandwasable todemonstrate itscompliancetointernaldataconsumers.Additionally,endusersreceived importantbusinessdefinitions throughbusinessintelligencereports.
A closer look at metadata services in IBM Information Server
Metadataservicesencompass awiderangeoffunctionalitythatformsthecore infrastructureofIBM InformationServer andalsoincludessomeseparately packagedcapabilities.
WebSphere Business Glossary
Managingbusinessmetadataeffectivelycanensurethatthesamedata“language”
appliesthroughouttheorganization.WebSphereBusiness Glossarygivesbusiness usersthetoolstheyneedtoauthorand ownbusinessmetadata.
For example,onedepartmentrefersto “revenues,”anotherto“sales.”Arethey talking aboutthesame activity?Onesubsidiaryunittalksabout“customers,”
anotherabout“users”or“clients.”Arethesedifferentclassifications ordifferent termsforthesameclassification?
WebSphereBusinessGlossary providesbusinessuserswith aWeb-based toolfor creating andmanagingstandarddefinitions ofbusinessconcepts, calledacontrolled vocabulary.Italso simplifiesthebuildingof abusiness-orientedclassification system andthecollaborativeauthoringofbusinessmetadata.
The toolsimplifies thetaskofmanaging, browsing,andcustomizingthebroad varietyofmetadatathatisstoredintherepositoryofWebSphereMetadataServer, metadatathatincludesdetailsabouttables, columns,models,schemas,operations, and othercomponentsofthedataintegrationprocess.
The tooldividesmetadataintocategories,eachofwhichcontainsterms.Youcan usetermsto classifyotherobjectsinthemetadatarepositorybasedontheneedsof your business.Youcanalsodesignateusersorgroupsasstewardsformetadata objects.
WebSphereBusinessGlossary helpsbusinessuserswiththefollowingtasks:
Developinga commonvocabularybetweenbusinessandtechnology
Acommonvocabularyallows multipleusersofdatatosharea common view ofthemeaningofdata.Users canassigncategories andtermstodata thatare meaningfulinabusinesscontext,andcreatea hierarchyof
categories foreaseofbrowsing.
Providing datagovernanceandstewardship
Data assuranceprogramsassignresponsibility tobusinessusers(data stewards) forthemanagementofdatathrough itslifecycle.
Findingbusinessinformation thatisderivedfrommetadata
Metadatahelpsbusinessuserstounderstandthemeaningofthedata,its currency,itslineage,and whoisresponsiblefordefiningandproducing
Figure11.WebSphereBusinessGlossaryuserinterface
thedata.Ifabusinessuserwantstoknowthedefinitionof atermsuchas
“corporateprice,”theglossarywillprovidethisinsight.
Accessing metadatawithoutcomplicatedtoolingandquerying
Metadataobjectscanbearrangedina hierarchicalfashiontosimplify browsing ofthedataobjects.
Providing collaborativeenrichmentofbusinessmetadata
Maintenance ofbusinessmetadataisanongoingprocessinwhich automated andmanualdatainputs evolve.Multiplebusinessuserscan collaboratetoaddnotes,annotations,categories,andsynonymstoenrich businessmetadata.
For example,multiplesystemsmaymaintaintablesofcustomer information, howeverthebusinessmayuncoverarequirement fortheconcept of“high-value”
customers. Thebusinessneedsaway todefinewhatahighvalue customeris, and how torecognizethem(forexample,a high-valuecustomer isa customerwith combinedaccountbalancesover$10,000).WebSphereBusinessGlossaryprovides a toolforrecordingthesedefinitions,andrelatingbusinessconceptstogether into taxonomies.Thisrecordsthebusinessrequirementsinthesame metadata foundationthattheprofilingandanalysisprocess uses.
WebSphere Business Glossary tasks
Major tasksinWebSphereBusinessGlossaryinclude creatingcategoriesand terms, browsing andsearching,enabling datastewardship,andannotatingdatafor collaboration.
WebSphereBusinessGlossary isa browser-basedapplicationthatyouaccessby usingMicrosoft InternetExplorer.
Enabling data stewardship
Data stewardshipisthemanagementofdatathroughoutitslifecycle.Stewardship includesmaking thedataavailable toallthosewho areauthorizedtoaccessit.It also includestheefficientmanagement andintegrationwithrelateddata.Perhaps mostimportantly,stewardshipincludestheresponsibility toensurethatdatais properlydefined,and thatallusersofthedataclearly understanditsmeaning.
WebSphereBusinessGlossary supportstheconceptof datastewardshipandhelps yousetand retrievestewardshipinformationforalldataassets.
Administrators candesignatea userorgroupasa steward.Administratorsand authorscanthenspecifythatthestewardisresponsibleforoneormoremetadata objects.Whenyouview thebrowsepageforan objectthathasasteward,youcan linktocontact informationforthesteward.
Creating categories and terms
AlthoughyoucanuseseveralmethodstofindmetadatainWebSphereMetadata Server, businessusersoftenfindsearchingdatabycategoryisthebeststrategy.
Data mustbe organizedintomeaningful taxonomiestoaidthenavigationof a businessglossarybycategory.
Figure12onpage22showstheCreateCategoryfunction inWebSphereBusiness Glossary. Youcreatea businessclassificationsystem ortaxonomythatacts asthe
hierarchical browsingstructure oftheglossary Website.Youcanalsoimport structure fromothertoolsorspreadsheets.
Atermisa wordorphrasethatcanbe usedtoclassifyand groupobjectsinthe metadatarepository.For example,youmightusetheterm“SouthAmerica Sales”
toclassifysomeofthetablesandcolumnsinthemetadatarepository,andtheterm
“AsianSales”toclassifyothertablesand columns.
Whenyoucreateor edita term,youcanspecifypropertiesand relationships amongterms, includingsynonymsand relatedterms.Youcanalsospecifyparent categories togroupsimilartermsandcandesignatestewards whohavethe responsibility formaintainingterms. Customattributesenableadministratorsto defineanynumber ofnewattributestobe appliedtoterms, categories,orboth.
Annotating data for collaboration
Whiledatastewardsare responsibleforspecific typesof data,creating abusiness glossary isa collaborativeeffortthatrequiressubjectmatterexperts fromdifferent partsoftheenterprise.WebSphereBusinessGlossary providestoolsforsubject matterexpertsandothers toannotateexistingdatadefinitions,editdescriptions, and assigndataobjecttocategories.
Theseannotations,ornotes,helpbusinessusersshare insightsaboutthe
informationassetsoftheenterprise.For example,an analystmight discoverthata database columnforcustomerinformationalso containsshippinginformationthat doesnotbelonginthecolumn.Theanalystcould sharethatinformationbyusing theNotes®feature. Noteshelpyoucaptureideas intheform ofunstructured metadata. Thisinformationmight otherwisebe unknowntoalargeportionofthe enterprise.
Figure12.Creatinganewcategory
Browsing the Business Glossary
Youcanstart browsingtheglossarystructurefromtheOverviewpage,which displays thetop-levelcategoriesthattheglossaryadministratorhasdesignatedas mostimportantfornavigationinthemetadatarepository.
Thebrowse bycategoryfunctionenablesdatastewardstofind descriptionsrelated totype ofdataeventhoughtheymaynotknowtheexactnameofthedataitems inquestion.
Whenyouselectanobject, itsbrowsepageisdisplayedontheBrowseGlossary tab, whichliststheobject’sname,class,stewardandotherimportantproperties.
Youcaninspectitsattributes,browseitsrelationshipstootherobjects,and send feedback totheadministrator.Administrators andauthorscanaddandeditnotes abouttheobject.
WebSphere Metadata Server
IBM InformationServer canoperateasa unifieddataintegrationplatformbecause of theshared capabilitiesof WebSphereMetadataServer.
Common repository
By storingallmetadatain asharedrepository,IBM InformationServer enables metadatatobe sharedactivelyacrossall tools.Therepositoryprovidesservices for twotypesofdata:
v Designmetadata, whichiscreatedasa partofthedevelopmentprocessand can beconfiguredtobe eitherprivateorshared bya teamof users.
v Operationalmetadata,whichiscreatedfromongoingintegrationactivity.This metadataismessage-orientedandtime-stampedtohelptrackthesequenceof events.
With asharedrepository,changes thataremadeinonepartof IBMInformation Server willbe automaticallyand instantlyvisiblethroughoutthesuite.The repositoryoffers thefollowingkeyfeatures:
Activeintegration
Applicationartifactsaredynamicallyintegrated acrosstools.
Multiuser development
Teamscancollaborateina sharedworkspace.
Thecommon repositoryisanIBM WebSphereJ2EEapplication.Therepository usesstandardrelationaldatabasetechnology(suchasDB2orOracle)for persistence. Thesedatabasesprovidebackup,administration,scalability, transactions,andconcurrentaccess.
Common model
MetadatafordataintegrationprojectscomesfrombothIBMInformationServer productsand vendorproducts.Therepositoryusesmetadatamodels (metamodels) todescribethemetadatafromthesesources.Metadatamodelsprovidea meansfor others tounderstandandshare metadatabetweenapplications.
Thecommon modelisthefoundationofIBMInformationServer.Metadata elementsthatare commontoallmetadatasourcesarediscovered andrepresented once, inaform andformatthatisaccessibletoallofthetools.Thecommonmodel
enablessharingandreuseof artifactsacrossIBMInformationServer.
Shared metadata services
WebSphereMetadataServerexposesa setofmetadatamanipulationandanalysis services foruseacrossIBMInformationServercomponents.Theseservicesenable metadatainterchange,integration,management,and analysis.Theyeliminatethe need forastandalone metadatamanagement productorrepositoryproductby activelymanagingmetadatainthebackground, andbyprovidingmetadata functionalityinthecontextofyour normaldailyactivities.
For example:
v AWebSphereDataStageuser wantsto understandthedependenciesbetween stagesinanETLjob.Byusingmetadataservices,shecanperforman impact analysisfromtheDesignerclientcanvas,never needingtoleavetheapplication foranotherinterface.
v Adataanalystwhoisworkingwith WebSphereInformationAnalyzercanadd businessterms,definitions,and notestodataunderanalysisforusebya data modelerorarchitect.
v
AWebSphereQualityStageuserneedstobetterunderstandthebusiness
semanticsthatareassociatedwitha datadomain.Byusingmetadataservices,he canaccessthebusinessdescriptionofthedomainandanyannotationsthatwere addedbybusinessusers.
v AWebSphereDataStagecomponentdeveloperwantstofinda functionthat performsa particulardataconversion.By usingmetadataservices,shecan performanadvancedsearchforthefunction.
WebSphereMetadataServeroffersthefollowingkeymetadataservices:
v Metadatainterchange v Impactanalysis v Integratedfind Metadatainterchange
WebSphereMetaBroker®and bridgesenableyoutoaccessand share metadatawiththebest-of-classtoolsformodeling,dataprofiling,data quality,ETL,OLAP, andbusinessintelligence.
Figure13onpage25showshowMetaBrokerswork.MetaBrokersconvert metadatafromoneformattoanotherbymappingtheelementstoa standardmodelcalledthehubmodel.Theselectedmetadataisthen importedandstored intherepository.Themetadataexchangeenables decomposition andrecompositionofmetadataintosimpleunitsof meaning.
IBM InformationServer nowsupportsmorethan20MetaBrokersand bridgestovarioustechnologiesandpartnerproducts.Youcanusemost MetaBrokers toimportmetadatafromaparticulartool, file,ordatabase intothemetadatarepositoryofWebSphereMetadataServer.
Table1 describesMetaBrokertypesandthedifferenttypesofmetadatathat youcanaccess.
Table1.MetaBrokertypes
TypeofMetaBroker Typeofmetadata
Designtool CAERwin,OracleDesigner,Rational®Data
ArchitectandtheUnifiedModeling Language(UML)
OLAPandbusinessintelligence CognosPowerPlay,IBMCubeViews™, ReportNet,BusinessObjects,andHyperion Operationalmetadata Metadatathatdescribesoperationalevents
suchasthetimeanddateofintegration processruns.
Impact analysis
Impact analysishelpsyoumanagetheeffectsofchanges todataby showingdependenciesamongobjects.Thistype ofanalysisextendsacross multiple tools,helpingyouassessthecostofchange.Forexample,a developercanpredicttheeffectsofa changetoatabledefinitionor businesslogic.
Figure14onpage26showstheWebSphereDataStageandQualityStage Designerbeingusedtoselectatable definitioncalled ProdDimfromthe metadatarepositorytoshowwhereuseddependencies.
METABROKER
External Tool
Metadata Interface
Decoder
Encoder
Mapper
Source (view) model Target (hub) model
Figure13.MetaBrokersconvertmetadatatoastandardmodelcalledthehubmodel.
TheImpactAnalysisPathViewerpresentsagraphical viewofthese relationships, asFigure15onpage27shows.
Figure14.UsingFindtoshowdependenciesforatabledefinitionintherepository
Thedependenciescanalsobe shownina textualview.Youcanalso runan impactanalysisreportthatcanbeviewedfromtheWebconsole.
Integratedfind
Metadataserviceshelp youlocateandretrieveobjectsfromtherepository byusingeitherthequickfindfeature ortheadvancedfindfeature.The quickfindfeature locatesanobjectbased onafullorpartialnameor description. Theadvancedfindfeaturelocates objectsbasedonthe followingattributes:
v
Type
v Creation data v
Lastmodified v Whereitisused
Figure15.ImpactAnalysisPathViewer
Information resources for metadata services
AvarietyofinformationresourcescanhelpyougetstartedwithIBMInformation Server’smetadataservices productmodules.
WebSphereBusinessGlossary
TheGettingStartedpane thatappearswhenyouclicktheGlossarytabof theIBMInformationServer consoledescribesthepurposeofthetaband
TheGettingStartedpane thatappearswhenyouclicktheGlossarytabof theIBMInformationServer consoledescribesthepurposeofthetaband