A Convergence in Application Architectures
and New Paradigms in Computing
SOA, Composite Applications and Cloud Computing
January 2009
Ravi Har Singh Khalsa
Adrian Reason
Mike Biere
Caryn Meyers
Andrea Greggo
Mac Devine
Visit us at Lotusphere
pedestal 216
Executive Summary
This paper describes four trends taking place in the software industry. These are all trends that tend to move away from monolithic software applications and toward a service-based approach to software use and delivery. Although these are four discrete trends, they are not happening in isolation, but are linked together, reinforcing one another in what is called a virtuous circle. The impact of these combined trends is forcing changes in existing business models and promises to change patterns of competitive dominance in the software industry. New paradigms are emerging which will reshape the software industry.
One such development has been the emergence of Cloud Computing as an important new IT segment. IBM has enthusiastically embraced the Cloud Computing model. In 2008 it set up a new Cloud Computing business group and is in the process of committing significant resources to developing cloud-based solutions and services.
It is important for customers to understand these trends and the new computing models that they will spawn. The days of the dedicated software application are numbered. In its place will emerge a new model of software delivery based on Service Oriented Architecture (SOA), composite applications and Software-as-a Service (SaaS). These technologies exist now. Adoption and mastery of these technologies now will help customers make the necessary transitions to new computing models in the near future.
It is important for customers to anticipate and plan for these changes so that they realize the transformative value of the new paradigms and not suffer their disruptive effects.
An Historic Convergence of Application Architectures
SOA, together with other powerful market forces, is driving a profound
transformation in application architectures and choice of software solutions. We see the development and convergence of four significant trends in the IT industry:
Changes in the nature of business intelligence
Changes in the form and function of desktop applications Changes in the ways we access data
Changes in how we collaborate using software
We will examine each of these trends and consider how the convergence of these forces is transforming the very nature of IT.
Trend 1: The transformation of Business Intelligence from stand-alone BI applications to embedded BI services.
In 2007 and 2008 we saw a major shift in the BI industry with major platform vendors buying or investing heavily in Business Intelligence products to support customer requirements for enterprise delivery of Business Intelligence. Until now, BI vendors have done well by building increasingly sophisticated suites of reporting and analysis tools that were primarily sold to senior management and financial managers. These BI suites are powerful, require extensive training, and are expensive. However, line of business users are now also asking for BI functionality, but not in the old model, rather in a way that is easy to use, affordable, and delivered in the context of their day-to-day work. Analysts refer to this as embedded BI or operational BI.
Dr. Wayne Eckerson Director of Research and Services at the Data Warehouse Institute states:
Rather than using stand-alone BI toolsets that require setup and training, business users will leverage embedded BI functionality that is an integral part of a larger application or package. Users will no longer shift software contexts when moving from operational processes to analytical ones. Currently, they must exit an operational application, and their train of thought and productivity, pull up a BI tool, find the appropriate report, analyze it, and then reenter the operational application and make appropriate updates based on the insights they gleaned in the BI tool. In the future, however, users will gain analytic insight within the context and flow of a single, process-driven application.
Trend 2: The transformation of desktop software delivery from dedicated applications to composite applications.
Traditionally, customers have purchased or developed software applications that have been designed to meet specific vertical or horizontal business requirements. As these applications have matured and grown, they have typically become more comprehensive and more powerful, providing more and more features and capabilities.
The dilemma for modern users is that they generally use only a small fraction of the features offered by any one application, and users rarely use just one application at any time. In fact, users typically switch back and forth between multiple applications to complete different parts of their work.
In order to complete a hypothetical activity, a knowledge worker might browse the web, search emails, locate and read relevant documents, study reports, consult colleagues, have a meeting, prepare a document or presentation, circulate it for feedback, and track the time spent on the activity. This could entail switching back and forth between as many as a dozen different applications. Clearly the structure and availability of dedicated desktop
applications has not adapted to these changing patterns of usage.
Composite applications promise a solution to this problem. Composite applications are built upon an application framework which contains a number of loosely coupled application components. The framework allows these components to work together as a unified whole.
Composite application components can be provided by the same vendors who now sell dedicated desktop applications. The difference with composite applications is that software vendors can now apply their expertise and proprietary technology in more localized and granular ways. Instead of building one monolithic application which dominates the desktop and includes every conceivable feature, the vendor can now build a set of discrete components, each of which provides a specific set of functions, but which all work together as a whole within the composite application framework.
Composite applications can be built by assembling components, often from different vendors, and then ‘wiring’ them together so that they work together in a meaningful way. The focus of the composite application becomes the user, not the software application. The structure and content of the composite application is defined by the role or activity context of the user. This context determines the components and connections required to build the composite solution. Subject matter experts, not dedicated programmers, should be able to build composite applications, and organizations should be able to provide personalized desktop interfaces to all users. The benefits to using composite applications are increased productivity and reduced software costs.
Trend 3: The transformation of data from an isolated resource to data as a service.
Through the application of Service Oriented Architecture we are seeing a transformation of data from a tedious and single-threaded process to information as a service.
Interfaces to data are now being packaged as web services, where a client application will access the service instead of the data source. Web services take care of querying the actual data sources (real time or in a data warehouse), transforming the data, and delivering the result to the client application in the format it can understand and use. Web services provide a layer of abstraction between the data and the consumers of the data. If a database is moved or its schema changed, only the associated web services must be updated. Client applications do not have to be modified as they are insulated from change by a metadata layer that maps the information requested to the actual source.
Trend 4: The transformation of collaboration from dedicated applications to pervasive collaboration.
Traditionally, the email filled the needs for collaboration in the organization. Using email, the user could send and receive messages, keep a basic contact list and maintain his or her calendar. In addition to the intended messaging functions, email has turned into a 'communication hub', serving the unintended
roles of activity inbox, workflow system, knowledge repository, and resource directory.
The collaborative capabilities of email are being further enhanced through the broad adoption of instant messaging technologies, but often this adds yet another application interface that the user must juggle on his desktop. Ever more sophisticated collaboration functions are becoming available as vendors roll out new tools for social networking, team interactions and document, knowledge and expertise management. How do you deliver all of these services to the desktop? The old email interface is already overloaded, and users simply cannot integrate more applications into their desktops.
The solution is moving these collaboration services out of the dedicated
applications and into the line of business and composite applications where users work. Collaboration is embedded through the use of collaborative services components which can be incorporated into any application.
In this way, presence awareness and collaboration can be directly imbedded into any other business processes. A manager reviewing a report of open support cases can see if the owner of a problematic case is online, and immediately contact her via instant messaging to help resolve the problem. An inventory manager can launch email messages to customers from the ERP application warning them of problems with product availability. The user no longer has to open the separate email or chat application and copy and paste the relevant context of the communication. Collaboration is done in the context of the activity and automatically encapsulates that business context.
Converging Trends
These four trends have been moving in parallel for some time. A normal business-as-usual approach would suggest that each trend would develop along its own path and perhaps merge or cross others at natural points in time. In fact, these trends are converging, and as they do so, they are fundamentally influencing and impacting each other. A unique opportunity is developing where the combined effect offers tremendous benefit and competitive advantage through the interplay of these trends. An integrated solution platform which capitalizes on this convergence can offer a much higher value than the sum of its component parts. Each of these software areas is impacted by each of the others.
Business Intelligence is moving from a stand-alone application model to an embedded BI model. The composite application is the natural delivery vehicle for embedded BI. The loosely coupled structure of composite applications requires ready access to information through SOA. Enterprise information architectures must include data services which support unstructured data and human resources. The integration of collaborative services directly into the fabric of BI and composite applications enhances usability, productivity and the quality of human interactions.
The convergence of these trends promises to become a virtuous circleof enhanced usability and productivity. IBM’s investment in supporting an ‘open architecture’ provides the opportunity for its customers to seamlessly integrate components whether they be from IBM, from partners such as SAP, or from competitors. Any ISV providing a line of business application may snap into this architecture and technology offering both flexibility and independence.
IBM Leverages Converging Trends
Where IBM is Now
IBM is ideally positioned to take advantage of these converging trends. It has industry leading software in each of the four critical areas.
Cognos is the flagship IBM Business Intelligence platform, and since its acquisition in 2008 has been rapidly integrated into the IBM Information Management group.
Lotus provides industry leading Composite Application services through its Lotus Notes 8 and WebSphere Portal products. IBM’s data as a service segment is well served through SOA tools provided by the WebSphere brand and data and metadata management tools from the InfoSphere group. Finally, Lotus provides a broad offering of industry leading collaboration products, including Notes/Domino for messaging and Sametime, Quickr and Connections for collaboration. Linkages between technology areas are currently somewhat limited. Lotus manages both the Composite Application and Collaboration products, so linkages naturally exist between these groups. The Information Management group includes both Cognos and InfoSphere and the interoperability between these areas is being rapidly enhanced.
The System z mainframe server platform is well supported by Information Management, now including Cognos. Most other software has been developed primarily for distributed platforms.
Where IBM is Going
IBM is taking active steps to take advantage of the trends we have described by enhancing the capabilities in each of these areas and providing closer integration across segments.
Moving forward, System z will play a more central role in IBM’s integrated software offerings. System z provides an ideal server platform for the large-scale, highly modular services based solutions that these trends predict. In addition the growing importance of Cloud Computing highlights the increased importance of virtualization of servers to provide on demand Software-as-a-Service. System z is highly optimized for these kinds of services and will become an increasingly central platform for IBM’s service-based offerings moving forward.
IBM is moving forward to provide highly modular service-based capabilities for embedded BI, data as a service and
collaboration as a service and to bring these together in a robust scalable composite application framework. System z will be central of this service-based architecture.
We describe some of the important areas IBM is working on to deliver this vision.
Integrating Cognos
Cognos was acquired by IBM in 2008. Its business intelligence software is being aggressively and rapidly integrated with other parts of the IBM software and hardware business.
The Cognos BI Server has been ported to the System z platform, providing a high performance, massively scalable offering to IBM mainframe customers. Cognos has been tightly integrated with the InfoSphere Data Warehouse and Information Server family of products, creating a complete, powerful and seamless set of offerings across the Information Management area.
Finally, Cognos has been working with Lotus to integrate Lotus Collaboration with Cognos business intelligence services. Lotus Sametime has been integrated into Cognos to provide live presence awareness and real-time collaboration capabilities within Cognos reports. Cognos has also been working to provide portlets and composite application components which can surface Cognos Business Intelligence charts and reports in the composite application interfaces provided by WebSphere Portal and Lotus Notes 8.
Lotus Composite Applications
Eclipse is an open source development platform developed and supported by the Eclipse Foundation. IBM has been a major contributor to the Eclipse Foundation and has adopted Eclipse as its development platform of choice for all IBM tools. IBM has also embraced Eclipse as the platform for building rich desktop
applications. Lotus Expeditor integrates powerful Eclipse extensions to provide a complete desktop application development environment.
Lotus Notes 8 now provides a complete environment for building and deploying desktop composite applications. Notes 8 is built upon Eclipse, includes Lotus Expeditor extensions, and provides a number of rich services which support composite application development and deployment.
• Property broker services which facilitate inter-component communication
• Server-based provisioning so that composite applications can be deployed and updated without having to make changes at the desktop
• Composite application wiring tool integrated into the Notes development environment
• Components which surface native Notes and Sametime messaging and collaboration functionality
Lotus provides multiple routes for composite application
development with Notes 8. Existing Notes customers can move in gradual steps towards composite application solutions by starting with their existing Notes applications and gradually enrich the composite application experience through the integration of additional Lotus and third party components.
ISVs and corporate developers can build new composite applications by moving existing application code into new composite application components, with the benefits of greater flexibility, personalization and off-line usage.
SOA and Metadata Standards
Metadata is defined in many ways; the most common definition is ‘data about data.’ More accurately, metadata is data that
describes processes, information, and objects. Those descriptions address characteristics such as technical attributes (for example, structure and behavior), business definitions (including vocabulary and taxonomy), and operational characteristics (such as activity metrics and usage history). Metadata encompasses so much that it's commonly misunderstood and seldom fully exploited.
But metadata is critical. In fact, it's the underlying enabler of a whole new wave of software innovation. Until recently, most metadata was used to describe technical things to people. Today, metadata is increasingly used to describe technical things to other technical things - removing manual translation steps entirely. SOA provides an excellent example of this trend, leveraging metadata to automate discovery, contract, and communication of services. IBM took a major role in driving the OMG XMI metadata standards
that allow customers to take a standards-based approach to creating metadata models that can now be read by all leading BI vendors and many of the smaller niche players as well.
As Service Oriented Architectures become the standard within the enterprise, the many sources of data must be made available to anyone, anywhere, at any time. IBM has made SOA a focus, with a consistent and coordinated message, across IBM Software Group. It understands that metadata management is critical to SOA success, and has recently enhanced its metadata management offerings through critical acquisitions. The InfoSphere Information Server provides a unified metadata infrastructure to support enterprise SOA.
SOA allows organizations to leverage these tools to enhance applications, processes and the user desktop experience. An important extension of this trend of data as a service will be to enable embedded and operational BI and
composite applications, since these are the areas where the enterprise will realize significant business value.
IBM & Collaboration
Lotus is the heart and hub for collaboration services within IBM Software Group. Lotus supplies an extensive set of collaboration tools, including Lotus
Notes/Domino, Lotus Sametime, Lotus Connections, Lotus Quickr and WebSphere Portal.
Lotus Notes has evolved through the years into a highly successful platform for messaging, application
development and collaboration, seamlessly integrating security, workflow and document management. Lotus has invested heavily in the portal and composite application areas to make collaboration possible and pervasive on these platforms.
Lotus Customers face the challenge of how to position Notes within the enterprise information architecture. Although Notes includes email, it also includes a wealth of additional collaboration capabilities. Until now, Notes data has been difficult to share with other enterprise applications and services. Notes excels at working with and collaborating around semi-structured and, to a lesser degree, unstructured data. Notes application data should be viewed as a critical component of any enterprise data architecture. It is a valuable and often strategic asset.
The convergence of the collaboration with the other three trends we have discussed offers the opportunity for IBM customers to gain significant leverage across its IT investments through productivity-enhancing integrated
collaboration.
The Unifying Role of Cloud Computing
Cloud computing has surfaced as ‘the next big thing’ in information technology. Cloud computing describes the delivery of mature Software-as-a-Service offerings through an amorphous 'cloud' of servers, software and services. All of this is transparent to the user, who simply accesses his software securely across the internet when and where he needs it. Cloud Computing offers massive economies as server resources can be virtualized and are allocated to specific customer needs as required. Analysts estimate that up to 25% of new software will be delivered as a service by the end of 2009. Established software
heavyweights like IBM, Oracle and Microsoft are being joined in this market by relative newcomers Google and Amazon.
Cloud Computing is a natural extension of the IBM On Demand message. At the same time, Cloud Computing forces a new look at software branding, software development, and software sales. It offers the opportunity for an established vendor like IBM to transform its software business.
IBM develops and sells software through specific software brands: WebSphere, Information Management, Lotus, Tivoli, etc. Software requirements and customer value have traditionally been compartmentalized according to these divisions. Cloud Computing promises to hide the seams between individual software products and software brands. If a customer wants to buy a cloud computing solution which integrates business intelligence, data warehouse and collaboration, she is going to focus on the set of functionality that she wants the vendor to deliver, not on the component software modules and the integration required between them. Ideally the software vendor will deliver a set of software services that is completely integrated to meet the customer requirements.
With the Cloud model, the greater breadth and depth of products and services the vendor can offer, the higher the value of the Cloud services the vendor can provide by bringing together these best of breed products. The delivery of Cloud Computing services crosses brand, application and platform boundaries and creates value based on the most complete integration of software services that can be delivered.
The Cloud Computing model promises to play to all of IBM's strengths. IBM leads the market in large scale server technology, both mainframe and distributed, it has the broadest and most complete platform and middleware software, and it has global consulting and software outsourcing services. As Cloud Computing is driving the integration of heretofore distinct technologies, the value of integrated collaboration becomes obvious. All software services are enhanced by including the ability for users to communicate, collaborate and share the content that they are otherwise viewing or using. Different pieces of information also take on a much greater value when combined into a composite application interface which is tailored to the user's role or activity.
Collaboration and composite application technology provided by the IBM Lotus brand lead the market. Whereas up until now Lotus software has been sold to meet specific collaboration requirements, it now promises to enhance the value of all IBM Cloud solutions through complete and seamless integration. Notes, the flagship of the Lotus brand, is central to an integrated IBM Cloud offering. This is driving the integration of Lotus technologies with other IBM services in new and compelling ways.
Cloud Computing – a Convergent Model
The Cloud Computing model is an example of how the converging trends that we described are accelerating and fostering the emergence of new software delivery models which cross the boundaries of hardware, software, brands and services. In the Cloud model, data is abstracted as a service. Users are not aware of servers, data warehouses or whirring hard disks. Business Intelligence becomes a capability, not an application. Users want to be able to visualize and
understand data, not build reports. User interfaces for Cloud services are naturally composite application interfaces. Since the application has disappeared as such, Cloud interfaces are built based on the user’s role and activities. Interfaces will aggregate components which draw data from across the Cloud and perform actions that may impact many types of services.
Cloud Computing forces a personalization of user interfaces, and these
personalized interfaces will be above all collaborative. The idea of working on a single application through a dedicated interface was a construct forced upon users by limitations of information technology. In reality people work first and foremost with other people and use information to enhance and enrich those
collaborative processes. In the new composite Cloud interfaces, collaboration will be truly pervasive and integrated as collaborative services throughout the application framework.
This changes the model for collaborative software. Instead of applications, collaborative software will become services, with virtualized and transparent back-end services. At the user interface, collaboration services will consist of standardized components and tag libraries based on standard published interfaces. Collaborative services can be linked to anything – data, documents, people, places, things – all based on standards-based metadata services for collaboration.
Metadata services are well developed and mature in the areas of data
management, business intelligence and service oriented architectures. Up until now metadata services have not been important to collaboration software. This is now changing. IBM and Lotus are working with business partner Sun & Son to provide these critical standards-based metadata services to Lotus collaboration products. This integration is occurring in multiple areas.
Expanding Data Cloud Architecture to include Notes Semi-structured Content
Metadata provides the critical link to Notes data for Data Cloud data warehouse and business intelligence services. Notes brings real-time Collaborative Business Intelligence services to data cloud services. Data cloud services are enhanced through the integration of the large body of semi-structured data stored in Notes applications and team spaces. This semi-structured data can be linked with structured data from enterprise applications to provide a seamless view of enterprise operations. By linking Notes documents with structured data,
unstructured file attachments acquire greater business context and become more widely accessible.
Delivering Collaborative Business Intelligence
Metadata services for Notes bring Collaborative BI capabilities to Business Intelligence. Collaborative BI links together the powerful reporting and analysis capabilities of Cognos BI with the rich context of Lotus Notes and Lotus collaboration. Business Intelligence tools are optimized for highly structured tabular data. Notes, on the other hand, manages rich semi-structured data which includes formatted rich text and attachments as well as routing and workflows. Collaborative BI brings these two worlds together in an integrated solution which realizes the best of both environments. BI applications can access Notes data directly through structured Notes fields. Hot links allow the user to drill down from a report directly into the highly collaborative Notes environment, either in the Notes client or in a web browser. The result is a higher value to the customer from a fully integrated solution.
Integrating Notes with Enterprise Information Architecture
Enterprise Information Architectures typically provide services for structured and unstructured data. They generally do not support the semi-structured data that comprises as much as 15% of the information in an organization. Since Notes is the leading platform for managing semi-structured data, the inclusion of Notes in an enterprise information architecture creates a much more complete and compelling architecture, including structured, semi-structured and unstructured information.
This can be done now using existing data management software tools and by leveraging Notes metadata. Notes metadata definitions can be imported into the InfoSphere Information Server using standard Cognos metadata formats. At this point, any of the InfoSphere data management tools can be used with any Notes data. This provides full metadata life cycle management for Notes data. Similarly, Notes metadata definitions tie Notes application data to Service
Oriented Architecture through business process modeling and brokering software.
Bringing Lotus Technologies Together
Metadata integration makes Notes a compelling part of the IBM data cloud architecture. As such, it serves to drive greater integration between Lotus products with other IBM software technologies and between Lotus products themselves. Cloud integration has stimulated discussion and planning around Domino server virtualization. This will not only integrate Notes more tightly into
the Cloud, but also promises to remove the scalability limitations that have hampered broader deployment of Notes in many organizations. As we have seen, the increased prominence for composite applications in the Cloud architecture leads to tighter integration between Notes and WebSphere Portal. Similarly, an enhanced role for Notes in the Cloud will create greater demand for other Lotus products such as Sametime, Quickr and Connections. In order for these products to better integrate into a Cloud architecture, they will also require access to metadata services. The natural approach will be to extend the technologies for Notes metadata management to these other Lotus products as well.
Conclusion
We have identified four critical trends in information technology: Changes in the nature of business intelligence
Changes in the form and function of desktop applications Changes in the ways we access data
Changes in how we collaborate using software
These trends are accelerating and converging; serving to disrupt existing business models and create whole new emergent opportunities. The trends challenge the established dominance of existing software, hardware and services vendors and create opportunities for flexible and innovative organizations to grow rapidly through new models for delivery.
Cloud Computing is an example of the disruptive new technology growing out of these converging trends. IBM has embraced these developments as opportunity instead of threat and has set up a new division dedicated to Cloud Computing. This promises to incorporate industry leading IBM software and hardware together into an unmatched software-as-a-service offering.
IBM is working to transform its existing software products across brands into software services which can work together as a unified whole across the Cloud. Key to this effort is the adoption and implementation of open metadata
standards across the entire software offering.
Cloud Computing will be delivered to the desktop through composite application interfaces. IBM Lotus Notes 8 and WebSphere Portal provide the composite application services to deliver these converged and integrated services to the user through rich and natural collaborative interfaces.
About the Authors
Ravi Har Singh Khalsa is CEO and Chief Architect at Sun & Son, an IBM business partner focused on collaborative business intelligence, composite applications and knowledge-based solutions. Ravi Har is the designer of the Data Modeler for Notes product.
Adrian Reason is the VP for Alliance’s at Sun & Son. Over the last 12 years he has been focused on solutions that bring Lotus into the mainstream of Enterprise Information management.
Mike Biere is a Business Intelligence specialist with 30 years experience at IBM and Cognos. He is currently a Sr. Marketing Manager for IBM Information management.
Caryn Meyers is the IBM WW Marketing Manager for Business Intelligence solution on System z with over 20 years experience of putting together Enterprise Class Solutions for IBM customers.
Andrea Greggo is the Worldwide Marketing Manager for IBM’s Cloud computing Division.
Mac Devine is an IBM Distinguished Engineer, Vice CTO for AIM, SaaS/Cloud CTO, SWG Networking Technical Lead.
For more information about services to plan and implement Composite Application interfaces, please contact:
T. Scott Rogers Solution Architect Office: 1-617-245-8070
