EAI & SOA
The Integration between EAI and SOA
Version: 1.0/EN
Document Release Control
Subject: The Integration between EAI and SOA
Description: This document details the integration between EAI and SOA Author: José Luiz Berg
Version Release Date Confidenciality Change By
1.0/EN 23/03/2011 PUBLIC José Luiz Berg
Declaration of Confidentiality and Intellectual Property
The contents of this document is property of JDBS. This document is protected by copyright laws and treaties around the world. All rights are reserved. This document may not be reproduced, stored or transmitted in any form or by any means, in whole or in part, without prior permission from JDBS.
If this document is marked as confidential in the table above, you can not read, reproduce or transmit this document unless you have concluded a nondisclosure agreement with JBDS or has received permission for its use.
To obtain permission to reproduce and for more information about the licensing of this document, please contact JBDS at the following address:
History of Systems Integration
In the beginning was the verb and the verb was “mainframes”. In these "super" computers, the environment was completely controlled, since as the hardware was proprietary; all software was developed by the equipment manufacturer, or by the internal development team of each company. At that time, there was only database integration, with many systems (usually developed in the same language) accessing data in the databases (sorted sequential files); there was not even a strict concept of system, but collections of individual programs that shared the same execution
environment.
In the late 70s, began appearing a few personal computers, but very expensive and limited. Finally in 1981, IBM introduced the first PC, which brought a silent revolution that went unnoticed by most people at that time: the hardware was no longer proprietary, and could be manufactured by anyone. Likewise, the software could be developed and purchased from any company, and the user began to choose which of them preferred to use. Such a choice was the seed of the whole system integration, because in a computer company, several different systems could be used for each area, and made no sense that the information had been retyped for use in each system, so they need to communicate somehow, to allow synchronization of information.
In the 90's saw the ERP systems, which were aimed at solving the problem of
synchronization of data, since its modular architecture were developed to provide separate modules for each business area, sharing the database, so data were exchanged between then. The problem with this approach is that the suppliers didn’t have all the modules required for the company, and there was no compatibility between modules from different vendors. Also, the modules not always met the needs of each area of business that when opted to use another system, returned to the same old problem of having to synchronize information.
Became obvious that, in a large company, same strategy would be required for systems to exchange information, avoiding rework and re-entering data, ensuring that all systems use the most updated information. Initially this strategy was to exchange files,
first on floppy disks, and with the introduction of local networks, through file sharing. At this time also began to emerge relational databases, using client server architecture, facilitating the sharing of data. With the evolution of networks and the adoption of the TCP/IP, began to appear integrations using sockets, allowing a higher speed for sending online information.
At that time, systems integration was called "middleware", or software that runs among other software, allowing
communication between them. With the exponential growth in the integration needs, the complexity of middleware systems has
System A
System C System B
System A System B
Espagueti Problem
Complexity grows exponentially with the inclusion of new components
Around 1998, began appearing a new concept: the use of an integration bus. With this concept, instead of each system to connect directly to other systems, all systems are connected to the bus and the bus has the intelligence to decide what systems will provide or receive information. With this new concept, was resolved the "spaghetti" problem, because the inclusion of a new system on the bus did not have a major impact on the overall integration complexity. Beyond this concept began to appear specific tools for integration,
which provided the bus and also features like data-mapping, persistence, state machines and connectors to allow communication using various technologies such as databases, sockets, FTP, HTTP, etc., and special connectors for major ERP and CRM systems.
Almost at the same time came the standard for distributed components by Microsoft, or DCOM (Distributed Component Object Model), which allowed a program to execute a function on a remote computer without the need for an external agent to perform the communication, because the support was built into the operating system itself. Indeed DCOM was based on fusion of the COM (Component Object Model), with the RPC (Remote Procedure Call), which was already used in X/Java systems for RMI or CORBA calls, but that had not avenged by the lack of standardization in the X world, since every operating system flavor used slightly different and sometimes incompatible versions. The use of COM/DCOM was a success, and allowed the development of systems and applications (when you click a Word document and the program is executed, loading the document, it is thanks to these technologies), but only works between computers using Windows, and most servers and enterprise systems at this time have X systems. With this limitation in mind, Microsoft and IBM, have sought a way to allow the same functionality across heterogeneous systems. The solution was to use HTTP as the data transport layer, because with the growth of the Internet, this service was standardized and available on all platforms, and the emerging XML standard for defining the data that were being exchanged. The WebService standards where created.
Today, WebServices are the most commonly adopted standard for systems integration, and the technology in fashion is service oriented architecture (SOA). The purpose of this document is not conceptualize SOA, but to show the relationship between EAI and this architecture, and present the points where both two meet, because despite some statements of the market that SOA "buried" the EAI, he is still alive, and ever more active, even if hidden under some other layer of "SOA tools." If you want a little more information on SOA, search for Oasis, SOA Magazine, SOA Manifesto, The Open Group Architecture Framework, or at the very end of the list, see my presentation called "SOA Facts & Actions".
Sistema A Sistema C Sistema B Sistema A Sistema C Sistema B Sistema D Sistema E Problema Espagueti Complexidade cresce exponencialmente com a inclusão de
novos componentes System A System C System B System D System E Integration Bus
Integration Concepts
An EAI system has some principles and goals. The main concept is to use an integration bus, acting as the controlled of the communication, without which it would be considered only a midleware. But other concepts are also very important:
Low Coupling
The concept of coupling in EAI can be defined as the complexity induced in the whole integration by changes in one single component. Any change on a system participating in the integration should generate little or no change on other components. Ideally, the coupling is the smallest possible, and the goal of EAI is to allow the complete abstraction of systems, allowing a system that meets a certain area of business to be changed without affecting other components, changing only its communication with the bus.
Data Model
Usually in an integration there are various data models involved, because each system has its vision of each piece of information. One of the primary functions of EAI is to provide the coupling, i.e., convert the data from the source model to the destination model, and vice versa. However, if we use the source model to pass by the bus data, we will be violating the first concept and inducing a coupling, because a change in the data model from the source system will generate impacts on integration of other systems. Therefore, the ideal is that the bus has its own data model, and provide the mapping of data to its internal model.
In reality there are several other reasons to use a bus data model, although apparently this generates more work than using the systems data models, because if you look at a simple interface that sends data from one system A to a system B, we have two data conversions: System A Model -> Bus Model -> System B Model, while utilizing systems models you have only one: System A Model -> System B Model. some of the other reasons are discussed later in the chapter specific to the data model. For a while it is important to understand only that this is one of the pillars for integration.
System A System B Conversion A x B System A System B Conversion A x Bus Conversion Bus x B
Apparently, using a bus data model requires more work to convert the data them directly from source to
destination, because two conversions are necessary, instead of only one
Separation of Functions
The goal of integration is to allow communication between systems, and not to steal its functionalities. This involves converting technologies, formats and data domains, and criticize requests that are not compliant to the standards. The problem is that with the increase in the resources offered by modern EAI tools shall be possible to implement various functions within them, often with features that should be in integrated systems, and not in the integration tool. Sometimes it is very difficult to identify the difference, because the boundaries are blurred. One of the
important parameters to verify this limit is the paternity of data: the basic rule is that EAI generates no information, only transforms information generated by the integrated systems. If your
implementation is generating some information, begin to suspect that you are extrapolating their functions.
Granularity
One of the most important settings for the EAI is the granularity of each interface, because it sets the unit of work and performance of the interface. Is often already said that EAI systems should not perform batch processing, and really, ETL systems are much more efficient for this type of
integration, but that doesn't mean the EAI can not be used, only that there must be specific
interfaces prepared for this type of processing. The problem typically occurs when someone has the idea of reading the batch file, and publish each record to the existing interface to be processed individually. It may seem easy and simple, but it sure will end up with your online processing, who will be hoping the batch end to go back to work.
The most important concept is that the granularity of the interfaces should always be constant and compatible with the type of process, i.e., if your procedure begins with a batch interface should continue until the end of processing.
Level of Cooperation
On integration, the level of cooperation indicates how system relates to the integration bus, i.e. how exchanges information with the outside world. We can define three distinct levels of cooperation:
Intrusive
The system is a bundle that has no programming or integration API. In this case, it is usuall to read/write directly to the system database, and the system doesn't notice that this process occurs. This type of integration must be built with care, why can generate inconsistencies with the native data.
Cooperative
Although the system has closed functionalities, has also an API for integration and some mechanism to receive and make calls to external systems. In this case, the integration is easier, but limited to the functionality provided by the system, and the technology available.
Programable
Most modern systems have some programming language or script, that allows to receive and make calls using various customised technologies. This is the optimal level of integration, allowing the system to be created with features "out-of-box", but you can add others, if required.
Complexity of the Problem X Solution
In systems integration there is no magic. If you have a complex problem, its solution is complex. For a simple problem, its solution is simple. Whenever someone says to you that has a simple solution to a complex problem, start distrusting that, or you overestimated the problem, or is being deceived. This seems like a simple and obvious concept, but many projects have already failed because forgotten this concept, and believed in magic and fast solutions.
With these concepts in mind, and knowing a little bit more about the history, principles and concepts of systems integration, let's now begin to understand the components and separate the functions so that whe can have efficient integration, with low coupling and a good maintainability.
Integration Layers
The main function of EAI is to allow communication between systems. But for doing that, it also needs to "know" what information needs to be sent or received from each system, and in what time. This knowledge is not part of the interface, because a system only needs to know how to send the information to the bus. So we have the separation of processing in two distinct layers:
Communications Layer
As communication, we can understand all tasks required to enable the exchange of information with the outside world. This communication can be one-way or two-way, synchronous or asynchronous, locally or remotely, and you can use one or a set of technologies to work. The main components of the communication layer are:
Adapters
Are the agents responsible for the "physical" communication. Can use various technologies and protocols. The most common are HTTP (S), FTP, files, database client, TCP/IP sockets, SOAP, REST, Corba, COM/DCOM, RMI, etc. In General, data is exchanged in the "physical" format of the external system.
Conversors
Are the components responsible for the conversion of data from the external format to a format more "friendly" to the bus, so it can be processed easily, rapidly and evenly. The format depends on the bus, but most modern systems typically use XML.
Processors
Processors are required in complex integrations. Are generally used when external systems have a logic of communication that needs to be implemented. May be the bus should wait several events to compose a request, breaking a request on various other specific events, wait or perform other tasks required for communication with the systems, or any similar taks. It should be remembered however that we are not running business rules in this point, but only specific rules to allow communication with a system. Processors generally use state diagrams to govern its functioning, and may have persistence mechanisms to hold the information until processing.
Mapmakers
Its function is converting data from the system model to the the internal model of the bus, i.e. change the "logical" format. This can result in concatenating attributes, use conversion tables, copy data from one attribute to another, perform the calculation and processing routines, validating and critisizing information and numerous other functions, but never forgetting that the bus cannot create any information, only transform what was received. Mapmakers work always under demand of processors, and data mapped are returned to them for processing.
Routers
Once the information has been received, converted, processed and mapped, next step is running the business rules. This information should be sent to the correct component in business layer, and this is the role of routing. It receives information from external systems and forwards it to the correct internal component, or the internal component receives information and forwards it to the external system using the router. Routers function as elements of abstraction, preventing that communication and business layers need to know each other to exchange information, reducing the coupling of the system.
Communication Layer System A Socket FTP XML Delimited Text XML XML XML XML Business Layer System Model Bus Model XML Processor Mapmaker Router Conversor Adapter Socket Adapter FTP
Business Layer
When it comes to the business layer, the information is already in the format and data model of the bus, and its function is to know what should be done with each request, performing what is called "business rule" of the bus. To allow this intelligence, the following components are required:
Processors
Is where the business logic is executed. As in the communication layer, usually consist of state diagrams and persistence mechanisms to store the information. May also have integrated workflow systems and user interaction support to enter data and information on the progress of tasks. Usually have any programming language or script, allowing complex tasks to be implemented, but you should be very careful that all processing is independent of the systems involved, ensuring independence and low coupling. Any processing that depends on a particular system should be done in that system’s communication layer.
Mapmakers
Usually the information need to be mapped in order to run the business requests. This implies copying data, compose and decompose into smaller requests, validate and critisize domains and matching informations, and countless other activities. Remembering always that we already received data in the correct “physical” and “logical” models. We are only converting data to match the required business rules, but the mapping output is also in the same model. Mapmakers work always under demand of processors, and data mapped are returned to them for processing.
Routers
In the same way as in the communication layer, we need to have a rotedor as an element of isolation between communication and business layers. Although initially this seems to be redundant, it introduces the concept of platform. Imagine a scenario where your company has 3 different billing systems, each with its own customers. Your business rule says that if a purchase order has already been terminated, then the request must be sent to the billing, but did not know which of customer billing systems should be used. If this decision is taken in the business processes, you have a high coupling between then and the systems, which is undesirable. By using this router, the business processe knows only that will send the request to the billing platform, and the router will have the intelligence to identify the correct system and forward the request to the router for the underliyng communication layer.
The separation of processing in these two layers can address one of the main goals of EAI, which is the independence of the specific systems. Theoretically, their business rules are separated from the communication, then a system could be replaced by another similar only by changing your
communication layer. It is clear that a change of this scale always involves changing business processes, but this way, the change can be made generating the lowest possible impact to the company. Communication Layer System A Business Layer System Model Bus Model XML Communication Layer System B XML XML XML Processor Mapmaker Router
Data Model
As has been said previously, the data model in EAI deserves a separate chapter. One of the biggest mistakes in systems integration is to find that their job is just to make the systems communicate. It is common for you to hear from integration teams that "they are only the pipe, and don't care about what's going on inside it". In fact, the integration concerns the exchange of information (represented by your data) between systems. This is your goal, and that is what adds value to the business. Of course this is only possible through communication, but if the information exchanged are incorrect, or simply arrive at the wrong time, they will be worth nothing to your business. Therefore, an old diskette sent by regular mail with the correct information recorded and delivered at the right time is worth much more than the most beautiful WebService, running at best SOA suite on the market, but that has the wrong information. And don't tell me that the exchange of disketes is not a form of integration.
So, defining the format and content of the data bus is so important, because handling these data is the objective of any integration. However, defining a data model for the bus is a difficult task, because they should be attached to the definition that each entity has to business processes. To create this model, we must first understand and document the business process, which is a time-consuming and complex task that involves interactions with the various areas of the company, and also implies that the areas need to be familiar with their business processes, which unfortunately is not very usuall. This bus data model will be named "corporate data model" from now on.
Returning to the previous discussion about using the systems data model in integration, we can see that, for simple integrations, involving only two systems, this may be easier, but for more complex problems, involving more than two systems, the mapping grows exponentially, and using the corporate data model becomes very easier and faster. The problem is that the complexity of interfaces always tends to increase, and simple interfaces become complex in a blink.
Indeed, in all cases the number of conversions required is expressed as (n-1)! for direct conversions, while for conversions using a bus model is n-1 (where n = number of systems) System C System A System B Transformation A x B Transformation B x C Transformation C x A System D Transformation A x D Transformation C x D Transformation B x D System A System B Transformation
A x Bus TransformationB x Bus System C Transformation C x Bus System D Transformation D x Bus
However, for 4 systems, 6 direct conversions are necessary, but using the bus model, only 4 conversions
System A System B Transformation A x Bus Transformation B x Bus System C Transformation C x Bus System C System A System B Transformation A x B Transformation B x C Transformation C x A
For conversions between 3 systems, 3 conversions are necessary for both cases System A System B Conversion A x B System A System B Conversion A x Bus Conversion Bus x B
Apparently, using a bus data model requires more work to convert the data them directly from source to destination, because two conversions are necessary, instead of only one
However, in addition to the complexity of mapping, there are other reasons to use a corporate data model, and the main one is that each system has its vision of the data. In a large organization, various systems are used for specific functions, and not always represent the vision of the company as a whole. When we take the data model of a specific system, we became hostages of that vision, but with time we ended up having to modify it to meet our needs, creating a hybrid model, which is no longer the original system model, but also doesn't correctly fit to our needs because was created with a limited view of the data. When this happens, the benefits of using the system model are gone.
When we decide to begin creating a corporate data model, despite the greater initial work, we can cast it to our needs, and we ended up creating a global vision of information, influencing the new systems that are developed or integrated, creating a virtuous cycle that ends up reducing the effort of integration and facilitating the understanding of the business process.
Business Process Model
Much has been said recently about BPM, and I believe it has been one of the bigger evolutions in systems integration. However, despite this, the concepts are not new, and already existed since the first EAI systems were designed. The key technology for building BPMs is the state diagram, which allows business process to be represented and carried out by the systems. They are the key parts of the processors in business layer, controlling the state of each request, storing information,
implementing and monitoring tasks (manual or automatic), and allowing the business processes to be realized by the integration system.
However, in old EAI systems, business was built on state diagrams, which use a technical language, requiring a professional with knowledge of a programming language, and causing the business analyst, who knows the company's business process, has to pass this information to the programmer to build the corresponding state diagrams. After construction, the system maintenance also needs to be done, i.e. whenever the business process changes, the business layer also needs to be changed. This whole process ends up generating inconsistencies due to flaws in understanding, and the dynamics of change ends up causing the updates not to be replicated to the integration.
Here comes in help the modern BPM systems and standards. The big advantage is that in addition to implement state diagrams, they also allow the construction of business diagrams, using a language that enables the business analyst to define their process, which are mapped directly to the systems without the need for a programming layer. Of course nothing is that easy, but the evolution of these systems will soon be the own business areas retain their processes, reducing the cost and time of change and minimizing the flaws. This development combined with the new tools
incorporated to BPM systems, such as the control of human workflows, rule engines and business monitoring, will give companies an agility and quality never before achieved.
Now we have another important point in favor of using a corporate data model: this model will be the common language that will be used by business process to exchange information. Without this common language, it is impossible to construct business processes that consider interactions between the various areas of the company.
EAI & SOA
Finally, after presenting the history, conceptualize the main components and objectives of EAI, we can better understand the interaction between EAI and SOA. However, before that, we still need to introduce some concepts about SOA, and clarify some common confusion in the market.
WebServices and SOA are two completely different concepts. SOA is a set of concepts used to create an architecture and WebService is a standard for remote communication between systems. Typically, the SOA architecture is implemented through WebServices, but the concept of SOA service does not specify the technology, and there are numerous other technologies that can be used for implementation.
So what is a WebService? Technically, it is a communication standard that uses XML messages validated by XSD schema definitions, and that usually exchange information using the HTTP(S) protocol. How is supported by most existing application servers on the market today, across all operating platforms, is turning to be the principal pattern for communication, especially between heterogeneous platforms.
So what is a service? A service is a real world action provided through a specification, and governed by restrictions and policies established through a contract. The technology used to build a service doesn't matter, and there are some other additional concepts, and the most important is isolation: the consumer of the service do not need to know what will happen internally, but only the information in the service description.
“Service Oriented Architecture (SOA) is a paradigm for organizing and utilizing
distributed capabilities that may be under the control of different ownership domains.” “Note that although SOA is commonly implemented using Web services, services can be made visible, support interaction, and generate effects through other
implementation strategies. Web service-based architectures and technologies are specific and concrete. While the concepts in the Reference Model apply to such systems, Web services are too solution specific to be part of a general reference model.”
Oasis - Reference Model for Service Oriented Architecture 1.0
“A service is a mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and is exercised consistent with constraints and policies as specified by the service description”
Having these concepts in mind, we can now try to understand how SOA and EAI interacts, and what is the relationship between then. From now on we will be using WebServices to exemplify SOA services, but always keep in mind that could be replaced by any other similar technology.
Communication Services
Within our EAI infrastructure, Web services can be found in two ways: when systems provide WebServices as part of your infrastructure, and when the integration bus provides the WebServices. Let’s see these two case in more detail:
System WebServices
System WebServices are found when trey have in-house application servers as part of its infrastructure, and offer services that can be used for remote execution of their functions by other systems, or even by the EAI. Naturally, the features provided by the system, and the data model used in services follows the system standards. The main advantage of these services, which enable the system to handle a job that was suppose to be done by EAI, is that other systems can make services calls directly. The disadvantage of this option is that we end up going back to the old espagueti problem, as all systems will start to run calls directly to each other, increasing the complexity.
Integration Bus WebServices
Integration Bus WebServices are found when we start to build services that enable systems that do not have the ability to provide WebServicess, to provide their functionality to other systems through the bus, which will forward requests to the systems using some old technologies already used by the integration. In this case, the functionality and data model also follows the system standards, because integration is only providing the technical resources (acting like and application server) to build the WebServices.
One important point is that theses services are in communications layer, so they are called "communication services". But if we want to build a service in the business layer? In this case, the service starts to became more interesting, and similarly to the other layer, let's call then "business services".
Business Services
We saw that in this layer, the tasks performed are the company's business processes, and the data model is the corporate model. Then we have a service that performs the business process, using common definitions adopted for the company. More importantly, this process is independent of the systems that are connected to the bus, because whatever they need, the business process will forward the request to the correct platform, and the communication layer will be responsible for making the requests to the correct system.
So existing services in this layer are quite powerful, because they correspond to the official process of the company, are not changed by infrastructure problems and changes, and are
independent of the systems that will execute the request. Then, once the clients are using one of these services, they know that the task will be performed correctly and homogenously, and that if a business process is changed, the new version will be automatically used by all clients simultaneously.
In fact, this is the goal of a SOA infrastructure: build a set of services that execute business rules, and that can be used by client applications, enabling a homogeneous and centralised vision of business process by abstracting the technical details, and ensuring that corporate objectives are achieved.
Let's see the (almost) final model of our integration architecture:
Whereas that the final goal of SOA is to use only business services, no longer makes sense to have communication services with the same functionality published to clients. Then the communication services exist only for the integration bus to use in communication layer. However, we know that the WebServices are not very efficient compared to other technologies. Its main advantage is the
compatibility, but if we have only the integration bus performing calls, this advantage is no longer relevant, and we can return to using other technologies more efficient to communicate with systems (except, of course, when the system
only supports WebServices).
Considering this change, we now have the final model of the integration architecture:
With the implementation of business services, a problem calls our attention: for the applications to call business services, using the corporate data model, they will have to make mapping from its data model to the corporate model, and vice versa. This adds to each application a level of complexity that only exists because of SOA architecture, which was previously provided by EAI. Each system has its degree of complexity and flexibility to deal with this problem, and there is not a definite rule for its resolution, but some assumptions can be followed:
When a system participates in a business process, usually uses the communication layer, and forwards its data in the native model for integration;
When a system makes calls to external processes, without participation in it, usually uses the business services, and need to manage the mapping of data;
In this case, a possible solution is to build a mapping service that could receive data in the system model and return in the corporate model and vice versa, being treated as another service provided by integration. It may seem interesting at a first glance, but this approach has performance and availabiblity constraints that need to be addressed very carefully before implementation;
Allways remember one of the basic concepts of integration: there are no simple solutions to complex problems.
Conclusions
In this document, we review the history of EAI, and describe the main blocks that make up its structure, showing where and how SOA infrastructure is connected to. With this vision, we can realize that rather than replace the integration, service oriented architecture fits and complements the integration architecture and the modern "SOA Tools", despite having many conceptual and technical enhancements, have all components before presented in traditional integration tools, and in reality exists as an evolution of then.
The conclusion therefore is that the two visions were complementary and fundamental for the construction of a system architecture that meets the needs of a modern company, using the new features and functionality without forgetting the old concepts and standards, by taking the best advantage of each one and fulfilling its role of bringing agility and quality for business.
