7-1-1_Publish_Subscribe_Developers_Guide

(1)

Publish-Subscribe

Developer’s Guide

Version 7.1.1

January 2008

Title Page

(2)

This document applies to webMethods Integration Server Version 7.1.1 and webMethods Developer Version 7.1.1 and to all subsequent releases. Specifications contained herein are subject to change and these changes will be reported in subsequent release notes or new editions. © Copyright Software AG 2008. All rights reserved. Copyright & Docu‐ ment ID

(3)

About this Book

webMethods Developer provides tools to integrate resources. It enables users to build integration solutions locally within one webMethods Integration Server or across multiple Integration Servers all exchanging information via a Broker. This guide is for developers and administrators who want to make use of this capability.

Document Conventions

Note: With webMethods Developer, you can create Broker/local triggers and JMS triggers. A Broker/local trigger is trigger that subscribes to and processes documents published/delivered locally or to the Broker. A JMS trigger is a trigger that receives messages from a destination (queue or topic) on a JMS provider and then processes those messages. This guide discusses development and use of Broker/local triggers only. Where the term triggers appears in this guide, it refers to Broker/local triggers. For information about creating JMS triggers, see the webMethods Integration Server JMS Client Developer’s Guide. Convention Description Bold Identifies elements on a screen. Italic Identifies variable information that you must supply or change based on your specific situation or environment. Identifies terms the first time they are defined in text. Also identifies service input and output variables.

Narrow font Identifies storage locations for services on the webMethods Integration Server using the convention folder.subfolder:service.

Typewriter

font Identifies characters and values that you must type exactly or _{messages that the system displays on the console.}

UPPERCASE Identifies keyboard keys. Keys that you must press simultaneously are joined with the “+” symbol. \ Directory paths use the “\” directory delimiter unless the subject is UNIX‐specific. [ ] Optional keywords or values are enclosed in [ ]. Do not type the [ ] symbols in your own code.

(10)

About this Book

Additional Information

The webMethods Advantage Web site at http://advantage.webmethods.com provides you with important sources of information about webMethods products:

Troubleshooting Information. The webMethods Knowledge Base provides

troubleshooting information for many webMethods products.

Documentation Feedback. To provide feedback on webMethods documentation, go to

the Documentation Feedback Form on the webMethods Bookshelf.

Additional Documentation. Starting with 7.0, you have the option of downloading the

documentation during product installation to a single directory called

“_documentation,” located by default under the webMethods installation directory. In addition, you can find documentation for all webMethods products on the

(11)

1

An Introduction to the Publish-and-Subscribe Model

What Is the Publish-and-Subscribe Model? . . . 12

webMethods Components . . . 13

(12)

1 An Introduction to the Publish-and-Subscribe Model

Introduction

Companies today are tasked with implementing solutions for many types of integration challenges within the enterprise. Many of these challenges revolve around application integration (between software applications and other systems) and fall into common patterns, such as Propagation. Propagation of similar business objects from one system to multiple other systems, for example, an order status change or a product price change. Synchronization. Synchronization of similar business objects between two or more systems to obtain a single view, for example, real‐time synchronization of customer, product registration, product order, and product SKU information among several applications. This is the most common issue requiring an integration solution. In a one‐way synchronization, there is one system (resource) that acts as a data source and one or more resources that are targets of the synchronization. In a two‐way synchronization, every resource is both a potential source and target of a synchronization. There is not a single resource that acts as the primary data resource. A change to any resource should be reflected in all other resources. This is called a two‐way synchronization. Aggregation. Information joined from multiple sources into a common destination system, for example, communicating pharmacy customer records and prescription transactions and Web site data into a central application and database. The webMethods product suite provides tools that you can use to design and deploy solutions that address these challenges using a publish‐and‐subscribe model.

What Is the Publish-and-Subscribe Model?

The publish‐and‐subscribe model is a specific type of message‐based solution in which messages are exchanged anonymously through a message broker. Applications that produce information that needs to be shared will make this information available in specific types of recognizable documents that they publish to the message broker. Applications that require information subscribe to the document types they need. At run time, the message broker receives documents from publishers and then distributes the documents to subscribers. The subscribing application processes or performs work using the document and may or may not send a response to the publishing application. In a webMethods system, the webMethods Integration Server or applications running on the webMethods Integration Server publish documents to the Broker. The Broker then routes the documents to subscribers located on other Integration Servers. The following sections provide more detail about these components.

(13)

webMethods Components

The Integration Server and the Broker share a fast, efficient process for exchanging documents across the entire webMethods system.

Integration Server

The Integration Server is the system’s central run‐time component. It serves as the entry point for the systems and applications that you want to integrate, and is the system’s primary engine for the execution of integration logic. It also provides the underlying handlers and facilities that manage the orderly processing of information from resources inside and outside the enterprise. The Integration Server publishes documents to and receives documents from the Broker. For more information about the Integration Server, see the webMethods Integration Server Administrator’s Guide.

Broker

The Broker forms the globally scalable messaging backbone of webMethods components. It provides the infrastructure for implementing asynchronous, message‐based solutions that are built on the publish‐and‐subscribe model or one of its variants, request/reply or publish‐and‐wait. The role of the Broker is to route documents between information producers (publishers) and information consumers (subscribers). The Broker receives, queues, and delivers

Integration Server Cluster

Resource Resource Resources Broker Integration Server Integration Server Adapters

(14)

1 An Introduction to the Publish-and-Subscribe Model The Broker maintains a registry of document types that it recognizes. It also maintains a list of subscribers that are interested in receiving those types of documents. When the Broker receives a published document, it queues it for the subscribers of that document type. Subscribers retrieve documents from their queues. This action usually triggers an activity on the subscriber’s system that processes the document. A webMethods system can contain multiple Brokers. Brokers can operate in groups, called territories, which allow several Brokers to share document type and subscription information. For additional information about Brokers, see the webMethods Broker Administrator’s Guide. For more information about how documents flow between the Integration Server and the Broker, see Chapter 2, “An Overview of the Publish and Subscribe Paths”.

Basic Elements in the Publish-and-Subscribe Model

The following sections describe the basic building blocks of an integration solution that uses the publish‐and‐subscribe model.

Documents

In an integration solution built on the publish‐and‐subscribe model, applications publish and subscribe to documents. Documents are objects that webMethods components use to encapsulate and exchange data. A document represents the body of data that a resource passes to webMethods components. Often it represents a business event such as placing an order (purchase order document), shipping goods (shipping notice), or adding a new employee (new employee record). Each published document includes an envelope. The envelope is much like a header in an email message. The envelope records information such as the sender’s address, the time the document was sent, sequence numbers, and other useful information for routing and control. It contains information about the document and its transit through your webMethods system.

Publishable Document Types

Every published document is associated with a publishable document type. A publishable document type is a named schema‐like definition that describes the structure of a particular kind of document that can be published and subscribed to. An instance of a publishable document type can either be published locally within an Integration Server, or can be published to a Broker. In a publication environment that includes a Broker, each publishable document type is bound to a Broker document type. Clients on the Broker subscribe to publishable document types. The Broker uses publishable document types to determine which clients to distribute documents to.

For more information about publishable document types, see Chapter 5, “Working with

(15)

Triggers (Broker/Local Triggers)

Trigger, specifically Broker/local triggers establish subscriptions to publishable document

types. Triggers also specify the services that will process documents received by the subscription. Within a trigger, a condition associates one or more publishable document types with a service.

For more information about triggers, see Chapter 7, “Working with Triggers”.

Services

Services are method‐like units of work. They contain logic that the Integration Server executes. You build services to carry out work such as extracting data from documents, interacting with back‐end resources, and publishing documents to the Broker. When you build a trigger, you specify the service that you want to use to process the documents that you subscribe to. For more information about building services, see the webMethods Developer User’s Guide.

Adapter Notifications

Adapter notifications notify your webMethods system whenever a specific event occurs on an adapterʹs resource. The adapter notification publishes a document when the specified event occurs on the resource. For example, if you are using the JDBC Adapter and a change occurs in a database table that an adapter notification is monitoring, the adapter notification publishes a document containing data from the event and sends it to the Integration Server. Each adapter notification has an associated publishable document type. The Integration Server assigns this document type the same name as the adapter notification but appends “PublishDocument” to the name. You can use triggers to subscribe to the publishable document types associated with adapter notifications. The service associated with the publishable document type in the trigger condition might perform some additional processing, updating, or synchronization based on the contents of the adapter notification.

Canonical Documents

A canonical document is a standardized representation that a document might assume while it is passing through your webMethods system. A canonical document acts as the intermediary data format between resources. Note: With webMethods Developer, you can create Broker/local triggers and JMS triggers. This guide discusses development and use of Broker/local triggers only. Where the terms “trigger” or “triggers” appear in this guide, they refer to Broker/local triggers.

(16)

1 An Introduction to the Publish-and-Subscribe Model purchase order format. This format is called the ʹcanonicalʹ form of the purchase order document. The canonical document is published, delivered, and passed to services that process purchase orders. By converting a document to a neutral intermediate format, subscribers (such as adapter services) only need to know how to convert the canonical document to the required application format. If canonical documents were not used, every subscriber would have to be able to decode the native document format of every publisher. A canonical document is a publishable document type. The canonical document is used when building publishing services and subscribed to when building triggers. In flow services, you can map documents from the native format of an application to the canonical format.

(17)

2

An Overview of the Publish and Subscribe Paths

Overview of the Publishing Path . . . . 18

Overview of the Subscribe Path . . . . 27

(18)

2 An Overview of the Publish and Subscribe Paths

Introduction

In the webMethods system, Integration Servers exchange documents via publication and subscription. One Integration Server publishes a document and one or more Integration Servers subscribe to and process that document. This chapter provides overviews of how the Integration Server interacts with the Broker to publish and subscribe to documents, specifically How the Integration Server publishes documents to the Broker. How the Integration Server retrieves documents from the Broker. How the Integration Server publishes and subscribes to documents locally.

Overview of the Publishing Path

When the Integration Server is configured to connect to a Broker, the Integration Server can publish documents to the Broker. The Broker then routes the documents to all of the subscribers. The following sections describe how the Integration Server interacts with the Broker in these publishing scenarios: Publishing a document to the Broker. Publishing a document to the Broker when the Broker is not available. Publishing a document to the Broker and waiting for a reply (request/reply).

Publishing Documents to the Broker

When the Integration Server sends documents to a configured Broker, the Integration Server either publishes or delivers the document. When the Integration Server publishes a document, it is broadcast to all subscribers. The Broker routes the document to all clients subscribed to that document. When the Integration Server delivers a document, the delivery request identifies the document recipient. The Broker places the document in the queue for the specified client only. Note: Unless otherwise noted, this guide describes the functionality and interaction of the webMethods Integration Server version 7.1 and the webMethods Broker version 7.1. Note: If a Broker is not configured for the Integration Server, all publishes become local publishes, and delivering documents to a specific recipient is not available. For more information about publishing documents locally, see “Overview of Local Publishing” on page 34.

(19)

The following diagram illustrates how the Integration Server publishes or delivers documents to the Broker when the Broker is connected.

Publishing to the Broker

Step Description 1 A publishing service on the Integration Server sends a document to the dispatcher (or an adapter notification publishes a document when an event occurs on the resource the adapter monitors). Before the Integration Server sends the document to the dispatcher, it validates the document against its publishable document type. If the document is not valid, the service returns an exception specifying the validation error. 2 The dispatcher obtains a connection from the connection pool. The connection pool is a reserved set of connections that the Integration Server uses to publish documents to the Broker. To publish a document to the Broker, the Integration Server uses a connection for the default client. 3 The dispatcher sends the document to the Broker. 4 The Broker examines the storage type for the document to determine how to store the document. If the document is volatile, the Broker stores the document in memory. If the document is guaranteed, the Broker stores the document in memory and on disk.

webMethods Integration Server webMethods Broker

3 4 2 Client Queue X Dispatcher Publishing

Service Connection Pool

Client Queue Y Memory Guaranteed Storage 1 5 6 7 7

(20)

2 An Overview of the Publish and Subscribe Paths Notes: You can configure publishable document types and Integration Server so that Integration Server does not validate documents when they are published. For more information about validating publishable document types, see “Specifying Validation for a Publishable Document Type” on page 68. If a transient error occurs while the Integration Server publishes a document, the audit subsystem logs the document and assigns it a status of FAILED. A transient error is an error that arises from a condition that might be resolved quickly, such as the unavailability of a resource due to network issues or failure to connect to a database. You can use webMethods Monitor to find and resubmit documents with a status of FAILED. 5 The Broker routes the document to subscribers by doing one of the following: If the document was published (broadcast), the Broker identifies subscribers and places a copy of the document in the client queue for each subscriber. If the document was delivered, the Broker places the document in the queue for the client specified in the delivery request. If there are no subscribers for the document, the Broker returns an acknowledgement to the publisher and then discards the document. If, however, a deadletter subscription exists for the document, the Broker deposits the document in the queue containing the deadletter subscription. For more information about creating deadletter subscriptions, see webMethods Broker Client Java API Reference Guide. A document remains in the queue on the Broker until it is picked up by the subscribing client. If the time‐to‐live for the document elapses, the Broker discards the document. For more information about setting time‐to‐live for a publishable document type, see “Setting the Time‐to‐Live for a Publishable Document Type” on page 67. 6 If the document is guaranteed, the Broker returns an acknowledgement to the dispatcher to indicate successful receipt and storage of the document. The dispatcher returns the connection to the connection pool. 7 The Integration Server returns control to the publishing service, which executes the next step. Step Description

(21)

Publishing Documents When the Broker Is Not Available

The Integration Server constantly monitors its connection to the Broker and will alter the publishing path if it determines that the configured Broker is not available. If the Broker is not connected, the Integration Server routes guaranteed documents to an outbound document store. The documents remain in the outbound document store until the connection to the Broker is re‐established. The following diagram illustrates how the Integration Server publishes documents when the Broker is disconnected.

Publishing when the Broker is not available

Step Description 1 A publishing service on the Integration Server sends a document to the dispatcher (or an adapter notification publishes a document when an event occurs on the resource the adapter monitors). Before the Integration Server sends the document to the dispatcher, it validates the document against its publishable document type. If the document is not valid, the service returns an exception specifying the validation error. 2 The dispatcher detects that the Broker is not available and does one of the following depending on the storage type of the document: If the document is guaranteed, the dispatcher routes the document to the outbound document store on disk. If the document is volatile, the dispatcher discards the document and the publishing service throws an exception. The Integration Server executes the next step in the publishing service. 3 When the Integration Server re‐establishes a connection to the Broker, the

4 2

Dispatcher Publishing

Service Connection Pool Memory

Guaranteed Storage 1 5 6 7 3 Client Queue X Client Queue Y Outbound Document Store

(22)

Notes:

If you do not want published documents placed in the outbound document store when the Broker is unavailable, you can configure Integration Server to throw a ServiceException instead. The value of the watt.server.publish.useCSQ_parameter

determines whether Integration Server places documents in the outbound document store or throws a ServiceException. After the connection to the Broker is re‐established, the Integration Server sends all newly published documents (guaranteed and volatile) to the outbound document store until the outbound store has been emptied. This allows the Integration Server to maintain publication order. After the Integration Server empties the outbound document store, the Integration Server resumes publishing documents directly to the Broker. 4 The Integration Server automatically sends the documents from the outbound document store to the Broker. To empty the outbound document store more rapidly, the Integration Server sends the documents in batches instead of one at a time. Note: The Integration Server uses a single connection to empty the outbound document store to preserve publication order. 5 The Broker examines the storage type for the document, determines that it is guaranteed and stores the document in memory and on disk. 6 The Broker routes the document to subscribers by doing one of the following: If the document was published (broadcast), the Broker identifies subscribers and places a copy of the document in the client queue for each subscriber. If the document was delivered, the Broker places the document in the queue for the client specified in the delivery request. If there are no subscribers for the document, the Broker returns an acknowledgement to the publisher and then discards the document. If, however, a deadletter subscription exists for the document, the Broker deposits the document in the queue containing the deadletter subscription. For more information about creating deadletter subscriptions, see webMethods Broker Client Java API Reference Guide. A document remains in the queue on the Broker until the subscribing client picks it up. If the time‐to‐live for the document elapses, the Broker discards the document. For more information about setting time‐to‐live for a publishable document type, see “Setting the Time‐to‐Live for a Publishable Document Type” on page 67. 7 The Broker returns an acknowledgement to the Integration Server to indicate successful receipt and storage of the guaranteed document. The Integration Server removes the document from the outbound document store. Step Description

(23)

2 An Overview of the Publish and Subscribe Paths If Integration Server makes 4 attempts to transmit a document from the outbound document store to the Broker and all attempts fail, the audit subsystem logs the document and assigns it a status of STATUS_TOO_MANY_TRIES. If a transient error occurs while the Integration Server publishes a document, the audit subsystem logs the document and assigns it a status of FAILED. You can configure publishable document types and Integration Server so that Integration Server does not validate documents when they are published. For more information about validating publishable document types, see “Specifying Validation for a Publishable Document Type” on page 68.

Publishing Documents and Waiting for a Reply

In a publish‐and‐wait scenario, a service publishes a document (a request) and then waits for a reply document. This is sometimes called the request/reply model. A request/reply can be synchronous or asynchronous. In a synchronous request/reply, the publishing flow service stops executing while it waits for a response. When the service receives a reply document from the specified client, the service resumes execution In an asynchronous request/reply, the publishing flow service continues executing after publishing the request document. That is, the publishing service does not wait for a reply before executing the next step in the flow service. The publishing flow service must invoke a separate service to retrieve the reply document. Tip! You can use webMethods Monitor to find and resubmit documents with a status of STATUS_TOO_MANY_TRIES or FAILED. For more information about using webMethods Monitor, see the webMethods Monitor documentation.

(24)

The following diagram illustrates how the Integration Server and Broker handle a synchronous request/reply.

Publishing a document to the Broker and waiting for a reply

Step Description 1 A publishing service sends a document (the request) to the dispatcher. The Integration Server populates the tag field in the document envelope with a unique identifier that will be used to match up the reply document with this request. The publishing service enters into a waiting state. The service will not resume execution until it receives a reply from a subscriber or the wait time elapses.The Integration Server begins tracking the wait time as soon as it publishes the document. Before the Integration Server sends the document to the dispatcher, it validates the document against its publishable document type. If the document is not valid, the service returns an exception specifying the validation error. The service unblocks, but with an exception. 2 The dispatcher obtains a connection from the connection pool. The connection pool is a reserved set of connections that the Integration Server uses to publish documents to the Broker. To publish a request document to the Broker, the Integration Server uses a connection for the request/reply client. Note: If the Broker is not available, the dispatcher routes the document to the outbound document store. For more information, see “Publishing Documents When the Broker Is Not Available” on page 21.

2 Dispatcher Publishing

Service ₁ Connection Pool Memory

4 5 7 3 Pending Replies 11 10 6 Publishing Server’s Request/Reply Client Queue 8 9 Client Queue X Client Queue Y Guaranteed Storage

(25)

2 An Overview of the Publish and Subscribe Paths 3 The dispatcher sends the document to the Broker. 4 The Broker examines the storage type for the document to determine how to store the document. If the document is volatile, the Broker stores the document in memory. If the document is guaranteed, the Broker stores the document in memory and on disk. 5 The Broker routes the document to subscribers by doing one of the following: If the document was published (broadcast), the Broker identifies subscribers and places a copy of the document in the client queue for each subscriber. If the document was delivered, the Broker places the document in the queue for the client specified in the delivery request. If there are no subscribers for the document, the Broker returns an acknowledgement to the publisher and then discards the document. If, however, a deadletter subscription exists for the document, the Broker deposits the document in the queue containing the deadletter subscription. For more information about creating deadletter subscriptions, see webMethods Broker Client Java API Reference Guide. A document remains in the queue on the Broker until it is picked up by the subscribing client. If the time‐to‐live for the document elapses, the Broker discards the document. For more information about setting time‐to‐live for a publishable document type, see “Setting the Time‐to‐Live for a Publishable Document Type” on page 67. 6 If the document is guaranteed, the Broker returns an acknowledgement to the dispatcher to indicate successful receipt and storage of the document. The dispatcher returns the connection to the connection pool. 7 Subscribers retrieve and process the document. A subscriber uses the pub.publish:reply service to compose and publish a reply document. This service automatically populates the tag field of the reply document envelope with the same value used in the tag field of the request document envelope. The pub.publish:reply service also automatically specifies the requesting client as the recipient of the reply document 8 One or more subscribers send reply documents to the Broker. The Broker stores the reply documents in memory. The Broker places the reply documents in the request/reply client queue for the Integration Server that initiated the request. Step Description

(26)

2 An Overview of the Publish and Subscribe Paths Notes: If the requesting service specified a publishable document type for the reply document, the reply document must conform to the specified type. Otherwise, the reply document can be an instance of any publishable document type. A single request might receive many replies. The Integration Server that initiated the request uses only the first reply document it retrieves from the Broker. The Integration Server discards all other replies. First is arbitrarily defined. There is no guarantee provided for the order in which the Broker processes incoming replies. All reply documents are treated as volatile documents. Volatile documents are stored in memory and will be lost if resource on which the reply document is located shuts down or if a connection is lost while the reply document is in transit. If the wait time elapses before the service receives a reply, the Integration Server ends the request, and the service returns a null document that indicates the request timed out. The Integration Server then executes the next step in the flow service. If a reply document arrives after the flow service resumes execution, the Integration Server rejects the document and creates a journal log message stating that the document was rejected because there is no thread waiting for the document. You can configure publishable document types and Integration Server so that Integration Server does not validate documents when they are published. For more information about validating publishable document types, see “Specifying Validation for a Publishable Document Type” on page 68. 9 The Integration Server that initiated the request obtains a request/reply client from the connection pool and retrieves the reply documents from the Broker. 10 The Integration Server uses the tag value of the reply document to match up the reply with the original request. 11 The Integration Server places the reply document in the pipeline of the waiting service. The waiting service resumes execution. Step Description

(27)

Overview of the Subscribe Path

When Integration Server is connected to a Broker, the path a document follows on the subscriber side includes retrieving the document from the Broker, storing the document on Integration Server, and processing the document. The subscription path for a document depends on whether the document was published to all subscribers (broadcast) or delivered to Integration Server directly. The following sections describe how Integration Server interacts with the Broker to retrieve published and delivered documents.

The Subscribe Path for Published Documents

When a document is published or broadcast, the Broker places a copy of the document in the client queue for each subscribing trigger. Each subscribing trigger will retrieve and process the document. The following diagram illustrates the path of a document to a subscriber (trigger) on the Integration Server. Note: For information about the subscribe path for documents that match a join condition, see “Subscribe Path for Documents that Satisfy a Join Condition” on page 167.

(28)

Subscribe path for published documents

Step Description 1 The dispatcher on the Integration Server uses a server thread to request documents from a trigger’s client queue on the Broker. Note: Each trigger on the Integration Server has a corresponding client queue on the Broker. 2 The thread retrieves a batch of documents for the trigger. 3 The dispatcher places the documents in the trigger’s queue in the trigger document store. The trigger document store is saved in memory. The dispatcher then releases the server thread used to retrieve the documents. Dispatcher Trigger Service X1 5 6 4 1 3

Memory

2

Trigger

Queue X Queue YTrigger Trigger Document Store

Trigger Service

X2

Trigger Service Y1 Trigge

r Servic e Y2 Client Queue X Client Queue Y Guaranteed Storage

(29)

2 An Overview of the Publish and Subscribe Paths Notes: After receiving an acknowledgement, the Broker removes its copy of the document from guaranteed storage. The Integration Server returns an acknowledgement for guaranteed documents only. 4 The dispatcher obtains a thread from the server thread pool, pulls a document from the trigger queue, and evaluates the document against the conditions in the trigger. Note: If exactly‐once processing is configured for the trigger, the Integration Server first determines whether the document is a duplicate of one that has already been processed by the trigger. The Integration Server continues processing the document only if the document is new. 5 If the document matches a trigger condition, the dispatcher executes the trigger service associated with that condition. If the document does not match a trigger condition, the Integration Server discards the document, returns an acknowledgement to the Broker, and returns the server thread to the server thread pool. The Integration Server also generates a journal log message stating that the document did not match a condition. 6 After the trigger service executes to completion (success or error), one of the following occurs: If the trigger service executed successfully, the Integration Server returns an acknowledgement to the Broker (if this is a guaranteed document). The Integration Server then removes the copy of the document from the trigger queue and returns the server thread to the thread pool. If a service exception occurs, the trigger service ends in error and the Integration Server rejects the document. If the document is guaranteed, the Integration Server returns an acknowledgement to the Broker. The Integration Server removes the copy of the document from the trigger queue, returns the server thread to the thread pool, and sends an error document to indicate that an error has occurred. If a transient error occurs during trigger service execution and the service catches the error, wraps it and re‐throws it as an ISRuntimeException, then the Integration Server waits for the length of the retry interval and re‐ executes the service using the original document as input. If the Integration Server reaches the maximum number of retries and the trigger service still fails because of a transient error, the Integration Server treats the last failure as a service error. For more information about retrying a trigger service, see “Configuring Transient Error Handling” on page 134. Step Description

(30)

2 An Overview of the Publish and Subscribe Paths document from the Broker when the server restarts or the connection is re‐established. (That is, the documents are redelivered.) For more information about guaranteed documents, see “Selecting a Document Storage Type” on page 65. If a trigger service generates audit data on error and includes a copy of the input pipeline in the audit log, you can use webMethods Monitor to re‐invoke the trigger service at a later time. For more information about configuring services to generate audit data, see the webMethods Developer User’s Guide. It is possible that a document could satisfy more than one condition in a trigger. However, the Integration Server executes only the service associated with the first satisfied condition. The processing mode for a trigger determines whether the Integration Server processes documents in a trigger queue serially or concurrently. In serial processing, the Integration Server processes the documents one at a time in the order in which the documents were placed in the trigger queue. In concurrent processing, the Integration Server processes as many documents as it can at one time, but not necessarily in the same order in which the documents were placed in the queue. For more information about document processing, see “Selecting Messaging Processing” on page 128. If a transient error occurs during document retrieval or storage, the audit subsystem logs the document and assigns it a status of FAILED. A transient error is an error that arises from a condition that might be resolved later, such as the unavailability of a resource due to network issues or failure to connect to a database. You can use webMethods Monitor to find and resubmit documents with a FAILED status. For more information about using webMethods Monitor, see the webMethods Monitor documentation. You can configure a trigger to suspend and retry at a later time if retry failure occurs. Retry failure occurs when Integration Server makes the maximum number of retry attempts and the trigger service still fails because of an ISRuntimeException. For more information about handling retry failure, see “Handling Retry Failure” on page 136.

The Subscribe Path for Delivered Documents

A publishing service can deliver a document by specifying the destination of the document. That is, the publishing service specifies the Broker client that is to receive the document. When the Broker receives a delivered document, it places a copy of the document in the queue for the specified client only. Typically, documents are delivered to the default client. The default client is the Broker client created for the Integration Server when the Integration Server first configures its connection to the Broker.

(31)

The following diagram illustrates the subscription path for a document delivered to the default client.

Subscribe path for documents delivered to the default client

Note: If a publishing service specifies an individual trigger as the destination of the document (the publishing service specifies a trigger client ID as the destination ID), the subscribe path the document follows is the same as the path followed by a published document. Step Description 1 The dispatcher on the Integration Server requests documents from the default client’s queue on the Broker. Note: The default client is the Broker client created for the Integration Server. The Broker places documents in the default client’s Broker queue only if the Dispatcher Trig ger Serv ice X1 5 6 6 1 Client Queue Default Client 4

Memory

2

Trigger

Queue X Queue YTrigger Trigger Document Store

Trig ger Se rvic e X2 Trig ger Serv ice Y1 Trigger Service Y2

Default Document Store 3 7 7 8 8 Guaranteed Storage

(32)

2 An Overview of the Publish and Subscribe Paths 2 The thread retrieves documents delivered to the default client in batches. The number of documents the thread retrieves at one time is determined by the capacity and refill level of the default document store and the number of documents available for the default client on the Broker. For more information about configuring the default document store, see the webMethods Integration Server Administrator’s Guide. 3 The dispatcher places a copy of the documents in memory in the default document store. 4 The dispatcher identifies subscribers to the document and routes a copy of the document to each subscriber’s trigger queue. In the case of delivered documents, the Integration Server saves the documents to a trigger queue. The trigger queue is located within a trigger document store that is saved on disk. 5 The Integration Server removes the copy of the document from the default document store and, if the document is guaranteed, returns an acknowledgement to the Broker. The Broker removes the document from the default client’s queue. 6 The dispatcher obtains a thread from the server thread pool, pulls the document from the trigger queue, and evaluates the document against the conditions in the trigger. Note: If exactly‐once processing is configured for the trigger, the Integration Server first determines whether the document is a duplicate of one already processed by the trigger. The Integration Server continues processing the document only if the document is new. 7 If the document matches a trigger condition, the Integration Server executes the trigger service associated with that condition. If the document does not match a trigger condition, the Integration Server, sends an acknowledgement to the trigger queue, discards the document (removes it from the trigger queue), and returns the server thread to the server thread pool. The Integration Server also generates a journal log message stating that the document did not match a condition. Step Description

(33)

2 An Overview of the Publish and Subscribe Paths Notes: The Integration Server saves delivered documents in a trigger document store located on disk. The Integration Server saves published documents in a trigger document store located in memory. If the Integration Server shuts down before processing a guaranteed document saved in a trigger document store on disk, the Integration Server recovers the document from the trigger document store when it restarts. Volatile documents are saved in memory and are not recovered up restart. If a service generates audit data on error and includes a copy of the input pipeline in the audit log, you can use webMethods Monitor to re‐invoke the trigger service at a later time. For more information about configuring services to generate audit data, see the webMethods Developer User’s Guide. It is possible that a document could match more than one condition in a trigger. However, the Integration Server executes only the service associated with the first matched condition. The processing mode for a trigger determines whether the Integration Server processes documents in a trigger queue serially or concurrently. In serial processing, the Integration Server processes the documents one at a time in the order in which the 8 After the trigger service executes to completion (success or error), one of the following occurs: If the trigger service executed successfully, the Integration Server returns an acknowledgement to the trigger queue (if this is a guaranteed document), removes the document from the trigger queue, and returns the server thread to the thread pool. If a service exception occurs, the trigger service ends in error and the Integration Server rejects the document, removes the document from the trigger queue, returns the server thread to the thread pool, and sends an error document to indicate that an error has occurred. If the document is guaranteed, the Integration Server returns an acknowledgement to the trigger queue. The trigger queue removes its copy of the guaranteed document from storage. If a transient error occurs during trigger service execution and the service catches the error, wraps it and re‐throws it as an ISRuntimeException, then the Integration Server waits for the length of the retry interval and re‐ executes the service using the original document as input. If the Integration Server reaches the maximum number of retries and the trigger service still fails because of a transient error, the Integration Server treats the last failure as a service error. For more information about retrying a trigger service, see “Configuring Transient Error Handling” on page 134. Step Description

(34)

2 An Overview of the Publish and Subscribe Paths necessarily in the same order in which the documents were placed in the queue. For more information about document processing, see “Selecting Messaging Processing” on page 128. If a transient error occurs during document retrieval or storage, the audit subsystem logs the document and assigns it a status of FAILED. You can use webMethods Monitor to find and resubmit documents with a FAILED status. For more information about using webMethods Monitor, see the webMethods Monitor documentation. You can configure a trigger to suspend and retry at a later time if retry failure occurs. Retry failure occurs when Integration Server makes the maximum number of retry attempts and the trigger service still fails because of an ISRuntimeException. For more information about handling retry failure, see “Handling Retry Failure” on page 136.

Overview of Local Publishing

Local publishing refers to the process of publishing a document within the Integration Server. Only subscribers located on the same Integration Server can receive and process the document. In local publishing, the document remains within the Integration Server. There is no Broker involvement. Local publishing occurs when the service that publishes the document specifies that the document should be published locally or when the Integration Server is not configured to connect to a Broker. The following diagram illustrates how the publish and subscribe paths for a locally published document.

(35)

Publishing a document locally

Step Description 1 A publishing service on the Integration Server sends a document to the dispatcher. Before the Integration Server sends the document to the dispatcher, it validates the document against its publishable document type. If the document is not valid, the service returns an exception specifying the validation error. 2 The dispatcher does one of the following: The dispatcher determines which triggers subscribe to the document and places a copy of the document in each subscriber’s trigger queue. The dispatcher saves locally published documents in a trigger document store located on disk. If there are no subscribers for the document, the dispatcher discards the document. Trigger Service 3 3 1 2 webMethods Integration Server

Trigger

Queue X Queue ZTrigger

Trigger Document Store

Trigger Service Trigger Service Trigger Service

Dispatcher 4 4 5 5 Publishing Service Trigger Queue Y

(36)

2 An Overview of the Publish and Subscribe Paths Notes: You can configure publishable document types and Integration Server so that Integration Server does not validate documents when they are published. For more information about validating publishable document types, see “Specifying Validation for a Publishable Document Type” on page 68. Integration Server saves locally published documents in a trigger document store located on disk. If Integration Server shuts down before processing a locally 3 The dispatcher obtains a thread from the server thread pool, pulls the document from the trigger queue, and evaluates the document against the conditions in the trigger. Note: If exactly‐once processing is configured for the trigger, the Integration Server first determines whether the document is a duplicate of one already processed by the trigger. The Integration Server continues processing the document only if the document is new. 4 If the document matches a trigger condition, the dispatcher executes the trigger service associated with that condition. If the document does not match a trigger condition, the Integration Server sends an acknowledgement to the trigger queue, discards the document (removes it from the trigger queue), and returns the server thread to the server thread pool. 5 After the trigger service executes to completion (success or error), one of the following occurs: If the trigger service executed successfully, the Integration Server sends an acknowledgement to the trigger queue (if this is a guaranteed document), removes the document from the trigger queue, and returns the server thread to the thread pool. If a service exception occurs, the trigger service ends in error and the Integration Server rejects the document, removes the document from the trigger queue, and returns the server thread to the thread pool. If the document is guaranteed, the Integration Server sends an acknowledgement to the trigger queue. If a transient error occurs during trigger service execution and the service catches the error, wraps it and re‐throws it as an ISRuntimeException, then the Integration Server waits for the length of the retry interval and re‐executes the service using the original document as input. If Integration Server reaches the maximum number of retries and the trigger service still fails because of a transient error, the Integration Server treats the last failure as a service error. For more information about retrying a trigger service, see “Configuring Transient Error Handling” on page 134. Step Description

(37)

2 An Overview of the Publish and Subscribe Paths published guaranteed document, Integration Server recovers the document from the trigger document store when it restarts. Integration Server does not recover volatile documents when it restarts. If a subscribing trigger queue reaches its maximum capacity, you can configure Integration Server to reject locally published documents for that trigger queue. For more information about this feature, see the description of the watt.server.publish.local.rejectOOS parameter in the webMethods Integration Server Administrator’s Guide. If a service generates audit data on error and includes a copy of the input pipeline in the audit log, you can use webMethods Monitor to re‐invoke the trigger service at a later time. For more information about configuring services to generate audit data, see the webMethods Developer User’s Guide. It is possible that a document could match more than one condition in a trigger. However, Integration Server executes only the service associated with the first matched condition. The processing mode for a trigger determines whether the Integration Server processes documents in a trigger queue serially or concurrently. In serial processing, Integration Server processes the documents one at a time in the order in which the documents were placed in the trigger queue. In concurrent processing, the Integration Server processes as many documents as it can at one time, but not necessarily in the same order in which the documents were placed in the queue. For more information about document processing, see “Selecting Messaging Processing” on page 128. You can configure a trigger to suspend and retry at a later time if retry failure occurs. Retry failure occurs when Integration Server makes the maximum number of retry attempts and the trigger service still fails because of an ISRuntimeException. For more information about handling retry failure, see “Handling Retry Failure” on page 136. You can configure Integration Server to strictly enforce a locally published document’s time‐to‐live and discard the document before processing it if the document has expired. For more information about this feature, see the description of the watt.server.trigger.local.checkTTL parameter in the webMethods Integration Server Administrator’s Guide.

(38)

(39)

3

Steps for Building a Publish-and-Subscribe Solution

Introduction . . . 40 Step 1: Research the Integration Problem and Determine Solution . . . 41 Step 2: Determine the Production Configuration . . . 41 Step 3: Create the Publishable Document Type . . . 41 Step 4: Make the Publishable Document Types Available . . . 42 Step 5: Create the Services that Publish the Documents . . . 42 Step 6: Create the Services that Process the Documents . . . 43 Step 7: Define the Triggers . . . 43 Step 8: Synchronize the Publishable Document Types . . . 43

(40)

3 Steps for Building a Publish-and-Subscribe Solution

Introduction

There are two sides of a publish‐and‐subscribe model integration solution. One side is the publishing side and the other is the subscribing side. The table below lists what you must create for each side of the integration solution. The following table lists the tasks that you need to perform to build an integration solution and whether the publishing side or the subscribing side is responsible for the task.

On the publishing side, create: On the subscribing side, create:

Publishable document types for the documents that are to be published Services that publish the documents Services to process the incoming documents that are published by the publishing side Triggers that associates the incoming documents with services that processes the documents

Step Task Publishing Subscribing

1 Research the integration problem and determine how you want to resolve it. D D 2 Determine the development environment. D D 3 Create the publishable document types for the documents to be published. D 4 Make the publishable document types available to the subscribing side. D D 5 Create the services that publish the documents. D 6 Create the services that process the published documents. D 7 Define the triggers that associate the published documents to the services that processes the document. D 8 Synchronize the publishable document types if necessary. D D

7-1-1_Publish_Subscribe_Developers_Guide

Publish-Subscribe

Developer’s Guide

Version 7.1.1

January 2008

Table of Contents

About this Book

Document Conventions

Additional Information

1

An Introduction to the Publish-and-Subscribe Model

Introduction

What Is the Publish-and-Subscribe Model?

webMethods Components

Integration Server

Broker

Basic Elements in the Publish-and-Subscribe Model

Documents

Publishable Document Types

Triggers (Broker/Local Triggers)

Services

Adapter Notifications

Canonical Documents

2

An Overview of the Publish and Subscribe Paths

Introduction

Overview of the Publishing Path

Publishing Documents to the Broker

Publishing Documents When the Broker Is Not Available

Publishing Documents and Waiting for a Reply

Overview of the Subscribe Path

The Subscribe Path for Published Documents

The Subscribe Path for Delivered Documents

Overview of Local Publishing

3

Steps for Building a Publish-and-Subscribe Solution

Introduction