Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition IBM Corporation







Full text


© 2011 IBM Corporation

Managed File Transfer Solutions using

DataPower and WebSphere


© 2011 IBM Corporation 2

 Introduction

 MQ FTE Overview

 DataPower File handling Overview

 File transfer scenarios with DataPower / MQ FTE


© 2011 IBM Corporation

Head Office

Head Office Regional LocationsLocationsRegional

Subsidiaries Subsidiaries Stores Stores Branches Branches Inventory Inventory Orders Orders Accounting Accounting Regional Offices Regional Offices Warehouses Warehouses Packaged Apps Packaged Apps CRM CRM Sales Sales

Moving Files and Documents

around internally within Head-office

Moving Files and Documents

between Head-office and


© 2011 IBM Corporation



– Typically File Transfer Protocol (FTP) is combined with writing and maintaining homegrown code to address its limitations

Why is FTP use so widespread?

– FTP is widely available – Lowest common denominator – Promises a quick fix – repent at leisure

– Simple concepts – low technical skills needed to get started – FTP products seem “free”, simple, intuitive and ubiquitous  Legacy File Transfer products

– A combination of products often used to provide silo solutions

– Often based on proprietary versions of FTP protocol – Can’t transport other forms of data besides files

– Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure – especially with SOA


– From IT Staff to Business staff and even Security Personnel – Using a combination of email, fax, phone, mail, memory


Most organizations rely on a mix of homegrown code, several legacy products and different technologies … and even people!


© 2011 IBM Corporation






Limited visibility

and traceability



Unreliable delivery – Lacking checkpoint restart – Files can be lost

Transfers can terminate without notification or any record – corrupt or partial files can be accidentally used

File data can be unusable after transfer – lack of Character Set conversion

Often usernames and

passwords are sent with file – as plain text!

Privacy, authentication and encryption often not be available

Non-repudiation often lacking

Transfers cannot be monitored and managed centrally or remotely

Logging capabilities may be limited and may only record transfers between directly connected systems

Cannot track the entire journey of files – not just from one machine to the next but from the start of its journey to its final destination

Changes to file transfers often require updates to many ftp scripts that are typically scattered across machines and require platform-specific skills to alter

All resources usually have to be available concurrently

Often only one ftp transfer can run at a time


© 2011 IBM Corporation


Auditable Who transferred a file? Where? When? Was it this file?

Reliable Automatic resumption of interrupted transfers. No partial file data left lying around

Secure Limits access to authorized users. Protects file data in transit

Automated Designed for “lights out” operation

Centralized Can be monitored and managed from one central location

Any file size Imposes no practical limits on file sizes. Efficient regardless of file size

Integrated Integrates well with applications that typically perform file processing

Cost Effective Reuses existing skills and infrastructure

No agreed specification for managed file transfer products to certify


General consensus that managed file transfer involves the following:


© 2011 IBM Corporation 7

 Introduction

 MQ FTE Overview

 DataPower File handling Overview

 File transfer scenarios with DataPower / MQ FTE


© 2011 IBM Corporation


Auditable Full logging and auditing of file transfers + archive audit data to a database

Reliable Checkpoint restart. Exploits solid reliability of WebSphere MQ

Secure Protects file data in transit using SSL. Provides end-to-end encryption using AMS

Automated Providing scheduling and file watching capabilities for event-driven transfers

Centralized Provides centralized monitoring and deployment of file transfer activities

Any file size Efficiently handles anything from bytes to terabytes

Integrated Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct

Cost Effective Reuses investment in WebSphere MQ. Wide range of support (inc. z/OS and IBM i)



WebSphere MQ File Transfer Edition


© 2011 IBM Corporation


Traditional approaches to file transfer

result in parallel infrastructures

– One for files – typically built on FTP

– One for application messaging – based

on WebSphere MQ, or similar

High degree of duplication in creating

and maintaining the two infrastructures

File Transfer Edition reuses the MQ

network for managed file transfer and


– Operational savings and simplification

– Reduced administration effort

– Reduced skills requirements and


File Transfers



Consolidated Transport

for Messages & Files


© 2011 IBM Corporation


Applications exchanging file data


– The endpoints for managed file transfer operations


– Send instructions to agents

Log database

– A historical record of file transfers

Coordination queue manager

– Gathers together file transfer events

WebSphere MQ

Agent Agent Agent

“Coordination” Queue Manager





© 2011 IBM Corporation



 Act as the end points for file transfers

 Long running MQ applications that transfer files by splitting them into MQ messages

– Efficient transfer protocol avoids excessive use of MQ log space or messages building up on queues

 Multi-threaded file transfers

– Can both send and receive multiple files at the same time

 Generate a log of file transfer activities which is sent to the “coordination queue manager”

– This can be used for audit purposes

 Associated with one particular queue manager (either v6 or v7)

– Agent state on queues

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination” Queue Manager





© 2011 IBM Corporation



 FTE agent processes define the end-points for file transfer. That is to say that if you want to move files off a machine, or onto a machine – that machine would typically need to be running an agent process

 Agent processes are long running MQ applications that oversee the process of moving file data in a managed way. Each agent monitors a ‘command’ queue waiting for

messages which instruct it to carry out work, for example file transfers

 The FTE agent process needs connectivity to an MQ queue manager to do useful work. It can connect either directly to a queue manager running on the same system, or as an MQ client using an embedded version of the MQ client library (which is kept completely separate to any other MQ client libraries that may or may not already have been installed onto the system)*

– Each agent requires its own set of MQ queues – which means that an

agent is tied to the queue manager where these queues are defined

– However – one queue manager can support multiple agents

* Note: availability of direct (bindings) connectivity or MQ client based connectivity is dependent on the version of MQ FTE in use

• WebSphere MQ File Transfer Edition on z/OS does not support the MQ client style of connectivity

• File Transfer Edition on distributed platforms has a ‘server’ and ‘client’ offering. The agent component of the ‘client’ offering is restricted to only supporting MQ client style connectivity. The agent component of the ‘server’ offering may be used either connectivity options







© 2011 IBM Corporation


1. Application writes file to file system

Existing Application WMQ FTE Agent WMQ FTE Agent Existing Application *tap*

2. Agent monitors file system, spots arrival of file and based on rules, transfers the file

3. FTE transports file to destination

4. At destination MQ FTE writes file to file system

5. FTE can also start another application to process the file


© 2011 IBM Corporation


Monitoring & Program Execution

 Resource monitors work in two stages:

1. Poll a resource (in this case the file system – but as we’ll see later the ‘resource’ can also be an MQ queue) and identify that a condition has been met (perhaps the appearance of a file matching a particular pattern)

2. Perform an action (which can include starting a managed file transfer, or running a script), optionally propagating information about the resource (for example the name of the file triggered on) into the action

 As shown on the previous slide, resource monitors are typically used to provide integration with an existing system without needing to make changes to the system

 Another function of FTE, used for integration with existing systems is the ability to execute programs or scripts both on the source or destination systems for a file transfer. This can be used to:

– Start a program, on the source system, which generates the file data to be transferred prior to performing the managed file transfer

– Start, or notify, a program on the destination system when the file data has been transferred – allowing it to process the data without having to poll






© 2011 IBM Corporation



Send instructions to agents

and display information about

agent configuration

Via MQ messages

Many implementations of


MQ Explorer plug-in

Command line programs

Open scripting language


Documented interface to

program to

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination” Queue Manager





© 2011 IBM Corporation



 “Commands” is the name we have given to anything which instructs the FTE agent. As described on the previous slide, there are a wide range of command

implementations including graphical and non-graphical command-line based commands

 Commands instruct the FTE agent by sending it messages. The messages

themselves use a documented format which can easily be incorporated into your own applications

 The commands that are supplied with FTE can connect either as an MQ client (again based on embedded client libraries) or directly to a queue manager located on the same system







© 2011 IBM Corporation


Log Database

Keeps a historical account of

transfers that have taken place

Who, where, when… etc.

Implemented by the ‘database

logger’ component which

connects to the coordination

queue manager

Stand alone application

Or JEE application

Queryable via Web Gateway

Also a documented interface

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination” Queue Manager





© 2011 IBM Corporation


Log Database

 File Transfer Edition can record a historical account of file transfers to a database using the ‘database logger’ component.

– This component is available to run as a stand alone process or as a JEE application

 The information used to populate the log database is generated, as MQ messages, by the FTE agents participating in file transfers. This is routed to a collection point in the MQ network, referred to as the ‘coordination queue manager’ (see next slides). The database logger component subscribes to the messages produced by agents and reliably enters them into a database.






© 2011 IBM Corporation


Coordination Queue Manager

Gathers together information

about events in the file transfer


Not a single point of failure

Can be made highly available

Messages stored + forwarded

MQ v7 publish / subscribe

Allows multiple log databases,

command installs

Documented interface

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination” Queue Manager





© 2011 IBM Corporation


Coordination queue manager

 The coordination queue manager is used as the gathering point for all the information about file transfers taking place between a collection of FTE agents

 The queue manager uses publish/subscribe (so it must be MQ version 7) to distribute this information to “interested parties” which typically include:

– The WebSphere MQ Explorer plug-in, which provides a graphical overview of FTE activity

– The database logger component, which archives the information to a database – Some of the command line utilities which are part of the FTE product

 The format used to publish information is documented and can be used to develop 3rd

party applications which process this data

 Although there is only a single ‘coordination’ queue manager for a given collection of agents it does not represent a point of failure:

– MQ stores and forwards messages to the coordination queue manager when it is available – so if the coordination queue manager is temporarily unavailable no log data is lost

– The ‘coordination’ queue manager can be made highly available using standard HA techniques such as MQ multi-instance queue manager or via a HA product such as PowerHA







© 2011 IBM Corporation  Command Line tools

– For operations (for example transferring a file) – Administration (for example starting an agent)


– eclipse based plug-in to MQ Explorer


© 2011 IBM Corporation

Page 22

 Command Line Interface is consistent across all supported platforms

 Transfer commands can be invoked from the supported Operating Systems shell environment

 Commands can be invoked from anywhere across the file transfer network

– i.e. Command could be invoked from a

Windows machine for transfers taking place between z/OS and Unix machines

 Developers can use any native command line language on the OS that can invoke these commands (shell, bat, cmd, etc.)

 Application Programs can place a request using a messaging interface in XML

 Examples:

fteCreateTransfer Starts a new file transfer from the command line

fteStartAgent Starts a File Transfer agent from the command line

fteStopAgent Stops a File Transfer agent in a controlled way

fteShowAgentDetails Displays the details of a particular File Transfer agent


© 2011 IBM Corporation integrated into


© 2011 IBM Corporation

Page 24

Choose when to start the scheduled transfer

Choose when to repeat the scheduled transfer and how often

Choose the trigger for the transfer

Choose advanced options


© 2011 IBM Corporation


© 2011 IBM Corporation 26

 Introduction

 MQ FTE Overview

 DataPower File handling Overview

 File transfer scenarios with DataPower / MQ FTE


© 2011 IBM Corporation Internal Employees Wholesalers Retailer Affiliates B2B B2C


© 2011 IBM Corporation 28


InfoSphere Guardium Real-time Database Monitoring and Security Proventia Management SiteProtector System

Proventia Network Enterprise Scanner Proventia Network Multi-Function Security Security Network Active Bypass

Security Network Intrusion Prevention System WebSphere DataPower XML Security Gateway


WebSphere Cast Iron Cloud Integration WebSphere DataPower B2B Appliance WebSphere DataPower Integration


WebSphere DataPower Caching Appliance WebSphere DataPower Low Latency Appliance WebSphere DataPower XML Accelerator

Power7 with WebSphere and DB2 Smart Analytics Optimizer


Application Manager for Smart Business IBM CloudBurst

Rational Power Appliance Express Rational Power Appliance Workgroup Rational Power Appliance Enterprise Service Manager for Smart Business Smart Business Appliance

Virtual Desktop for Smart Business WebSphere CloudBurst Appliance


Information Archive SAN Volume Controller

Scale Out Network Attached Storage System Storage Productivity Center Real Time Compression Appliance


Netezza Retail Analytic Appliance Netezza Skimmer

Netezza TwinFin


© 2011 IBM Corporation





© 2011 IBM Corporation

1) Take some characteristics from integration software stacks

2) Take some characteristics of networking & firewall devices

3) Combine them into a hardware device (aka hardware form factor) that has characteristics of both application stacks and networking gear.

• Programmability by developers

• Message level routing and security (including by partner for B2B)

• Integration with other SOA infrastructure & governance software such as Registry, Repository, etc…

• Conditional Processing (if, else, split, join)

• Transformation of any-2-any formats

• High throughput

• Many concurrent connections

• IP level security and routing (can’t see inside message but route by source and target)

Simplified Management and reduced maintenance costs through firmware updates

• Reliability


© 2011 IBM Corporation  Simple architecture:

– microcode firmware + purpose-built hardware  Delivered from the factory with everything you

need to connect to the network and start working

– No need to provision anything but the Ethernet network and CAT cables to get started

 All computationally-significant components sealed within a temper-proof casing

– Chips – Memory

– Boards and cards

– Flash-based file system (signed and encrypted) – Parsing and xform accelerators (patented) – Cryptographic accelerators (patented)


© 2011 IBM Corporation


Business Requirements

Configuration driven, not programmable

Relief from maintaining diverse development and system

administrator skills

Simplified IT deployments

Appliances are

Fit for Purpose, Easily Consumable

A drop-in, “plug-n-work” device

Hardened, with numerous functions integrated into a single device

Easy configuration-driven set up and management

Purpose-built hardware / software that delivers on specific business



© 2011 IBM Corporation

• 2U high density rackmount design

• New customer-replaceable Ethernet modules with far more ports than the current generation (4)

– Two 10 Gbps ports – Eight 1 Gbps ports

 Higher performance chips, memory, flash, and hard drives

 Far more memory than ever before

– Memory aggregation at the domain level (announced for firmware 4.01)

 Enhanced hardware diagnostics

 Customizable intrusion detection

33 8 1-Gigabit Ethernet NICs 8 1-Gigabit Ethernet NICs RAID mirroring and striping across four drives RAID mirroring and striping across four drives 2 10-Gigabit Ethernet NICs 2 10-Gigabit Ethernet NICs


© 2011 IBM Corporation

DataPower appliances have been the best selling WebSphere product since the DataPower company acquisition in 2005

Guiding philosophy is to take rote, repeatable integration tasks and lock them down in the appliance form factor, including

– Parsing (XML, Flat)

– Validation (XML, Flat)

– Transformation (XML, Flat)

– Security (Transport, XML, other)

– Integration (Multi-transport, SOA, ESB)

– B2B (Partner Mgt, Persistent Tx Store)



© 2011 IBM Corporation



Trusted Domain

Partner Apps Devices

3 B2B Gateway

Integration Gateway functions +

 Partner Management

 Enhanced qualities of service

1 Web Gateway

 Security policy enforcement

 SLA policy enforcement

 Web Application Firewall


System z


Web Apps 2 Integration Gateway

Web Gateway functions +

 Any-to-any transformation

 Protocol Bridging

2.1 ESB

Integration Gateway functions +

 Routing

 Database connectivity

 more...

2.2 Service Enablement

ESB functions +


© 2011 IBM Corporation


Proxying and Enforcement

• Terminate incoming connection

• Terminate transport-level security

• Enforce Service Level Agreement policies • Inspect message content, filter, pattern-match • Enforce security policies on message content • Call out to Access Control List(s)

• Detach binaries and call out to virus checker • Transform content (XSLT, XML-to-XML) • Establish a new connection to pass results

Connection from client

New connection to target


Virus Scanner

Partner App


© 2011 IBM Corporation



 All the same capabilities as the XS40 to proxy and protect traffic, PLUS…

 Support for additional transport protocols:

– FTP(s), SFTP(SSH), WebSphere MQ, WebSphere JMS, TIBCO EMS, MQ FTE

 Database connectivity (ODBC)

 File system access (NFS)

 Web 2.0 standards: ReST, JSON

 Onboard engine for parsing and transforming virtually any message format

– WebSphere Transformation Extender (WTX)



© 2011 IBM Corporation

 All of the capabilities of the XS40 to proxy and enforce policies

 All of the additional capabilities of the XI50 for protocol bridging and universal transformation, PLUS…

 Partner Management functions:

– Define partners with the web management console – Associate partners with network endpoints

– Attach metadata about the partners to their definitions

 Enhanced Qualities of Service

– Onboard persistent transaction store – Search messages by partner, time, etc – Replay messages if necessary

– ebXML/ebMS, AS1, AS2, and AS3 protocol bindings for greater reliability across traditionally unreliable protocols


© 2011 IBM Corporation

Purpose-built B2B hardware for simplified deployment, exceptional

performance and hardened security

Extend integration beyond the enterprise with B2B

Hardened Security for DMZ deployments

Easily manage and connect to trading partners

using industry standards

Simplified deployment and ongoing management

• Trading Partner Management for B2B Governance; B2B protocol policy enforcement, access control, message filtering, and data security

• Application Integration with B2B Gateway Service capabilities supporting B2B patterns for AS1, AS2, AS3, ebMS v2.0, FTP(S), SFTP, HTTP(S), SFTP/POP3, MQ, MQ FTE, JMS and more

• Full featured User Interface for B2B configuration and transaction viewing; correlate documents and acknowledgments displaying all associated events

• Simplified deployment, configuration and management providing a quicker time to value by establishing rapid connectivity to trading partners


© 2011 IBM Corporation A pp lic ati o ns DMZ Trusted Domain

Standalone Consolidated Patterns

– Deploy standalone for purpose built B2B gateway functionality in the DMZ utilizing exceptional security and B2B transaction volumes for quickly connecting to trading partners

Enterprise B2B Integration Pattern

– Deploy with MQFTE for B2B enabled Managed File Transfer – Deploy with WTX-TM for end-to-end EDI Processing – Deploy as B2B entry point for BPM and ESB solutions

– Supplement WPG or Sterling Integrator by offloading security functions and advanced web services


WebSphere DataPower XB60

WebSphere Partner Gateway / Sterling Integrator

WebSphere Transformation Extender / Trading Manager




© 2011 IBM Corporation 41

 Introduction

 MQ FTE Overview

 DataPower File handling Overview

 File transfer scenarios with DataPower / MQ FTE


© 2011 IBM Corporation

to-end file transfer

Browser (LOB User) XB60





g P




XB60 B2B Gateway Service Transaction Viewer Multi-Protocol Gateway Service Profile Mgmt StoreData Internet Browser (Admin) Browser (Partner view) Server MQ Explorer FTE Viewer WS MQ Agent01 Server WS MQ Agent02 Data Store Application WAN 1 2 3 4b 4a 6 5



© 2011 IBM Corporation 43 ... ... ... ... ... ... ...

If the payload of the received message is too big, the message payload will be segmented into smaller pieces before forwarding to the queue


© 2011 IBM Corporation 44

Partner File Transfer

Sender SourceAgent NetworkMQFTE DestinationAgent

Appliance MQ queue

DataPower portion of transfer MQ File Transfer Edition portion of transfer

Logical flow of data

1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower

2. Destination agent places file data as one or more messages in a group.

3. MQFTE front-side handler consumes message(s) from the queue, reassemble the message if there is more than one segments, and sends the data to the external partner


© 2011 IBM Corporation 45

 Introduction

 MQ FTE Overview

 DataPower File handling Overview

 File transfer scenarios with DataPower / MQ FTE


© 2011 IBM Corporation




MQ queue : FTE2DP FTE Agent : APC QMGR : QMPC


© 2011 IBM Corporation




MQ queue : FTE2DP FTE Agent : APC QMGR : QMPC


© 2011 IBM Corporation 48

 MQFTE Source Agent can optionally configure the following metadata when integrating

with XB60.

 When set, B2B Gateway will use these information to match the right partner profile settings.

DPMQFTESenderID : the business ID of internal partner

DPMQFTEReceiverID : the business ID of external partner


© 2011 IBM Corporation




MQ queue : FTE2DP FTE Agent : APC QMGR : QMPC


© 2011 IBM Corporation 50

The Queue manager that host the shared queue The queue where we retrieve the message


© 2011 IBM Corporation 51

Partner File Transfer

Sender SourceAgent NetworkMQFTE DestinationAgent


Get Queue

DataPower portion of transfer MQ File Transfer Edition portion of transfer

Logical flow of data

1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower

2. Destination agent places file data as one or more messages in a group. 3. MQFTE front-side handler consumes message(s) from the queue

4. Error happens in transaction processing and MQFTE front-side handler rolls the transfer back and. Go back to step 1 to retry the transfer.

5. If the number of retries reaches the “backout threshold”, then MQFTE front-side handler will send the transfer to the “backout queue”.




4 (error)

Backout Queue


© 2011 IBM Corporation

Agent Network Agent



MQ queue


© 2011 IBM Corporation

Agent Network Agent

Appliance ... MQ queue dpmqfte://FTEWA/? RequestQueue=DP2FTE&DestAgent =APC& DestQM=QMPC&DestFile=testcompl ete.xml Network


© 2011 IBM Corporation 54

 Mandatory

– DestAgent : the destination agent in MQFTE network that will ultimately receive the file

– DestQM : the destination queue manager the source agent should send the messages to

– DestFile : the filename to use when the destination agent need to store the received

messages into file.

– Unique filename is supported.

 Optional

– SenderID : the business ID of sender (e.g. external partner )

– ReceiverID : the business ID of receiver (e.g. internal partner)

– ContentType : the content type of the message payload

 Example :



© 2011 IBM Corporation

Agent Network Agent

Appliance ... MQ queue agent="${DPMQFTEDestinationAgent}" QMgr="${DPMQFTEDestinationQM}" <file>c:\temp\$ {DPMQFTEDestinationFile}</file>



© 2011 IBM Corporation 57

1. Outbound gateway sends a B2B message to inbound gateway

2. The DataPower appliance writes data to an MQ queue as one or more messages in a group. The message ID assigned to the first message in the group becomes the Integration ID. Other information

can be passed in the RFH 2 header of the first message

3. FTE consumes the group of messages and uses the information present in the first message in the group to route the data to a back end system, where it is written as a file

4. B2B Viewer of the outbound gateway retrieve the transaction metadata from the MQFTE Logger Database

Back-end System

Agent Network Agent


MQ queue

DataPower portion

of transfer MQ File Transfer Edition portion of transfer

   Outbound GW Inbound GW MQFTE Logger 4


© 2011 IBM Corporation 58

 A new column to store the integration ID which could be used to correlate a B2B Transaction with a backend transaction

 For MQFTE integration, the integration ID maps to the “transfer ID” of MQFTE network

 For inbound, the integration-ID is the Message ID of first message we sent to the shared queue

 For outbound, the integration-ID is the Message ID of the first message we received from the shared queue


© 2011 IBM Corporation 59

 A pop-up to display the MQFTE metadata when the integration-id is clicked ( for MQFTE-related transaction only)

 Depending on weather the transaction is inbound or outbound, the popup window will show different metadata


© 2011 IBM Corporation  Integration between the B2B appliance and backend application is over WebSphere MQ

instead of a shared file system.

 Files transferred between the B2B appliance and MQ FTE can be correlated using the integration ID from MQ; this ID can also be seen in the B2B Viewer.

 The combined B2B messaging flow through the B2B appliance and file transfer flow through MQ FTE can be viewed through the B2B Viewer on the appliance. This provides the user with an end-to-end view of the file transfer.

 File transfers can be set up to occur at specified times or dates, or repeated at specified intervals. File transfers can also be triggered by a range of system events, such as new files or updated files.