Talend Platform for Data Management. Installation and Upgrade Guide 5.2.1

(1)

Talend Platform

for Data Management

Installation and Upgrade Guide

(2)

Talend Platform for Data Management

Notices

All brands, product names, company names, trademarks and service marks are the properties of their respective owners.

(3)

Talend Platform for Data Management Installation and Upgrade Guide

Preface

1. General information

1.1. Purpose

This Installation Guide explains how to install and configure Talend Platform for Data Management modules and related applications. For detailed explanation on how to use and fine-tune Talend

Platform for Data Management applications, please refer to the appropriate Administrator or User

Guides of Talend Platform for Data Management solutions.

Information presented in this document applies to release 5.2.1 of Talend Platform for Data

Management.

1.2. Audience

This guide is devoted for administrators of Talend Platform for Data Management solutions. The layout of GUI screens provided in this document may vary slightly from your actual GUI.

1.3. Typographical conventions

This guide uses the following typographical conventions:

• text in bold: window and dialog box buttons and fields, keyboard keys, menus, and menu and options,

• text in [bold]: window, wizard, and dialog box titles, • text in courier: system parameters typed in by the user, • text in italics: file, schema, column, row, and variable names, •

The icon indicates an item that provides additional information about an important point. It is also used to add comments related to a table or a figure,

•

The icon indicates a message that gives information about the execution requirements or recommendation type. It is also used to refer to situations or information the end-user needs to be aware of or pay special attention to.

Any command is highlighted with a grey background or code typeface.

2. Feedback and Support

Your feedback is valuable. Do not hesitate to give your input, make suggestions or requests regarding this documentation or product and find support from the Talend team, on Talend’s Forum website at:

(6)

Feedback and Support

(7)

Chapter 1. Prior to installing the Talend

products

This chapter provides useful information on software and hardware prerequisites you should be aware of, prior to starting the installation of the Talend modules.

There are two ways to install the Talend solutions. You can either install them manually, one by one, or use the Talend Installer wizard to get out-of-the-box solutions. The sections below provide information on both installation modes.

In the following documentation:

• certified: designates an environment certified by Talend through QA tests prior to the release of the software;

• supported: designates an environment that can be provided by Talend for problem reproduction and testing within 24 hours;

• supported with limitations: designates an environment that is supported by Talend under certain conditions explained in notes.

(8)

License key

1.1. License key

You should have received an email from Talend, including the following information: • your personal license key in a file with no extension:

The license key is mandatory to be able to access each module of Talend Platform for Data Management. Keep

this file at hand in a safe place.

For more information on how to use your license key, see section Configuring the Web application access and section Configuring the Talend Studio.

The availability of different perspectives in the Studio you want to install and use depends on your license or on the type of the remote project to which you connect. For further information on licenses and perspectives in the Studio, see the appropriate Administrator or User Guide.

• the software parts in archive files:

Unzip each archive file at the root or in a short path of the machine where they are to be deployed (Talend Studio on a developer’s machine and Talend Administration Center on the execution server for example). If you are on Windows, we recommend that you use 7-zip to unzip files, as it handles long file names well. • the documentation (including the one you are reading now):

Each module of Talend has a dedicated Administrator or User Guide available in pdf.

1.2. Installation requirements

To make the most out of the Talend products you subscribed to, please consider the following hardware and software requirements.

1.2.1. Memory usage

Memory usage heavily depends on the size and nature of your Talend projects. However, in summary, if your Jobs include many transformation components, you should consider upgrading the total amount of memory allocated to your servers, based on the following recommendations.

Product Client/Server Recommended alloc. memory

Talend Administration Center + Talend Activity Monitoring Console Web application

Server 4GB minimum, 8GB recommended

CommandLine Server 2GB minimum, 5 GB recommended

JobServer Server 1GB minimum, more recommended1

Studio Client 3GB minimum, 4 GB recommended

Talend Runtime Server 2GB minimum, 4 GB recommended1

1. Memory requirements depend on the executed processes.

1.2.2. Disk usage

(9)

Environment variable configuration

Talend Platform for Data Management Installation and Upgrade Guide 3

Product Client/Server Required disk space

for installation

Required disk space for use

Talend Administration Center with Talend Artifact Repository (for Software update) + Talend Activity Monitoring Console Web application

Server 800MB + more than

50MB for Talend Activity Monitoring

Console Web

application

800MB minimum + project size = 20GB+ recommended

CommandLine Server 3GB 2GB minimum + project size = 20

GB+ recommended

JobServer Server 20MB 2GB minimum + Jobs deployed =

20+ GB recommended

Studio Client 3GB 3+ GB

Talend Runtime Server 400MB 400+ MB

1.2.3. Environment variable configuration

Prior to installing your Talend solutions, you have to set the JAVA_HOME Environment variable: 1. Define your JAVA_HOME environment variable so that it points to the JDK directory.

For example, if the JDK path is C:\Java\JDKx.x.x\bin, you must set the JAVA_HOME environment variable to point to: C:\Java\JDKx.x.x.

It is highly recommended that the full path to the server installation directory is as short as possible and does not contain any space character. If you already have a suitable JDK installed in a path with a space, you simply need to put quotes around the path when setting the values for the environment variable.

If you use Talend Installer, you also have to set the Path system variable:

2. Add the previously defined JAVA_HOME variable to the Path environment variable. For example, <PathVariable>;%JAVA_HOME%\bin.

1.2.4. Naming conventions for software packages

This section applies only to manual installations.

In the email you received from Talend are listed a number of links to software modules for you to download. The file naming convention is as follows:

Zip/jar file naming convention Description

Talend-Installer-Starter-rYYYYY-VA.B.C.zip Talend Installer: wizard-based application which guides you step by step through the installation and configuration of the Talend modules.

Talend-Studio-rYYYYY-VA.B.C.zip CommandLine interface to the IDE + Studio IDE (GUI)

Talend-AdministrationCenter-rYYYYY-VA.B.C.zip Talend Administration Center: Web-based application used to administrate Talend Platform for Data Management projects and users + Talend Artifact Repository.

Talend-JobServer-rYYYYY-VA.B.C.zip JobServer: Standalone execution server

Talend-Runtime-VA.B.C.zip Talend Runtime: OSGi Container including JobServer. Talend Runtime is a standalone equivalent to the Talend ESB OSGi Container (container folder) of Talend ESB.

Talend-AMC_Web-rYYYYY-VA.B.C.zip Talend Activity Monitoring Console Web application used to monitor Talend Jobs and projects.

(10)

Talend Installer specific prerequisites

Zip/jar file naming convention Description

Talend-BRMS-rYYYY-VA.B.C.zip Drools: Business Rules Management System based on JBoss Rules Governor.

Talend-DQPortal-rYYYY-VA.B.C.zip Talend Data Quality Portal: portal which provides customizable web-based data quality monitoring and reporting.

Talend-DSC-rYYYY-VA.B.C.jar Talend Data Stewardship Console: stewardship tool that helps to manage manually data resolution and data integration tasks that are the output of data matching and data integrity processes. Where:

• YYYYY: Revision number,

• A.B.C.: Major. Minor. Patch: revision level if relevant.

The software modules must be all in the same versions/revisions! This means that both YYYY and A.B.C must match on both client side and server side.

1.3. Talend Installer specific prerequisites

Prior to launching the Talend Installer, please check that the following software and modules are properly set up on the station where you are to install the Talend modules:

• Operating System Linux 32/64 bits, Windows 32/64 bits or Mac OS X; • JDK 1.6.0 or higher must be installed;

Talend Installer automatically checks for JDK version 1.6 or higher version. If no instance of JDK is found, the installer will shut down.

• .NET version 2 or higher is required by Talend Installer to install services on Windows; • (optional) a mail server (to send notifications by email).

For Linux users:

Note that Talend Installer does not support the sdshell utility. IMPORTANT:

Talend Installer allows you to get out-of-the-box Talend solutions that do not require any manual installation.

However, these solutions are not provided in a production-ready environment as they may require additional configurations or optimizations according to your specific needs.

For example, you may want to change the H2 database that is embedded by default in Talend Administration

Center with your own database (MySQL or Oracle for example).

Talend Installer is used only for first installations of Talend solutions. Therefore, if you want to know more about the migration and upgrade processes, please refer to the chapter Upgrading your Talend products .

1.4. User interface and installation modes of

Talend Installer

This section provides information about the different installation modes that Talend Installer can run in. Note that the log files generated during the installation can be found in C:\Users\<username>\AppData\Local\Temp on Windows, and in /tmp/ on Linux.

(11)

User interface and installation modes of Talend Installer

Talend Platform for Data Management Installation and Upgrade Guide 5 Three main installation modes are available:

• The Full Graphic mode: allows full interactivity through a graphical user interface.

• The Text mode: provides full interactivity with users in the command line. It is equivalent to any GUI mode but the pages are displayed in text mode in a console.

The image below shows an example of text mode where the user enters the --mode text option from the command line on a OS X system:

This installation mode is only available on Unix platforms. It is automatically used if no graphical mode is available but it also can be forced using the --mode text command.

• The Unattended mode: is especially useful for automating the installation processes. This silent mode will perform an unattended installation that will not prompt the user for any information.

To perform this Unattended installation:

1. Write a simple .txt script in which you will define the options values:

The list of values available can be displayed by entering the help command in the command line.

In this example, the script details the silent installation of the Client type. The installation directory that will be created is called Install and the license file used is located in the C:/licenses directory.

2. Launch the silent installation using the --optionfile <filename> command, where <filename> is the name of the script which contains the list of pairs <key>=<value>.

An unattended installation is performed.

If you want to install Talend products as services via the Installer and you are on Windows Vista or Windows 7, you are required to run the application as Administrator OR to disable User Account Control.

For more information on these installation modes, please refer to the online Bitrock documentation: http:// installbuilder.bitrock.com/docs/installbuilder-userguide.html.

(12)

Studio specific prerequisites

1.5. Studio specific prerequisites

To use the Studio properly, you first need to install external programs specific to bulk components (if you want to use Oracle, Sybase, Informix or Ingres bulk functionality).

On Windows XP and Windows Server 2003, the GDI is already installed. However, on Windows 2000, this installation is required. The GDI can be downloaded from Microsoft’s Website. For further information, visit Eclipse’s FAQ.

1.5.1. Installing database client software (for bulk

mode)

Some bulk components, like Oracle, Sybase, Informix or Ingres, require database client software to run properly: • OracleBulkExec uses the sqlldr external utility. This utility is available in Oracle clients that must be installed

on the computer.

• Informix uses the dbload external utility. • Ingres uses the sql external utility.

• Sybase uses the bcp.exe external utility. This utility is asked for in the Sybase bulk components’ Basic Settings view. For more information, see tSybaseBulkExec, tSybaseOutputBulk and tSybaseOutputBulkExec components on the appropriate Talend Components Reference Guide.

1.6. Compatible Platforms

Despite our intensive tests, you might encounter some issues when installing our products on some Operating Systems.

Please refer to the grids below for a summary of supported OS and Java Runtime environments.

1.6.1. Talend Studio

On Linux, the xulrunner package is required to run the Studio. Please install mozilla-xulrunner192 – Mozilla Runtime Environment 1.9.2. and add the line

-Dorg.eclipse.swt.browser.XULRunnerPath=</usr/lib/xulrunner-1.9.2.17>

at the end of the .ini file that corresponds to your Studio (where </usr/lib/xulrunner-1.9.2.17> is the xulrunner installation path).

OS Name Version Processor Java JDK/JRE1 Support type

Unix Linux Ubuntu 12.04 64 bits Oracle Java 7 certified

Unix Linux Ubuntu 12.04 32/64 bits Oracle Java 6 supported

Unix Linux Ubuntu 11.10/10.04 32/64 bits Oracle Java 6/7 supported

Unix Redhat Linux Enterprise Server Edition/CentOS

5.3 to 5.6 32/64 bits Oracle Java 6 supported

Unix Redhat Linux Enterprise Server Edition/CentOS

6.X (>=6.1) 64 bits Oracle Java 6/7 supported

(13)

Talend server modules: Talend Administration Center, JobServer, CommandLine,Talend Runtime, Talend Data Quality Portal

Windows Microsoft Windows 8 64 bits Oracle Java 7 certified

Windows Microsoft Windows 7 64 bits Oracle Java 6 certified

Windows Microsoft Windows XP SP3 32/64 bits Oracle Java 6 supported

Windows Microsoft Windows Vista SP1 32/64 bits Oracle Java 6/7 supported

Windows Microsoft Windows 7 32 bits Oracle Java 6/7 supported

MAC OS MAC OS Lion/10.7 64 bits Oracle Java 6 certified2

MAC OS MAC OS Lion/10.7 64 bits Oracle Java 7 supported

MAC OS MAC OS Mountain

Lion/10.8

64 bits Oracle Java 6/7 supported

1. It is recommended to use a recent update of JDK 1.6 (Update 11 or higher). 2. Need to set security settings to accept non MAC-registered applications.

1.6.2. Talend server modules: Talend Administration

Center, JobServer, CommandLine,Talend Runtime,

Talend Data Quality Portal

Unix Linux Ubuntu 12.04 64 bits Oracle Java 7 certified

Unix Red Hat Linux Enterprise Server Edition

6.1 64 bits Oracle Java 7 certified

Unix SUSE SLES 11 64 bits IBM Java 6 certified2

Unix Linux Ubuntu 12.04 32/64 bits Oracle Java 6 supported

Unix Linux Ubuntu 11.10/10.04 32/64 bits Oracle Java 6/7 supported

Unix Red Hat Linux Enterprise Server Edition/CentOS

5.3 to 5.6 32/64 bits Oracle Java 6 supported

Unix Red Hat Linux Enterprise Server Edition/CentOS

6.X (>=6.2) 64 bits Oracle Java 6/7 supported

Unix SUSE SLES 10/11 32/64 bits Oracle Java 6/7 supported

Unix AIX 7.1 64 bits IBM Java 6 supported3

Unix Solaris (SunOS) 10/11 64 bits Oracle Java 6/7 supported

Windows Microsoft Windows Server 2008 R2 64 bits Oracle Java 6 certified

Windows Microsoft Windows Server 2012 64 bits Oracle Java 7 certified

Windows Microsoft Windows Server 2003 32/64 bits Oracle Java 6 supported

1. It is recommended to use a recent update of JDK 1.6 (Update 11 or higher).

2. Certification/support of IBM JDK is restricted to CommandLine, Jobserver. Not supported: Talend Administration Center, DQ Portal. No Installer support.

3. Only for the Runtime Container and the JobServer.

1.7. Compatible Databases

Please refer to the grids below for a summary of supported databases.

(14)

Talend Administration Center

1.7.1. Talend Administration Center

Databases Version Support type

MySQL 5.5 certified1 Oracle 11 certified2 H2 1.3 certified3 MySQL 5.0/5.1 supported MS SQL Server 2005 supported MS SQL Server 2008 supported Oracle 10 supported 1. http://www.mysql.com 2. http://www.oracle.com

3. http://www.h2database.com, H2 is provided by Talend with Talend Administration Center.

1.7.2. Talend Activity Monitoring Console

MySQL 5.5 certified Oracle 11 certified IBM DB2 10.1 certified MySQL 5.0/5.1 supported H2 1.3 supported1 MS SQL Server 2005 supported MS SQL Server 2008 supported Oracle 10 supported

Derby DB >10.8 supported with limitations2

1. Requires an H2 Database Server separated from Talend Administration Center.

2. Only for development and test purposes.

1.7.3. Talend Data Quality Portal

MySQL 5.0+ certified

Oracle 10g supported

HSQL 1.8.0 supported

If you are using MySQL, the my.ini file of your MySQL server installation directory must be edited in order to add the following line:

max_allowed_packet = 64M

The 64M value is only given as example. It represents the maximum size of a query packet the server can handle as well as the maximum query size the server can process. You can replace it with any value superior to 16M.

(15)

Talend Data Stewardship Console

1.7.4. Talend Data Stewardship Console

The table below indicates the Talend Data Stewardship Console embedded databases and their supported versions.

Databases Version

MySQL 4.1/5.01

HypersonicSQL 2.02

1. Support of versions 5.1 to 5.5 requires the installation of a Connector/J driver v.5.1.

2. Versions 1.61 to 1.8 are also supported.

It is possible to make Talend Data Stewardship Console work with other databases (HSQL, Oracle, SQLServer, etc.) by editing the properties of the tdsc-database.properties file which can be found in the <DSC>\WEB-INF\classes folder where <DSC> is the Talend Data Stewardship Console installation directory. For more information, see the section Replacing

the default database.

1.8. Compatible Runtime Containers

Please refer to the grids below for a summary of supported Web application server.

1.8.1. Talend Administration Center

Runtime Containers Version Support type

Apache Tomcat 6 certified

Apache Tomcat 7 supported

Weblogic 11 supported

JBoss Application Server 4.2.2 supported

1.8.2. Talend Data Quality Portal

Runtime Containers Version Support type

Apache Tomcat 6 certified

Apache Tomcat 7 supported

1.9. Compatible execution servers

Make sure the execution server version is compatible with the Talend Administration Center, CommandLine and Studio versions.

Note that the information contained in this section is valid at the date of publication, but may be subject to change at a later date.

(16)

Job servers (JobServer and Talend Runtime)

1.9.1. Job servers (JobServer and Talend Runtime)

JobServer JobServer & Talend Runtime (holding Job-Server)

JobServer & Talend Runtime (holding Job-Server)

JobServer & Talend Runtime (holding Job-Server) Version 4.2.x 5.0.x 5.1.x 5.2.x Talend Administration Center, Studio, CommandLine 4.2.x Talend Administration Center, Studio, CommandLine 5.0.x Talend Administration Center, Studio, CommandLine 5.1.x Talend Administration Center, Studio, CommandLine 5.2.x

1.9.2. ESB OSGI servers (Talend Runtime)

1.10. Compatible SVN

Please refer to the tables below for a summary of supported SVN clients/servers that you can use to store your projects.

Subversion client Version Support type

Apache Subversion (former Tigris) 1.7 or earlier supported1

1. http://subversion.apache.org

Subversion servers Version Support type

Visual SVN Server 2.5 or earlier certified1

Bitnami Subversion Stack based on SVN v.1.7 or earlier supported

Svnserve based on SVN v.1.7 or earlier supported

1. http://www.visualsvn.com/server/download

1.11. Compatible Web browsers

Please refer to the table below for a summary of supported Web browsers.

Web browser Talend Administration Center

Mozilla Firefox >= 3.6 certified

(17)

Port information

Web browser Talend Administration Center

Apple Safary 5 supported

Google Chrome >= 18 supported with limitations1 1. Graphical restrictions.

2. Graphical restrictions, a patch can be provided on demand.

1.12. Port information

This section describes the most important TCP/IP ports the Talend products use. Please make sure that your firewall configuration is compatible with these ports or change the default ports where needed.

Table Information :

Port: a TCP/IP port or a range of ports

Active: Active for a standard installation of the product (Standard Installation is defined here as Server or Client

installation using Talend Installer with the default values provided in the Installer User Interface)

Direction: In (Inbound); Out (Outbound) - related to the communication direction (e.g. a HTTP Port for a Service

we listen on request) will be an ‘Inbound’ port. A Browser who send a request e.g. to port 7080 will have this port as ‘Outbound’ port in this list.

Usage: which part of the Product component uses this port (e.g. 1099 is used by the JMX Monitoring component

of Talend Runtime).

Config: the file or location where the value can be changed. Remark: anything which is important to mention additionally.

1.12.1. Talend Studio and CommandLine

Port Direction Usage Config Remark

8002

Active: Y

IN CommandLine port Windows: commandline.bat

Linux: commandline-linux.sh MacOS: commandline-mac.sh

1.12.2. Talend Administration Center

8080

Active: Y

IN Talend Administration Center Server -Apache Tomcat HTTP Port

/conf/server.xml

8009

Active: Y

IN Talend Administration Center Server -Apache Tomcat AJP Connector Port

/conf/server.xml

(none)

Active: Y*

OUT Talend Administration Center Server -Database

Configuration Page in TAC Web-UI

* By default an embedded H2 Database is used (not network accessible). If another database should be used

(18)

Talend Runtime

the port is related to the type and configuration of this database.

1.12.3. Talend Runtime

Port Direction Usage Config (./etc/ ) Remark

8000 Active: Y IN Talend JobServer -Command Port org.talend.remote.jobserver.server. cfg 8001 Active: Y

IN Talend JobServer - File Transfer Port org.talend.remote.jobserver.server. cfg 8888 Active: Y IN Talend JobServer -Monitoring Port org.talend.remote.jobserver.server. cfg

1.12.4. JobServer

8000 Active: Y IN Talend JobServer -Command Port org.talend.remote.jobserver.server. cfg 8001 Active: Y

IN Talend JobServer - File

Transfer Port org.talend.remote.jobserver.server. cfg 8888 Active: Y IN Talend JobServer -Monitoring Port org.talend.remote.jobserver.server. cfg

1.12.5. Talend Artifact Repository

8082

Active: Y

IN Talend Artifact Repository -Jetty HTTP Port

/conf/jetty.xml

8443

Active: Y

IN Talend Artifact Repository -Jetty HTTPS Port

/conf/jetty.xml (not required)

(*)

Active: N*

OUT Talend Artifact Repository -Database

(/conf/jetty.xml)* *By default an embedded Apache Derby Database is used (not network accessible). If another database should be used, the port is related to the type and configuration of this database.

1.12.6. Talend Data Quality Portal

8580 IN Standard port /tdqp/apache-tomcat/conf/

(19)

Talend Data Quality Portal

(20)

(21)

Chapter 2. Introducing Talend Platform for

Data Management

This chapter introduces Talend Platform for Data Management. It provides a detailed description of the architecture and lists all modules that are required for the installation and configuration of Talend Platform for

(22)

Introduction to Talend Platform for Data Management

2.1. Introduction to Talend Platform for Data

Management

The present section lists all the elements required for using Talend Platform for Data Management. To ease their management, we recommend you to centralize all the server modules on one single system.

All Talend applications to be installed must be the same version.

• An application server (Apache Tomcat server + CommandLine) that hosts Talend Administration Center. • A database server storing the administration metadata of Talend Administration Center (by default, an embedded

H2 database is used).

• A database server storing data quality reports and analyses managed by Talend Data Quality Portal.

• A database server storing golden or master data records managed by Talend Data Stewardship Console (if your license includes the Data Stewardship option).

• A SVN server for Project metadata. • A Web browser to access Web application:

• Talend Administration Center where projects, users and processes can be managed and administrated. For more information, see the Talend Administration Center User Guide.

• Data Quality Portal that extends the dynamic reporting and monitoring capabilities of Talend Studio. For more information, see the Talend Data Quality Portal User and Administrator Guides.

• An Artifact Repository (based on Apache Archiva) for software updates.

• Execution servers (JobServers) or Talend Runtime execution containers (based on Apache Karaf) to deploy and execute processes.

• A Studio API to carry out technical processes. For more information, see the Talend Studio User Guide. • A Talend Activity Monitoring Console, providing detailed monitoring capabilities that can be used to consolidate

the collected log information. For more information, see the Talend Activity Monitoring Console User Guide. Each of these elements is detailed in the following sub-sections.

2.1.1. Apache Tomcat Server

The Apache Tomcat server is an application server that hosts Talend Administration Center. This Web application gives access to all management and administration functionalities for an integration project, allowing users to (depending on their role):

• Create and manage projects.

• Create and manage user accounts and roles/rights.

• Access the Job Conductor to schedule, deploy and execute Jobs. • Access the Dashboard to monitor the execution of Jobs.

Note that the best recommended versions for Tomcat are 6.0 and 7.0, however the 5.5 version is also supported.

Talend Administration Center can also be hosted by JBoss application server. For more information on how to deploy Talend Administration Center on Apache Tomcat or JBoss, see the section Deploying Talend Administration Center on an

(23)

Database

2.1.2. Database

The administration database server is used to store administration information and manage the persistence in Talend Administration Center. By default an embedded H2 database is used, but you can also use MySQL, MS SQL Server, or Oracle to store all cross-project data (users, projects, authorization, license, tasks, triggers, dashboard).

The administration database will be named <talend_administrator> in the rest of this document.

The <talend_administrator> administration database will contain all the data related to project information and administration including: administration data, project declaration, user declaration and authorization, task list, etc. The tables in this database are automatically created when connecting for the first time to Talend Administration

Center. The created tables include (among others):

• a Users table, • a Projects table, • a Rights table.

These tables are created, populated and managed automatically by Talend Platform for Data Management, users do not need to take any action.

Additionally, to perform Data Quality operations, a HSQL, MySQL or Oracle database is used to store report and analysis results and a stewardship console database is used to store golden or master data records. For more information about those databases, see the relevant sections in the present guide.

2.1.3. SVN

The SVN server is used to manage the persistence of all the data relating to the objects (Jobs, Business Models, Metadata, Routines, Documentation, etc.) stored in “svn” mode in the Repository of Talend Studio.

We recommend you to store several projects per repository, simply in order not to have too many repositories to deal with. However you can choose to store only one project per SVN repository, if you prefer so.

You can also have several SVN repositories each containing several projects. For more information on how to create projects and store them in an SVN repository, see the Talend Administration Center User Guide.

2.1.4. Talend Artifact Repository

Talend Artifact Repository, based on Apache Archiva, is a preconfigured application centralizing the

management and usage of the Software Update.

Software Update is used to manage application updates (versions and patches) distributed by Talend. By default

two repositories are embedded within Software Update: one to publish the updates and one to retrieve them. These repositories allow the user to visualize the updates available.

For detailed information, see the section Software update repositories.

2.1.4.1. Software update repositories

The following image shows the architecture of Software Update linked to Talend Administration Center and to the Studio.

(24)

Talend Runtime

To download and install some software updates, you need to use Software Update (integrated within Talend

Artifact Repository) and its two embedded repositories:

• an admin repository named talend-update-admin used to publish the updates; • a user repository named talend-update-user used to retrieve the updates.

To do so, you must install Talend Artifact Repository on your machine and log in once to its Web interface following the procedure described in the section Installing and configuring the artifact repository.

The Talend Artifact Repository archive file is embedded in the Talend Administration Center archive file that is provided by Talend.

In Talend Administration Center, the new versions and patches are detected and the administrator can accept them. Talend Studio is connected to Talend Administration Center to retrieve the repository connection information and the version updates are detected and installed automatically.

For more information on how to install and configure Talend Artifact Repository and Software Update, see the section Installing and configuring the artifact repository.

For more information on how to check updates via these repositories, see the Talend Administration Center and

Talend Studio User Guides.

2.1.5. Talend Runtime

Talend Runtime (based on Apache Karaf) is an execution container in which you can deploy and execute all your

(25)

JobServer

Talend Platform for Data Management Installation and Upgrade Guide 19 For more information on the installation of Talend Runtime, see the section Installing Talend Runtime.

2.1.6. JobServer

The JobServer is an application that allows a system installed on the same network as Talend Administration

Center to declare itself as an execution server. These systems must obviously have a working JVM version 1.6. For

more information on the installation of JobServers, see the section Installing the execution servers (JobServers).

2.1.7. Talend Studio

Talend Studio is a rich client that allows the user (such as a project manager, a developer or a DBA) to work on

any Talend project for which he has authorization.

Talend Studio offers a comprehensive set of tools and functions for all its key capabilities including:

• Integration

• Activity monitoring Console • Data profiling

• Data quality

These tools are ALL accessible in different perspectives from one Studio.

The availability of perspectives in your Studio depends either on the license you have when you are working in a local project, or on the type of the remote project itself when you are working in remote projects.

For further information on user authorization on remote project, see the Talend Administration Center User Guide. For further information about the different perspectives available in the studio, see the user guide corresponding to each perspective.

2.1.8. Talend Activity Monitoring Console log database

If you want to use the Talend Activity Monitoring Console, an <AMC> log database must be created, which can be installed on any server. This <AMC> database will initially be empty. Its name may be modified, but you must take into account this modification in the rest of this document.

The <AMC> database will contain three tables that collect data allowing users to monitor Jobs. The three tables will collect data from the following components:

• tFlowMeterCatcher, • tLogCatcher, • tStatCatcher.

Instructions on how to create these tables and their structure is provided in the Talend Activity Monitoring Console

(26)

Operating principles of Talend Platform for Data Management

A corresponding SQL user must be created and thus mapped to have access to this database. This user should be granted the “create” and “update” rights.

2.2. Operating principles of Talend Platform

for Data Management

The operating principles of Talend Platform for Data Management could be summarized as briefly as the following topics:

• building technical or business-related processes and data profiling analyses (graphically represented in light blue and yellow),

• administrating users, projects, access rights and processes and their dependencies (graphically represented in violet),

• deploying and executing technical processes (graphically represented in red and dark blue), • monitoring the execution of technical processes (graphically represented in orange).

Depending on your license, some of the functional blocks may not be available to you.

So, each of the above topics can be isolated in different functional blocks and the different types of blocks and their interoperability can be described as below:

2.2.1. Building and administrating

• The light blue block includes one or more Talend Studio APIs and Web browsers that could be on the same or on different machines.

(27)

Deploying and executing

Talend Platform for Data Management Installation and Upgrade Guide 21 From the Studio API, end-users can carry out technical processes and data profiling analyses and reports regardless of data volume and process complexity. The Talend Studio allows the user to work on any project for which he has authorization. For more information, see the Talend Studio User Guide.

From a Web browser, end-users connect to the remotely based Administration Center through a secured HTTP protocol. The end-user category in this description may include developers, project managers, administrators and any other person involved in building data flows, Each of these end-users will use either the Studio or the Administration Center or both of them depending on the company policy.

Additionally, end-users also use a Web browser to connect to Talend Data Quality Portal that plugs to the data quality data mart (yellow block) and extends the dynamic reporting and monitoring capabilities of the Talend

Studio, and to Talend Data Stewardship Console to resolve the data issues included in the listed tasks to reach

the golden data record (if they subscribed to this option).

• The violet block includes a web-based Administration Center (application server) connected to two shared repositories: one based on an SVN server and one based on a database server.

The Administration Center enables the management and administration of all projects. Administration metadata (user accounts, access rights and project authorization for example) is stored in the database server and project metadata (Jobs, Business Models, Routines for example) is stored in the SVN server (to easily share them between the different end-users).

The Administration Center also enables to configure the tasks that handle job executions and triggers. It also looks after the job generation and deployment to the execution servers. For more information, see the Talend

Administration Center User Guide.

2.2.2. Deploying and executing

• The dark blue block represents the Talend Artifact Repository that stores all the: • Software Updates available for download.

• The red block represents the execution servers that run technical processes according to the execution scheduling set up in the Talend Administration Center Web application. Those execution servers can be of: • One or more Talend Runtimes (execution container) deployed inside your information system. The Talend

Runtime deploys and executes the technical processes according to the set up defined in the Talend

Administration Center Web application. Those processes are Jobs built from the Studio and centralized on

the SVN server. .

• One or more JobServers deployed inside your information system that run technical processes (Jobs) according to scheduled time, date or event set in the Talend Administration Center Web application. The end-user can transfer technical processes to a remote execution server directly from the Studio (distant run).

You must install the JobServer files (“Agent”), delivered by Talend, on each of the execution servers to become operational.

For more information, see the section Installing the execution servers (JobServers).

2.2.3. Monitoring

(28)

Monitoring

The Activity Monitoring Console allows end-users to monitor the execution of technical processes. It provides detailed monitoring capabilities that can be used to consolidate log information collected, understand the interaction between underlying data flows, prevent faults that could be unexpectedly generated and support system management decisions. For more information on the Activity Monitoring Console, see the Talend

(29)

Chapter 3. Installing Talend Platform for

Data Management for the first time

We strongly encourage you to read the chapter Prior to installing the Talend products and the chapter Introducing

Talend Platform for Data Management before starting this chapter.

This chapter details the procedures required to install each part of Talend Platform for Data Management. The following parts usually need to be installed in the following order:

1. SVN Server, see the section Setting up your project server on SVN.

2. Application server (Tomcat or JBoss), see the section Deploying the Web application on Tomcat. 3. High Availability, see the section Setting up High Availability.

The implementation of this architecture depends on your license.

4. Administration center Web application, see the section Configuring Talend Administration Center (Web

application).

5. CommandLine, see the section Installing CommandLine.

6. Talend or Nexus Artifact Repository, see the section Installing and configuring the artifact repository. 7. Execution servers (Jobservers or "agents"), see the section Installing the execution servers (JobServers). 8. Talend Runtime, see the section Installing Talend Runtime.

9. Talend Data Quality Portal, see the section Installing Talend Data Quality Portal.

10.Activity Monitoring Console Web application, see the section Installing the Talend Activity Monitoring Console

Web application.

11.JBoss Business Rules Management System, see the section Installing a Business Rules Management System

(30)

12.Data Stewardship Console as a standalone application (optional), see the section Installing the standalone

Talend Data Stewardship Console (optional).

13.Automatic log purge (optional), see the section Setting up the automatic log purge (optional).

In the following documentation, <TomcatPath> designates the Tomcat installation path and <ApplicationPath> designates the directory in <TomcatPath>/Webapps where Talend Administration Center war file has been deployed.

(31)

Setting up your project server on SVN

3.1. Setting up your project server on SVN

This section describes how to install the SVN server. This server will store all your project data (Jobs, Database connections, Routines, Joblets, etc.).

The installation instructions for Subversion will differ depending on: • your development environment,

• the access method, • the authentication mode.

The access methods and authentication modes to the Subversion server include:

• Apache, a Web server and can be used to access a Subversion repository. Only the authenticated access is documented here.

Note that VisualSVN has both Apache and Subversion embedded

• SSH, another method for accessing a Subversion repository. Authentication is mandatory and brings a higher level of security thanks to encrypted communication.

Choose the procedure that is most relevant to your environment among the following instruction sections. These sections are independent from each other.

3.1.1. Windows

1. Download the SVN server installer, for example: http://www.visualsvn.com/server/download/ 2. Launch the wizard.

3. Clear the Use secure connection (https://) check box so that you use unsecured HTTP. 4. Complete the installation process without changing the other parameters.

5. Launch VisualSVN.

In the main window of VisualSVN Server Manager:

Check in the server URL that the port is 80, if not then perform the following operation: 1. Right-click on VisualSVN Server on the left tree view.

2. Select Properties. 3. Click on the Network tab. 4. Change the port selection to 80.

5. Click Apply then OK.

Then create a new repository:

1. Right-click on Repositories in the main window of VisualSVN. 2. Click Create new repository...

(32)

Debian, apache

3. Name the repository newly created, for example: <repotis>.

4. Click OK.

Having done this, create a new user:

1. Click Create new user... in the main window of VisualSVN. 2. Then define a user name and his password.

Then grant read-write authorization to this user: 1. Right-click on <repotis>.

2. Select Properties.

3. Change the default user (everyone) permissions to no access.

4. Click Add.

5. Select your username in the list displaying and click OK. 6. Check that the granted permissions are read/write.

Your SVN server is now installed and the repository is ready to store all Talend Platform for Data Management projects.

One thing left to do is to link Talend Administration Center to this newly created repository. In VisualSVN:

1. Right-click on your new repository: <repotis>.

2. Select Copy URL to Clipboard.

This copied URL is to be used along with the username and password when you configure the Talend

Administration Center application (see section Configuring the Web application access).

The SVN URL should read:

http://host_or_ip_address:port80/svn/repository_name such as: http://talend-esa.talend.com:80/svn/repotis/

3.1.2. Debian, apache

In the following instructions, lines starting with "#" mean that they must be executed as root. A command line starting with "$" means that it must be executed as standard user.

# apt-get install subversion apache2 libapache2-svn # mkdir /svn

# chown www-data:www-data /svn

Switch to webserver user, which is www-data by default on Debian systems. # su - www-data

$ svnadmin create /svn/repotis

(33)

Debian, ssh

Talend Platform for Data Management Installation and Upgrade Guide 27 # vi /etc/apache2/mods-enabled/dav_svn.conf <Location /repotis> DAV svn SVNPath /svn/repotis AuthUserFile /svn/passwd Require valid-user AuthType basic AuthName "Subversion" </Location> # /etc/init.d/apache2 restart

Your SVN server is now installed and the repository is ready to store Talend Platform for Data Management projects.

One thing left to do is to link Talend Administration Center to this newly created repository.

You will need the URL as well as the username and password of your newly created SVN when configuring

Talend Administration Center (see section Configuring the Web application access).

The URL syntax should read: http://host_or_ip_address/repotis/project_name. For example:

• http://10.42.0.10/repotis/first_project • http://tisserver/repotis/another_project

Then give the user “tisadmin”, password “secretpassword” as described in the passwd file.

3.1.3. Debian, ssh

You can start with the following line: # apt-get install ssh subversion

Create a user "tisadmin" with password "secretpassword". # adduser tisadmin

# mkdir /svn

# chown tisadmin:tisadmin /svn Switch to “tisadmin” user.

# su - tisadmin

(34)

Redhat (Fedora), Apache

Your SVN server is now installed and the repository is ready to store Talend Platform for Data Management projects.

One thing left to do is to link Talend Administration Center to this newly created repository.

You will need the URL as well as the username and password of your newly created SVN when configuring

Talend Administration Center (See section Configuring the Web application access).

The URL syntax should read:

svn+ssh://tisadmin@host_or_ip_address/svn/repotis/project_name.

For example:

• svn+ssh://[email protected]/svn/repotis/first_project • svn+ssh://tisadmin@tisserver/svn/repotis/another_project Then give the user "tisadmin", password "secretpassword".

3.1.4. Redhat (Fedora), Apache

As User: apache

-OR-As Root

# yum install httpd mod_dav_svn The above command installs the following: • - httpd-2.2.9-1.fc9.i386.rpm

• - mod_dav_svn.i386 0:1.4.6-7 • - subversion.i386 0:1.4.6-7 # mkdir /var/www/svn

# svnadmin create /var/www/svn/repository

# htpasswd -cmb /var/www/svn/passwd [email protected] admin # chown -R apache.apache /var/www/svn

# vim /etc/httpd/conf.d/subversion.conf Add: <Location /svn> DAV svn SVNPath /var/www/svn/repository Require valid-user AuthType basic AuthName "Subversion"

(35)

Mac OS X

Talend Platform for Data Management Installation and Upgrade Guide 29 AuthUserFile /var/www/svn/passwd

</Location>

# /etc/init.d/httpd start

3.1.5. Mac OS X

1. Download the SVN server installer, for example Bitnami Subversion Stack at http://bitnami.org/stack/ subversion.

2. Install it in the Apllications/Utilities directory.

Talend Administration CenterChoose a port according to your environment. The default port 8080 clashes with .

3. Launch Bitnami in a terminal. It will provide a step-by-step wizard.

In Bitnami, during the setup:

1. Create a new repository and name it <repotis>.

2. Create a new user and define his user name and password. 3. Grant read-write authorization to this user.

Your SVN server is now installed and the repository is ready to store all Talend Platform for Data

Management projects.

One thing left to do is to link Talend Administration Center to this newly created repository. 4. Copy the URL of your new <repotis> repository.

This copied URL is to be used along with the username and password when you configure the Talend

Administration Center application (see section Configuring the Web application access).

For more information, please refer to the Bitnami Subversion Stack documentation. We recommend that you use Bitnami to create repositories and users, as it is a preconfigured stack, but you also have the possibility to install Subversion server manually on your Mac OS X.

(36)

Deploying Talend Administration Center on an application server

3.2. Deploying Talend Administration Center

on an application server

This section shows how to deploy the Web application on an application server; it can be an Apache Tomcat or a JBoss server application.

3.2.1. Deploying the Web application on Tomcat

First, install the Apache Tomcat application server and stop the Tomcat service if it is automatically started. In the rest of this documentation, <TomcatPath> points out the path where Tomcat has been installed.

Indeed we recommend you to stop this service prior to deploying the Web application (org.talend.administrator.war).

Using 7zip:

1. Unzip the package delivered by Talend: Talend-AdministrationCenter-rYYYYY-VA.B.C.

This will give you access to the different components needed to benefit from all the Talend Administration

Center functionalities:

• org.talend.administrator.war, the archive containing the actual Talend Administration Center Web application.

• Talend-Artifact-Repository-VA.B.C.zip, the archive containing an artifact repository software, based on Apache Archiva, that will be used to handle software updates. For more information, see chapter

Introducing Talend Platform for Data Management.

• an endorsed folder containing a library necessary to use the ESB Conductor of Talend Administration

Center, only if you subscribed to one of our ESB oriented solutions.

2. Copy the Web application, org.talend.administrator.war, into the following directory of Tomcat:

<TomcatPath>/webapps/.

Once you have copied this war file, you can either unzip it manually under the same directory, or let Tomcat unzip the web application at startup.

3. If you subscribed to one of our ESB oriented solutions, to be able to use the ESB Conductor module of Talend Administration Center, copy the endorsed folder provided into the root directory of Tomcat.

On Linux:

1. Edit the following file:

etc/default/tomcat6

2. and uncomment the tomcat security setting and change the default setting, in order to read: #TOMCAT6_SECURITY=no

The storage of log outputs is managed by Tomcat application server, by default, but you are also able to define your own path for storing the logs. From 4.0, you can configure the path directly from Talend Administration Center. For more information on manual configuration in prior versions, refer to section Defining the log storage mode.

(37)

Deploying Talend Administration Center on JBoss

Linux Windows

Start the Tomcat service sh <TomcatPath>/bin/startup.sh net start “Apache Tomcat” Stop the Tomcat service sh <TomcatPath>/bin/shutdown.sh net stop “Apache Tomcat”

For reasons of right management, make sure you launch Tomcat using the same administrator account as for the CommandLine. For exemple: Create an account TISAdmin for both Tomcat and CommandLine.

If you deploy a large number of applications on Tomcat, you should increase its memory to improve its performance. For more information on this process, see section about Tomcat server configuration in the section Increasing the memory of Tomcat .

3.2.2. Deploying Talend Administration Center on

JBoss

First, install the JBoss application server and stop the JBoss service if it is automatically started. In the rest of this documentation, <JBossPath> points out the path where JBoss has been installed.

Indeed we recommend you to stop this service prior to deploying the Web application (org.talend.administrator.war).

1. Using 7zip, unzip the archive delivered by Talend.

2. Manually change the file extension from org.talend.administrator.war to org.talend.administrator.zip. 3. Unzip the file to an org.talend.administrator folder into the following directory:

<JBossPath>/server/default/deploy/

The storage of log outputs is managed by JBoss application server, by default, but you are also able to define your own path for storing the logs. From 4.0, you can configure the path directly from Talend Administration Center. For more information on manual configuration in prior versions, refer to section Defining the log storage mode.

Then start JBoss using the following commands:

Linux Windows

Start the JBoss service sh <JBossPath>/bin/run.sh net start “JBoss” Stop the JBoss service sh <JBossPath>/bin/shutdown.sh net stop “JBoss”

For reasons of right management, make sure you launch JBoss using the same administrator account as for the Commandline. E.g.: Create an account TISAdmin for both JBoss and CommandLine.

3.2.3. Increasing the memory of Tomcat

To increase the memory heap size of the server in order to make all the applications work properly, proceed as follows:

3.2.3.1. On Windows

1. Go to <TomcatPath>/bin where <TomcatPath> is the Tomcat installation directory, and edit the file

(38)

Synchronizing Web application and server time zones

2. Add the following line:

set JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512m -Xmx1024m -Xms256m

The Tomcat memory size is now increased and the server can hold several web applications.

3.2.3.2. On Linux

1. Change directory to <TomcatPath>/bin where <TomcatPath> is the Tomcat installation directory, and edit the file catalina.sh.

2. Add the following line:

JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1024m -Xms256m"

The Tomcat memory size is now increased and the server can hold several web applications.

3.2.4. Synchronizing Web application and server time

zones

To make sure that the DST change and the time zones are correctly taken into account, check that your OS includes an environment variable set as follows:

On Windows: TZ=Europe/Paris

On Linux: Export TZ="Europe/Paris"

If you wish to implement high availability to the task execution scheduling using Talend clustering feature, then read section

Setting up High Availability.

3.2.5. Defining the log storage mode

The log outputs are stored by default in the server application standard log file (STDOUT) as defined in the

Log4j.xml file. However you can store the log in a different file by setting the path to this file in the Log4j.xml file.

To do so, simply set the path in the Configuration page in Talend Administration Center. For more information, refer to your Talend Administration Center User Guide.

If you leave the Path field blank in the Configuration page, then you can also customize the Log4j.xml to address your custom needs.

3.3. Setting up High Availability

The availability of this feature depends on your license.

Using Talend clustering system, you can add high availability and failover to your task execution scheduling in

Talend Administration Center. This is achieved through the ability of deploying multiple Job conductors and Job

Talend Platform for Data Management. Installation and Upgrade Guide 5.2.1

Talend Platform

for Data Management

Installation and Upgrade Guide

Notices

Table of Contents

Preface

1. General information

1.1. Purpose

1.2. Audience

1.3. Typographical conventions

2. Feedback and Support

Chapter 1. Prior to installing the Talend

products

1.1. License key

1.2. Installation requirements

1.2.1. Memory usage

1.2.2. Disk usage

1.2.3. Environment variable configuration

1.2.4. Naming conventions for software packages

1.3. Talend Installer specific prerequisites

1.4. User interface and installation modes of

Talend Installer

1.5. Studio specific prerequisites

1.5.1. Installing database client software (for bulk

mode)

1.6. Compatible Platforms

1.6.1. Talend Studio

1.6.2. Talend server modules: Talend Administration

Center, JobServer, CommandLine,Talend Runtime,

Talend Data Quality Portal

1.7. Compatible Databases

1.7.1. Talend Administration Center

1.7.2. Talend Activity Monitoring Console

1.7.3. Talend Data Quality Portal

1.7.4. Talend Data Stewardship Console

1.8. Compatible Runtime Containers

1.8.1. Talend Administration Center

1.8.2. Talend Data Quality Portal

1.9. Compatible execution servers

1.9.1. Job servers (JobServer and Talend Runtime)

1.9.2. ESB OSGI servers (Talend Runtime)

1.10. Compatible SVN

1.11. Compatible Web browsers

1.12. Port information

1.12.1. Talend Studio and CommandLine

1.12.2. Talend Administration Center

1.12.3. Talend Runtime

1.12.4. JobServer

1.12.5. Talend Artifact Repository

1.12.6. Talend Data Quality Portal

Chapter 2. Introducing Talend Platform for

Data Management

2.1. Introduction to Talend Platform for Data

Management

2.1.1. Apache Tomcat Server

2.1.2. Database

2.1.3. SVN

2.1.4. Talend Artifact Repository

2.1.4.1. Software update repositories

2.1.5. Talend Runtime

2.1.6. JobServer

2.1.7. Talend Studio

2.1.8. Talend Activity Monitoring Console log database

2.2. Operating principles of Talend Platform

for Data Management

2.2.1. Building and administrating

2.2.2. Deploying and executing

2.2.3. Monitoring

Chapter 3. Installing Talend Platform for

Data Management for the first time

3.1. Setting up your project server on SVN

3.1.1. Windows

3.1.2. Debian, apache

3.1.3. Debian, ssh

3.1.4. Redhat (Fedora), Apache

3.1.5. Mac OS X

3.2. Deploying Talend Administration Center

on an application server

3.2.1. Deploying the Web application on Tomcat