Kognitio Technote Kognitio v8.x Hadoop Connector Setup

(1)

Kognitio Technote

Kognitio v8.x

Hadoop Connector Setup

For External Release

Kognitio Document No

Authors

Stuart Watt

Reviewed By

Authorised By

Document Version

v0.1 – Draft

(2)

Document Control ... 3

1. Introduction ... 4

2. Installing Software For Kognitio Hadoop Access... 5

Appendices

A. Testing MapReduce From The Linux Command Line ... 11

(3)

Document Control

Distribution List

Name

Company

Reason

Kognitio

For external release

Revision History

Version

Revision Date

Summary Of Changes

v0.1 – Draft

17

th

June 2013

• _{First version}

•

• Document Location

Office

Bracknell

Machine

Babbage

Filename

G:\Playground\Stuart.Watt\Hadoop\Kognitio Technote - v8.x Hadoop Connector Setup.docx

(4)

1. Introduction

This document describes how to set up a Kognitio v8.x system to interoperate with a Hadoop

cluster using Kognitio’s external table technology. This document specifically covers the

following Hadoop distributions currently:

• _{Hortonworks HDP v1.2.x;}

• _{Cloudera CDH 4;}

• _Apache.

Other Hadoop distributions may work with Kognitio, but as yet have not been verified.

Note: The setup of the Hadoop cluster itself is outside the scope of this document.

(5)

2. Installing Software For Kognitio Hadoop Access

This section outlines the steps necessary on all nodes to allow a Kognitio system, via its external

table technology, to connect to a Hadoop cluster:

1. Install the latest Kognitio v8 Technology Preview Release (at least v7.9.4-130523 which

contains important MapReduce fixes), ensuring that it has a new generation licence that

supports the v8.x features;

2. Install the IBM JVM on all the Kognitio nodes. Note that it needs to be the IBM JVM as the

Oracle JVM won’t work. Accept all the defaults as shown:

% wget https://kognitio-usa.s3.amazonaws.com/software/hadoop/ibm-java-i386-jre-6.0-11.0.bin --2013-06-14 10:53:01-- https://kognitio-usa.s3.amazonaws.com/software/hadoop/ibm-java-i386-jre-6.0-11.0.bin Resolving kognitio-usa.s3.amazonaws.com... 72.21.195.161 Connecting to kognitio-usa.s3.amazonaws.com|72.21.195.161|:443... connected.

HTTP request sent, awaiting response... 200 OK Length: 68596835 (65M) [application/octet-stream] Saving to: `ibm-java-i386-jre-6.0-11.0.bin'

100%[======================================>] 68,596,835 9.33M/s in 7.3s 2013-06-14 10:53:08 (8.95 MB/s) - `ibm-java-i386-jre-6.0-11.0.bin' saved [68596835/68596835] % chmod +x ibm-java-i386-jre-6.0-11.0.bin % ./ibm-java-i386-jre-6.0-11.0.bin Preparing to install...

Extracting the JRE from the installer archive... Unpacking the JRE...

Extracting the installation resources from the installer archive... Configuring the installer for this system's environment...

Launching installer...

Graphical installers are not supported by the VM. The console mode will be used instead... =========================================================================== ==== Choose Locale... --- 1- Català 2- Deutsch ->3- English 4- Español 5- Français 6- Italiano 7- Português (Brasil) CHOOSE LOCALE BY NUMBER: 3

(6)

=========================================================================== ====

IBM 32-bit Linux Runtime for Java v6 (created with InstallAnywhere)

---

Preparing CONSOLE Mode Installation...

=========================================================================== ====

License Agreement ---

Installation and Use of IBM 32-bit Linux Runtime for Java v6 Requires Acceptance of the Following License Agreement:

International License Agreement for Non-Warranted Programs Part 1 - General Terms

BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, OR USING THE PROGRAM YOU AGREE

TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF

ANOTHER PERSON OR A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT AND WARRANT

THAT YOU HAVE FULL AUTHORITY TO BIND THAT PERSON, COMPANY, OR LEGAL ENTITY TO

THESE TERMS. IF YOU DO NOT AGREE TO THESE TERMS,

- DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, OR USE THE PROGRAM; AND

- PROMPTLY RETURN THE PROGRAM AND PROOF OF ENTITLEMENT TO THE PARTY FROM WHOM

YOU ACQUIRED IT TO OBTAIN A REFUND OF THE AMOUNT YOU PAID. IF YOU DOWNLOADED

THE PROGRAM, CONTACT THE PARTY FROM WHOM YOU ACQUIRED IT.

"IBM" is International Business Machines Corporation or one of its subsidiaries.

"License Information" ("LI") is a document that provides information specific

to a Program. The Program's LI is available at http://www.ibm.com/software/sla/

. The LI may also be found in a file in the Program's directory, by the use of

a system command, or as a booklet which accompanies the Program. PRESS <ENTER> TO CONTINUE:

etc.

*** Need to press ENTER about 40 times

(7)

a) You shall not modify, delete, suppress, or obscure any copyright, trademark

or other legal notice (whether from IBM or any third party) which may be displayed by or included within the Program.

b) Java and all Java-based Trademarks are trademarks of Sun Microsystems, Inc.

in the United States, other countries, or both.

c) You recognize IBM's and Sun Microsystems, Inc.'s ownership and title to their respective trademarks and of any goodwill attaching thereto,

including

goodwill resulting from use. You will not use or attempt to register any trademark which is confusingly similar to such IBM or Sun trademarks. 3. PROOF OF ENTITLEMENT

This License Agreement constitutes your Proof of Entitlement.

PRESS <ENTER> TO CONTINUE:

D/N: L-RVEK-75GKYF P/N: L-RVEK-75GKYF

DO YOU ACCEPT THE TERMS OF THIS LICENSE AGREEMENT? (Y/N): y

=========================================================================== ====

Introduction ---

InstallAnywhere will guide you through the installation of IBM 32-bit Linux Runtime for Java v6.

It is strongly recommended that you quit all programs before continuing with

this installation.

Respond to each prompt to proceed to the next step in the installation. If you

want to change something on a previous step, type 'back'. You may cancel this installation at any time by typing 'quit'. PRESS <ENTER> TO CONTINUE:

=========================================================================== ====

Choose Install Folder ---

(8)

Default Install Folder: /opt/ibm/java-i386-60

ENTER AN ABSOLUTE PATH, OR PRESS <ENTER> TO ACCEPT THE DEFAULT :

=========================================================================== ====

Pre-Installation Summary ---

Please Review the Following Before Continuing: Product Name:

IBM 32-bit Linux Runtime for Java v6 Install Folder:

/opt/ibm/java-i386-60

Disk Space Information (for Installation Target): Required: 99,442,965 bytes

Available: 7,987,286,016 bytes PRESS <ENTER> TO CONTINUE:

=========================================================================== ==== Installing... --- [==================|==================|==================|================= =] [---|---|---|---] =========================================================================== ==== Installation Complete ---

Congratulations. IBM 32-bit Linux Runtime for Java v6 has been successfully installed to:

/opt/ibm/java-i386-60

PRESS <ENTER> TO EXIT THE INSTALLER: %

3. Edit

/etc/profile.local

to add the following lines so that the JRE is in the

PATH

for all

users:

export JAVA_HOME=/opt/ibm/java-i386-60/jre export PATH=$PATH:$JAVA_HOME/bin

4. Log out and log back in and then check that Java is operational:

(9)

java version "1.6.0"

Java(TM) SE Runtime Environment (build pxi3260sr11-20120806_01(SR11)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr11-20120801_118201 (JIT enabled, AOT enabled)

J9VM - 20120801_118201

JIT - r9_20120608_24176ifx1 GC - 20120516_AA)

JCL - 20120713_01

5. Install the appropriate Hadoop client from the chosen Hadoop distribution on all the

Kognitio nodes. This step is outside the scope of this document;

6. Edit

/etc/profile.local

so that the Hadoop client home is declared for all users:

Hadoop Distribution

Hadoop Client Home

Hortonworks HDP

/usr/lib/hadoop

Cloudera

/usr/lib/hadoop/client

Apache

/usr/lib/hadoop-x.xx-mapreduce

export HADOOP_HOME=<hadoop_client_home>

The exact directory required varies between Hadoop distributions. In general terms, the

HADOOP_HOME

setting needs to point at the Hadoop directory that contains the files of the

form

‘hadoop-xxxx-1.0.n.jar’

_.

7. Log out and log back in and then check that the Hadoop client on each Kognitio node can

connect to the Hadoop cluster:

% hadoop fs -ls /

Found 3 items

drwx--- - mapred hdfs 0 2013-06-14 11:48 /mapred drwxrwxrwx - hdfs hdfs 0 2013-06-13 09:12 /tmp drwxr-xr-x - hdfs hdfs 0 2013-06-13 13:16 /user

The exact directories shown will depend on the Hadoop distribution, e.g. the example above is

from Hortonworks HDP.

8. Install the Kognitio-compiled version of

libhdfs

on all the Kognitio nodes:

wget https://kognitio-usa.s3.amazonaws.com/software/hadoop/libhdfs.so.0.0.0 mkdir /usr/local/lib/hdfs cd /usr/local/lib/hdfs cp $HOME/libhdfs.so.0.0.0 . ln -s libhdfs.so.0.0.0 libhdfs.so ln -s libhdfs.so.0.0.0 libhdfs.so.0 echo -e '/opt/ibm/java-i386-60/jre/lib/i386\n/opt/ibm/java-i386-60/jre/lib/i386/j9vm\n/usr/local/lib/hdfs' >>/etc/ld.so.conf ldconfig

9. Make sure that the output from

‘cat /etc/ld.so.conf’

looks similar to this:

% cat /etc/ld.so.conf etc. /usr/lib64 /usr/lib include /etc/ld.so.conf.d/*.conf /opt/ibm/java-i386-60/jre/lib/i386 /opt/ibm/java-i386-60/jre/lib/i386/j9vm

(10)

/usr/local/lib/hdfs

10. The Cloudera Hadoop client keeps a key generic

jar

file in a different location to standard

Hadoop, which means that the following commands must be run. This step is not required

for other Hadoop distributions:

# Allow MapReduce to work on Cloudera by allowing access to Hadoop client library

cd /usr/lib/hadoop/client

ln -s /usr/lib/hadoop-0.20-mapreduce/contrib contrib

11. Update the Kognitio database configuration file to ensure that the v8.x features are enabled:

[boot options]

external_tables=yes # Hadoop external tables

external_scripts=yes # External scripting functionality

fixed_pool_size=20 # Only needed if lots of external scripting

12. Create the Kognitio Hadoop module to reference the Hadoop client path set at Step 6

above. This must be done as soon as the Kognitio server is commissioned, i.e. before the

Hadoop connectors are defined. The port numbers of the namenode and jobtracker nodes

vary between Hadoop distributions:

Hadoop Distribution

Namenode Port

Jobtracker Port

Hortonworks HDP

8020

50300

1

Cloudera

8020

8021

Apache

9000

9001

create module hadoop using

'/opt/kognitio/wx2/current/software/Linux/hadoop.wxpi

hadoop_home=<hadoop_client_home_dir> java_home=/opt/ibm/java-i386-60/jre'; alter module hadoop set mode active;

create connector hadoop_hdfs source hdfs target 'namenode

<namenode_internal_ip_address>:<namenode_port>, user <hadoop_user>'; grant connect on hadoop_hdfs to public; -- If public access required create connector hadoop_mr source hadoopmap target 'namenode

<namenode_internal_ip_address>:<namenode_port>, jobtracker

<jobtracker_internal_ip_address>:<jobtrack_port>, subnets 10.0.0.0/8'; grant connect on hadoop_mr to public; -- If public access required

The

‘user’

clause should be specified if the Hadoop cluster is accessed with a user other than

‘root’

_{. The}

‘subnets’

_{clause will definitely be required in Amazon EC2 environments and}

may be required in other environments as well.

13. Create an external table that connects to a test file in HDFS, e.g.:

create external table test(

<column_definitions>)

from hadoop_hdfs -- Connector name defined above, could use hadoop_mr target 'file /user/zzzz/test.txt';

(11)

A. Testing MapReduce From The Linux Command Line

If a SQL statement fails when using the Kognitio

‘hadoopmap’

(MapReduce) connector, it is

sometimes helpful to run a test MapReduce job from the Linux command line to verify the basic

operation of MapReduce independently of Kognitio:

% hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-1.2.0.1.3.0.0-107.jar -fs hdfs://<namenode>:<namenode_port> -jt <jobtracker_node>:<jobtracker_port> -D mapred.reduce.tasks=0 -input hdfs://<namenode>:<namenode_port>/user/hadoop/test.txt -output hdfs://<namenode>:<namenode_port>/tmp/testoutput -mapper cat

This MapReduce job will take the HDFS file

‘/user/hadoop/test.txt’

and simply copy it

(‘cat’

) to the HDFS directory

‘/tmp/testoutput’

as a way of verifying that MapReduce is

functioning correctly. The exact name and location of the

‘hadoop-streaming’jar

file will

depend on the Hadoop distribution being used.

A.1 Common MapReduce Failures

A common source of MapReduce job failures when running with the

‘hadoopmap’

connector is

HDFS file permissions issues. This is because MapReduce jobs always run as

‘root’

, but the

HDFS files being accessed may, by default, be inaccessible to the

‘root’

user. The permissions

on the HDFS files can be checked using the

‘hadoop fs –ls’

command:

% hadoop fs -ls /user/hadoop Found 1 item

-rwx--- 3 hadoop hdfs 18 2013-06-17 15:58 /user/hadoop/test.txt

In this case, the HDFS file

‘/user/hadoop/test.txt’

_{cannot be accessed by the}

‘root’

_user

and therefore the permissions need to be adjusted appropriately:

% hadoop fs -chmod g+rx,o+rx /user/hadoop/test.txt % hadoop fs -ls /user/hadoop

Found 1 item

Kognitio Technote Kognitio v8.x Hadoop Connector Setup