Implementation Guide X6 1.9.96 13

(1)

Implementation Guide for System x

®

_{X6 Servers}

Lenovo Development for SAP Solutions In cooperation with: SAP AG Created on 3rd July 2015 15:13 – Version 1.9.96-13 © Copyright Lenovo, 2015

(2)

X6 Systems Solution for SAP HANA Platform Edition

Dear Reader,

We wish to explicitly announce that this guide book is for the System X6 based servers for SAP HANA Platform Edition (Type 6241 Model AC3/AC4/Hxx) based on Intel® Xeon® IvyBridge or Haswell EX Family of Processors.

Type 3837 of System X6 based servers and the System eX5 based servers for SAP HANA Platform Edition (models 7147-H** and 7143-H**) are not discussed in this manual.

The Lenovo Systems X6 solution for SAP HANA Platform Edition is based on System X6 Architecture building blocks that provide a highly scalable infrastructure for the SAP HANA Platform Edition ap-pliance software. The Systems x3850 X6, x3950 X6 and software, such as IBM General Parallel File System™ (GPFS) will be used to run SAP HANA Platform Edition appliance software.

Lenovo has created orderable models upon which you may install and run the SAP HANA Platform Edi-tion appliance software according to the sizing charts coordinated with SAP AG. For each workload type, special ordering options for the System x3850 X6 and System x3950 X6 Type 6241 Models AC3/AC4/Hxx have been approved by SAP and Lenovo to accommodate the requirements for the SAP HANA Platform Edition appliance software.

(3)

Copyrights and Trademarks

Lenovo may not offer the products, services, or features discussed in this document in all countries. Consulty our local Lenovo representative for information on the products and services currently available in your area. Any reference to a Lenovo product, program, or service is not intended to state or imply that only that Lenovo product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any Lenovo intellectual property right may be used instead. However, it is the user’s responsibility to evaluate and verify the operation of any other product, program, or service.

Lenovo may have patents or pending patent applications covering subject matter described in this doc-ument. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

Lenovo (United States), Inc. 1009 Think Place - Building One Morrisville, NC 27560

U.S.A.

Attention: Lenovo Director of Licensing

LENOVO PROVIDES THIS PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EI-THER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WAR-RANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Neither this documentation nor any part of it may be copied or reproduced in any form or by any means or translated into another language, without the prior consent of Lenovo.

This document could include technical inaccurancies or errors. The information contained in this doc-ument is subject to change without any notice. Lenovo reserves the right to make any such changes without obligation to notify any person of such revision or changes. Lenovo makes no commitment to keep the information contained herein up to date.

Any references in this publication to non-Lenovo Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of those Web sites is at your own risk.

Information concerning non-Lenovo products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by Lenovo. Sources for non-Lenovo list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide home pages. Lenovo has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-Lenovo products. Questions on the capability of non-Lenovo products should be addressed to the supplier of those products.

Edition Notice: 3rd July 2015

(4)

Lenovo, the Lenovo logo, System x and For Those Who Do are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. Other product and service names might be trademarks of Lenovo or other companies.

A current list of Lenovo trademarks is available on the web at:

http://www.lenovo.com/legal/copytrade.html.

IBM, the IBM logo, and ibm.com are trademarks of International Business Machines Corp., registered in the United States and/or other countries.

Adobe and PostScript are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Fusion-io is a registered trademark of Fusion-io, in the United States.

Intel, Intel Xeon, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

SAP HANA is a trademark of SAP Corporation in the United States, other countries, or both. Other company, product or service names may be trademarks or service marks of others.

(5)

List of Figures

1 Current SAP HANA Appliance Scenarios . . . 8

2 Hardware Overview . . . 9

3 SAP HANA Multiple Single Node Example . . . 10

4 SAP HANA Clustered Example with Backup . . . 11

5 Workload Optimized System x3850 X6 2 Socket Rear View . . . 17

7 Workload Optimized System Storage Book. This contains slots 11, 12 and slots 43, 44 on x3950 X6 in an additional Storage Book . . . 18

9 G8264 RackSwitch front view . . . 25

14 Cluster Node Network Diagram . . . 31

15 Cluster Switch Networking Example . . . 32

16 License Agreement . . . 53

17 Hostname and Domain Name . . . 54

18 Network Configuration . . . 54

19 Cluster Node NIC Configuration dialog bond0 . . . 56

20 Clock and Time Zone . . . 57

21 Advanced NTP Configuration . . . 57

22 Password for the System Administrator . . . 58

23 Installation Mode Selection . . . 63

24 GPFS IP Configuration Dialog . . . 63

25 HANA Password Input Dialog . . . 64

26 DR Architectural Overview . . . 68

27 DR Data Distribution in a Four Node Cluster . . . 69

28 Logical DR Network Setup . . . 70

29 DR Networking View (with no client uplinks shown) . . . 70

30 SAP HANA DR using storage expansion - architectural overview . . . 88

31 Single Node with High Availability . . . 103

32 File System Layout - Single Node HA . . . 104

33 Network Switch Setup for Single Node with HA . . . 107

34 Single Node with stretched HA - Two Site Approach . . . 112

35 Single Node with stretched HA - Three Site Approach . . . 112

36 File System Layout - Single Node stretched HA . . . 113

37 Single Node with Disaster Recovery - Two Site Approach . . . 114

38 Single Node with Disaster Recovery - Three Site Approach . . . 114

39 File System Layout - Single Node with DR with Storage Expansion . . . 115

40 Single Node with HADR using IBM GPFS Storage Replication . . . 116

41 File System Layout - Single Node HADR . . . 117

42 File System Layout - Single Node HADR with Storage Expansion . . . 119

43 Single Node DR with SAP System Replication . . . 120

44 Single Node DR with SAP System Replication . . . 120

45 File System Layout of Single Node DR with SAP System Replication . . . 121

46 File System Layout of Single Node DR with SAP System Replication with Storage Expansion122 47 Single Node with HA using IBM GPFS Storage Replication and DR using System Replication123 48 Single Node with HA using IBM GPFS Storage Replication and DR using System Repli-cation without remote site Switches . . . 124

49 File System of Single Node with HA and DR with System Replication . . . 125 50 File System of Single Node with HA and DR with System Replication and Storage Expansion126

(12)

51 login to ESXi . . . 129

52 configure management network . . . 130

53 display network adapters . . . 130

54 display network adapters 2 . . . 131

55 IP configuration . . . 131

56 Set IP,NETMASK,GW . . . 132

57 Set DNS and Hostname . . . 132

58 Set DNS suffix . . . 133

59 ESXi 5.x filesystems on a System x3850 X6. The VFAT filesystems belong to the USB device . . . 135

60 ESXi5.5 Storage Path . . . 137

61 ESXi 5.1 WEB Welcome . . . 138

62 Create new virtual machine . . . 139

63 Choose custom configuration . . . 139

64 Choose a name . . . 140

65 Choose disk storage for VM files . . . 140

66 Newest virtual machine hardware version . . . 141

67 Configure the use of more than 32 CPUs . . . 141

68 Choose Operating System . . . 142

69 Choose number of CPUs . . . 142

70 Choose Memory . . . 143

71 Choose Network Cards . . . 143

72 Choose SCSI controller . . . 144

73 Create new HANA datastore . . . 144

74 Choose datastore size . . . 145

75 Choose datastore . . . 145

76 Choose SCSI Node . . . 146

77 Add a new CD/DVD device . . . 146

78 Select ISO image . . . 147

79 Select IDE device 0:0 . . . 147

80 Finish creation of SLES ISO mount . . . 148

81 Upgrade virtual hardware . . . 148

82 Confirm upgrade . . . 149

83 Upgrade virtual hardware . . . 149

84 Changing the autoyast parameter for installation . . . 150

85 Adding kickstart parameter for install . . . 151

86 Overview of Backup/Restore Operations . . . 190

(13)

List of Tables

1 Network Switch Options . . . 12

2 System x3850 X6 Single Node Configurations . . . 13

3 IBM System x3950 X6 Single Node Four Socket Configurations . . . 13

4 System x3950 X6 Single Node Eight Socket Configurations . . . 13

5 System x3950 X6 Single Node Four Socket Configurations with Storage Expansion . . . . 14

6 System x3950 X6 SAP ERP on SAP HANA Single Node Configurations . . . 14

7 System x3850 X6 Cluster Node Configurations with Storage Expansion . . . 14

8 System x3950 X6 Cluster Node Configurations . . . 15

9 Slots which may be used for additional NICs . . . 16

10 Card assignments for a two socket x3850 X6 . . . 16

11 Card assignments for a four socket x3850 X6 . . . 17

12 Network interface card assignments for an eight socket x3950 X6 . . . 19

13 Card placement for x3950 X6 four socket and eight socket . . . 19

14 Customer infrastructure addresses . . . 22

15 IP address configuration . . . 23

16 Numbering conventions . . . 24

17 G8264 RackSwitch port assignments . . . 25

22 Installation Process and Phases . . . 41

23 SAP HANA references . . . 42

24 DVD Part Numbers . . . 43

25 Supported Firmware, Software and Driver Levels . . . 44

26 Required Operation Modes UEFI settings . . . 46

27 Required Processors UEFI settings . . . 47

28 Required Power UEFI settings . . . 47

29 Required Memory UEFI settings . . . 47

30 Boot options and boot loaders used . . . 48

31 x3850 X6 RAID Controller Configuration . . . 49

32 x3950 X6 RAID Controller Configuration . . . 50

33 Partition Scheme for Single Nodes and Cluster Installations . . . 51

34 DVD/ISO Media Install Options . . . 52

35 Hostname Settings for DR . . . 72

36 Extra Network Settings for DR . . . 72

37 GPFS Settings for DR Cluster . . . 76

38 eX5 T-Shirt Size to X6 Model Mapping . . . 91

39 Stanza file for X6 servers in eX5 clusters . . . 93

41 eX5 T-Shirt Size to X6 Model Mapping . . . 98

44 IBM System x3550 M4 GPFS quorum node . . . 105

45 Single Node with HA OS Partitioning . . . 106

46 Single Node with HA OS Networking Setup . . . 106

47 Single Node with HA Network Switch Definitions . . . 107

48 SAP HANA Virtual Machine Sizes by Lenovo . . . 128

49 RAID array and RAID controller overview . . . 155

50 x3850 X6 Memory DIMM Placement . . . 160

51 x3950 X6 Memory DIMM Placement . . . 160

(14)

53 Upgrade GPFS Portability Layer Checklist . . . 168

54 GPFS Upgrade Checklist . . . 172

55 Required SAP HANA directories for restore . . . 200

56 Topology Vectors in a 8 node DR-cluster . . . 208

57 Lenovo MTM Mapping & Model Overview . . . 214

58 Lenovo MTM Mapping & Model Overview . . . 215

59 ServeRAID M5120 Firmware Issues . . . 222

List of Listings

1 SSH login screen . . . 183

2 Support script usage . . . 183

3 Support script output . . . 184

4 Example SUSE fstab entries . . . 191

5 Example Red Hat fstab entries . . . 192

6 Creating a copy of the motd file . . . 192

7 Example SLES primary fstab file . . . 192

8 Example SLES backup fstab file . . . 192

9 Example RHEL backup fstab file . . . 193

10 Changing files for backup partition . . . 193

11 Example UEFI Configuration for Primary Partition . . . 193

12 Example GRUB Configuration for Primary Partition . . . 194

13 Example GRUB Configuration for Backup Partition . . . 195

14 Example rsync command . . . 195

15 Example rsync command . . . 196

16 Example SLES primary fstab file . . . 196

(15)

List of Abbreviations

ASU Lenovo Advanced Settings Utility

BIOS Basic Input / Output System

DR Disaster Recovery (previously SAP Disaster Tolerance)

DT SAP Dynamic Tiering (not to be confused with Disaster Recovery (DR), previously Disaster Tolerance (DT))

ELILO EFI Linux Loader

IBM GPFS IBM General Parallel File System

GRUB Grand Unified Bootloader

GSS GPFS Storage Server

IMM Integrated Management Module

LILO Linux Loader

MTM Machine Type Model

NIC Network Interface Controller

OLAP On Line Analytical Processing

OLTP On Line Transaction Processing

OS Operating System

RHEL Red Hat Enterprise Linux

SAP HANA SAP HANA Platform Edition

SLES SUSE Linux Enterprise Server

SLES for SAP SUSE Linux Enterprise Server for SAP Applications

UEFI Unified Extensible Firmware Interface

UUID Universally Unique Identifier

VLAG Virtual Link Aggregation Group

(16)

1 Abstract

This document provides general information specific to the Lenovo Systems Solution for SAP HANA Platform Edition (short: Lenovo Solution). This document assumes that the reader understands the basic structure and components of the SAP HANA Platform Edition (SAP HANA) software, that he has a solid understanding of Linux administration processes, and that he has been instructed how to install the SAP HANA1 _{software on Lenovo Systems hardware.}

Lenovo Solution is built with Lenovo Systems hardware based on Intel Xeon Architecture as building blocks for a scale-up or scale-out SAP HANA system. These provide a highly-scalable infrastructure for SAP HANA. The Lenovo Systems servers with local storage and Lenovo Systems Networking switches will be used to run SAP HANA.

Lenovo has created orderable models upon which you may install and run the SAP HANA according to the sizing charts coordinated with SAP AG. For each workload type, special ordering options for the Lenovo System servers, storage and switches have been approved by SAP and Lenovo to accommodate the requirements for the SAP HANA.

Attention

IMPORTANT: Please do not attempt to install a system without having been instructed

about the content of this document.

Note

It is considered best practice to create backups before and recover the SAP HANA system after a major failure instead of relying on a fresh install with the help of this document. For details on Backup and Recovery please refer to the Lenovo Solution Backup & Restore Guide as well as the Lenovo Solution Hardware, Operating System & GPFS Operations Guide (SAP Note 1650046).

Neither this documentation nor any part of it may be copied or reproduced in any form or by any means or translated into another language, without the prior consent of Lenovo.

Lenovo makes no warranties or representations with respect to the content hereof and specifically disclaims any implied warranties of merchantability or fitness for any particular purpose. Lenovo assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. Lenovo reserves the right to make any such changes without obligation to notify any person of such revision or changes. Lenovo makes no commitment to keep the information contained herein up to date.

Edition Notice: 3rd July 2015

This is the published edition of this document. The online copy is the master.

(17)

1.1 Preface & Scope

The objective of this paper is to document the installation and configuration of the SAP HANA Platform Edition (SAP HANA) on System x hardware using a managed set up rather than manually installing each node from scratch. The major products installed here are SAP HANA, IBM General Parallel File System (IBM GPFS) and the operating systems SUSE Linux Enterprise Server for SAP Applications (SLES for SAP), or Red Hat Enterprise Linux (RHEL).

For instructions how to administrate SAP HANA Platform Edition (SAP HANA) please refer to the

SAP HANA Technical Operations Manual2. Instructions how to administrate and maintain the other components delivered with the System x solution can be found in the SAP Note1650046– Lenovo Systems Solution Hardware, Operating System & GPFS Operations Guide. The Lenovo System x solution for SAP HANA Quick Start Guide provides an overview of the complete solution and instructions how to find service and support for your Lenovo Solution.

1.2 Acknowledgements

The authors of this document are:

• Martin Bachmaier, Lenovo Development for SAP Solutions, Germany • Florian Bausch, Lenovo Development for SAP Solutions, Germany • Hans-Peter Droste, Lenovo Development for SAP Solutions, Germany • Detlev Freund, Lenovo Development for SAP Solutions, Germany • Patrick Hartman, Lenovo Development for SAP Solutions, Germany • Guido Kampe, Lenovo Development for SAP Solutions, Germany • Nils König, Lenovo Development for SAP Solutions, Germany • Christoph Nelles, Lenovo Development for SAP Solutions, Germany • Richard Ott, Lenovo Development for SAP Solutions, Germany • Volker Pense, Lenovo Development for SAP Solutions, Germany • Michael Reumann, Lenovo Development for SAP Solutions, Germany The authors would like to thank the following Lenovo and IBM colleagues:

• Herbert Diether, Lenovo Development for SAp Solutions, Germany • Oliver Rettig, Lenovo Development for SAP Solutions, Germany • Keith Frisby, Lenovo Systems Lab Services, US

• Thorsten Nitsch, IBM GTS, Germany

• Alexander Trefs, Lenovo Technical Sales, Germany

And many people at SAP Development, Walldorf, Germany; specifically: • Abdelkader Sellami, SAP HANA Support, Walldorf, Germany • Adolf Brosig, SAP HANA Development, Walldorf, Germany • Helmut Cossmann, SAP HANA Development, Walldorf, Germany • Henning Sackewitz, SAP Development, Walldorf, Germany

(18)

• Michael Becker, SAP HANA Support Development, Walldorf, Germany • Oliver Rebholz, SAP HANA Development, Walldorf, Germany

1.3 Feedback

We are interested in your comments and feedback. Please send it [email protected]. The full guidebook can be downloaded, depending on its version, from following community (SAP HANA Support Document section) –SAP Solutions at Lenovo Community.

1.4 Disclaimer

This document is subject to change without notification and will not cover the issues encountered in every customer situation. It should be used only in conjunction with the official product literature. The information contained in this document has not been submitted to any formal test and is distributed AS

IS.

All statements regarding Lenovo future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local Lenovo office or Lenovo authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a defini-tive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in Lenovo product announcements. The information is presented here to communicate Lenovo’s current investment and development activities as a good faith effort to help with our customers’ future planning.

This document is for educated service personnel only. If you are not familiar with the described system, we will ask you to restrain from trying to apply what is described herein – you could void the preloaded system installation – and void the SAP certified configuration. This will void the warranty and support of said machine. Please contact [email protected] get enrolled for education prior to installing an Lenovo Solution appliance.

In case of issues with the SAP HANA appliance, the customer is asked to open a SAP Help Desk request (OSS ticket) first and foremost. Only by following this path, can we ensure the proper configuration of the Lenovo Solution. If the customer would open an Lenovo support ticket for the system, he might be requested to perform system upgrades to firmware or software to the latest available levels which might not be supported with the SAP HANA appliance. If identified as a hardware or file system issue, the ticket will be forwarded to the Lenovo support team and handled appropriately. Although this may be contrary to standard Lenovo Support processes, it is the approved and accepted support process for all SAP Appliances including the SAP HANA appliance.

1.5 Support

The System x SAP HANA development team provides new images for the SAP HANA appliance at reg-ular intervals. These images have dependencies regarding the hardware, operating systems, and hardware drivers. The use of the latest image for maintenance and installation of SAP HANA appliance is highly recommended.

Whenever the firmware level recommendations (fixes known firmware issues) for the Lenovo components of the SAP HANA appliance are given by the individual System x support representatives, it is the customers’ responsibility to upgrade (or downgrade) to the recommended levels as instructed by System x support representatives. A list of the minimally required versions can be found in SAP Note1880960

(19)

Whenever the operating systems recommendations (fixes known operating systems issues) for the SUSE Linux components of the SAP HANA appliance are given by the SAP, SUSE, or IBM/Lenovo support representatives, it is the customers’ responsibility to upgrade (or downgrade) to the recommended levels as instructed by SAP through an explicit SAP Note or a Customer OSS Message. SAP describes their operational concept, including updating of the operating system components in SAP Note1599888– SAP HANA: Operational Concept. If the Linux kernel is updated, you have to recompile IBM GPFS software as well.

Whenever other hardware or software recommendations (that fix known issues) for components of the SAP HANA appliance are given by the individual Lenovo support representatives, it is the customers’ responsibility to upgrade (or to downgrade) to the recommended levels as instructed by Lenovo support representatives.

If software and documentation updates are available, you can download them from the respective Lenovo, IBM, SUSE or SAP website. To check for updates, go to the following websites. Follow the procedure in the included documentation to update the software.

• Firmware and drivers for System X6 Servers

– You can obtain updates for System x3850/x3950 X6 servers on the IBM support website (Fix

Central) athttp://www.ibm.com/support/fixcentralusing the the ’Find product’ tab. • IBM General Parallel File System (IBM GPFS3_{) and IBM Spectrum Scale updates}

– You can obtain updates for GPFS on the IBM support website for GPFS 3.5.0,GPFS 4.1.0

andIBM Spectrum Scale/GPFS 4.1.1

• SUSE Linux Enterprise Server for SAP Applications 11 SP3

– You can download the installation package from the SUSE website at http://download.

novell.com/Download?buildid=XL0RqEykZpc~

• SUSE Linux patches and updates

– You can obtain the latest code updates for SUSE from the SUSE website athttp://download.

novell.com/patch/finder/

• Red Hat Enterprise Linux 6.5 and 6.6

– You can download the installation package from the Red Hat website athttp://www.redhat.

com/en/technologies/linux-platforms/enterprise-linux

• VMware ESX Server patches and updates

– You can obtain the latest code updates for vSphere ESX server from the VMware website at

http://www.vmware.com/support/

• SAP HANA appliance updates

– You can obtain the latest code updates from SAP at the SAP Service Marketplace athttp:

//service.sap.com/swdc

Lenovo recommends that customers follow the software upgrade recommendations set out by SAP in the SAP HANA Technical Operations Manual4 (TOM). It is important to understand that the corrections listed in this note are those known to be a solution to a definite problem when running SAP HANA appliance on the System x solutions. This knowledge was derived from internal testing, or customers who ran into a specific problem. In parallel, the organizations owning the individual products provide a lot more fixes that are unknown to the Lenovo-SAP team, yet are recommend to be applied, nevertheless. In particular, there are fixes that IBM/Lenovo recommends to install that are not listed here. It is expected

3_{IBM General Parallel File System}

(20)

that you contact your IBM/Lenovo service contact to get a list of those fixes as well as a reasonably current service level in general.

(21)

2 Introduction

2.1 Purpose

This document is intended to provide a single point of reference for techniques and product behaviors when dealing with SAP HANA.

2.2 Applicability

The techniques and product behaviours outlined in this document apply to: • SAP HANA appliance Platform Edition v1.0

• SLES for SAP5 _{11 SP3}

• RHEL6 _{6.5 and 6.6}

• IBM GPFS 3.5 and 4.1

• Lenovo Systems solution for SAP HANA appliance based on the:

– System x3850/x3950 X6 Workload Optimized Server 2.2.1 SAP HANA Platform Edition Versions

In this document, we reference to several different versions of the Lenovo Solution guided installation software. The following numbering refers to the corresponding SAP HANA Platform Edition version. 1.7.x SAP HANA Platform Edition v 1.0 SPS07 - First release on IBM/Lenovo Systems X6 hardware 1.8.x SAP HANA Platform Edition v 1.0 SPS08

1.9.x SAP HANA Platform Edition v 1.0 SPS09

2.3 Exclusions and Exceptions

The techniques and product behaviours outlined in this document may not be applicable to future releases.

2.4 Conventions

This guide uses several conventions to improve the reader’s experience and the ease of understanding.

2.4.1 Icons Used

The following information boxes indicate important information you should follow according to the level of importance.

Attention

ATTENTION – pay close attention to the instructions given

5_{SUSE Linux Enterprise Server for SAP Applications} 6_{Red Hat Enterprise Linux}

(22)

Warning

WARNING – this is something to take into consideration

Note

INFORMATION – extra information describing in detail

2.4.2 Code Snippets

When reading code snippets you have to note the following: Lines of code that are too long to be shown in one line will be automatically broken. This line break is indicated by an arrow at the end of the first and an arrow at the start of the second line:

1 This is a code snippet that is too long to be printed in one single line, therefore

←-,→you will see an automatic line break.

There are also line numbers at the left side of each code snippet to improve the readability.

Code examples that contain commands that have to be executed on a command line follow these rules: • Lines beginning with a# indicate commands to be executed by the root user.

• Lines beginning with a$ indicate commands to be executed by an arbitrary user.

3 Solution Overview

This document provides general information specific to the Lenovo Solution. This document assumes that the reader understands the basic structure and components of the SAP HANA Platform Edition. SAP HANA should be installed on hardware that has been specifically certified for SAP HANA by SAP. This hardware may not be configured from individual parts, rather it is to be ordered and delivered as a single unit using an Lenovo manufacturer type/model number specified later.

3.1 The SAP HANA Appliance Software

The Lenovo Solution is based on building blocks that provide a highly scalable infrastructure for SAP HANA based on the System x architecture: x3850/x3950 X6 as well as software, such as IBM GPFS, that will be used to run SAP HANA.

Lenovo has created several system models upon which you may install and run SAP HANA according to the sizing charts coordinated with SAP. For each workload type a special System x type/model has been approved by SAP and Lenovo to accommodate the requirements for the SAP HANA Platform Edition.

3.2 Definition of SAP HANA

The following picture defines the current SAP HANA scenarios that can be leveraged through the System x solution for the SAP HANA Platform Edition.

(23)

SAP ERP (CRM SRM,SCM) SAP HANA DB Appliance 1.0 SPS 05 SAP HANA Data Mart SAP HANA 1.0

Corporate Business Intelligence (BI)

SAP Business Warehouse

SAP HANA DB Appliance 1.0 SPS 03

SAP ERP n (CRM, SRM,SCM) SAP HANA DB Appliance 1.0 SPS 05 SAP HANA Data Mart SAP HANA 1.0

…

Customer Application SAP HANA DB Appliance 1.0 SPS 05 SAP HANA Data Mart SAP HANA 1.0 Local BI Local BI

(24)

4 Hardware Configurations

The System X6 Workload Optimized servers for SAP HANA are based upon two building blocks that can be used to fulfill the hardware requirements for SAP HANA. The SAP HANA appliance software must be installed only on a certified and tested hardware configuration based on one of these two models. Lenovo provides a model/type number for four (4) socket and eight (8) socket systems that are to be setup for each certified model by SAP. A customer needs only to choose the model and the extra options to fulfill their requirements. Models created manually will neither be supported by Lenovo nor SAP due to the high-performance criteria set out by SAP during certification.

(a) System x3850 X6 (b) System Storage EXP2524 (c) System x3950 X6 Figure 2: Hardware Overview

System x3850 X6 Workload Optimized Server for SAP HANA (Figure 2a)

• 2×–4×Intel Xeon E7-8880v27,8_{or E7-8880v3}9 _{Family of Processors}

• 128-2048GB DDR3 Memory • Internal Storage:

– 6×1.2TB 2.5" HDD for RAID1 and RAID5 – 2×400GB SSD for LSI CacheCade

• One (1) External Storage (EXP2524) for systems > 512GB (stand-alone configurations) or ≥ 512GB (cluster configurations)

• 2 ×Dual-Port 10GbE NICs • 1 ×Quad-Port 1GigE NICs • IBM General Parallel File System

• Certified for SLES for SAP OS and SAP HANA appliance software

Optional System Storage EXP2524 (Figure 2b)

• Up to 20×1.2TB 2.5" HDD RAID510 • Up to 4×400GB SSD for LSI CacheCade

System x3950 X6 Workload Optimized Server for SAP HANA (Figure 2c)

• 4×–8×Intel Xeon E7-8880v2/v3 and other Family of Processors (refer to System x3850 X6) • 512GB–6TB DDR3 Memory

7_{For improved performance, E7-8890v2 is supported as an optional feature.}

8_{For customers who confirm that an upgrade to an 8 socket system will never be desired, the Intel processors E7-4880v2} or E7-4890v2 will also be supported as optional alternate features.

9_{E7-8890v3 (for improved performance) or E7-8880Lv3 (for improved efficiency) are supported as optional feature} 10_{RAID6 optional}

(25)

• Internal Storage:

– 12×1.2TB 2.5" HDD for RAID1 and RAID5 – 4×400GB SSD for LSI CacheCade

• One (1) External Storage (EXP2524) for systems ≥ 3TB (stand-alone configurations) or > 1024GB (cluster configurations)

• 2 ×Dual-Port 10GbE NICs • 1 ×Quad-Port 1GigE NICs • IBM General Parallel File System

• Certified for SLES for SAP OS and SAP HANA appliance software

4.1 SAP HANA Platform Edition T-Shirt Sizes

Lenovo and SAP have certified a set of configurations to be used with the SAP HANA Platform Edition that are based on the Intel Xeon IvyBridge EX E7-4880v2, E7-4890v2, E7-8880v2, E7-8890v2 or Intel Xeon Haswell EX E7-8880v3, E78880Lv3, E7-8890v3 processor family.

4.2 Single Node versus Clustered Configuration

The Systems X6 Solution servers can be configured in two ways:

1. As a single node configuration with separate, independent HANA installations (example: produc-tion, test, development). These servers all have individual GPFS clusters that are independent from each other. These should be installed as single servers.

Server 1 Server 1 ( (ProductionProduction)) SAP HANA SAP HANA database database GPFS GPFS SAP ERP SAP ERP (Prod)

(Prod) ClientsClients

Internal storageInternalstorage

Server 2 Server 2 (Test) (Test) SAP HANA SAP HANA database database GPFS GPFS Internal storage Server 3 Server 3 (Development) (Development) SAP HANA SAP HANA database database GPFS GPFS Internal storage SAP ERP SAP ERP (Dev)

(Dev) ClientsClients SAP ERP

SAP ERP (Test)

(Test) ClientsClients

(26)

2. As a clustered configuration with a distributed HANA instance across servers. All server (nodes) form one HANA cluster. All servers (nodes) form one GFPS cluster. These should be installed as clustered servers. SAN Storage SAN Storage Server 1 Server 1 SAP HANA SAP HANA Database Database Master node Master node GPFS GPFS Primary Primary node node SAP BW SAP BW Clients Clients Server 2 Server 2 SAP HANA SAP HANA Database Database Worker node Worker node GPFS GPFS Secondary Secondary node node Server 3 Server 3 SAP HANA SAP HANA Database Database Standby node Standby node GPFS GPFS Additional node Additional node G P F S C l u s t e r S A P H A N A C l u s t e r SAN storage SAN storage SAN storage SAN storage Backup/Recovery Internal Storage SAP HANA Data&Log Internal Storage SAP HANA Data&Log Internal Storage SAP HANA Data&Log Internal Storage SAP HANA Data&Log SAP ERP SAP ERP

Figure 4: SAP HANA Clustered Example with Backup

The term scale-out or cluster is used interchangeably in this document. What is meant is the use of multiple single Lenovo workload optimized servers connected via one or more configuration specific network switches in such a way that all servers act as one single high performance SAP HANA instance. These servers will need to be configured different from a single node system and are therefore defined here explicitly. Further documentation will differentiate between non-clustered (single or consolidated) and clustered installations.

4.2.1 Network Switch Options

For clustered configurations, extra hardware such as network switches and adapters need to be pur-chased in addition to the clustered appliances. Currently, the supported network switches for the Lenovo Workload Optimized server in a clustered configuration are:

(27)

Network Description Part Number 10Gb Ethernet RackSwitch G8296 (Rear-to-Front) 7159GR6 RackSwitch G8296 (Front-to-Rear) 7159GF5 RackSwitch G8272 (Rear-to-Front) 7159CRW RackSwitch G8272 (Front-to-Rear) 7159CFV RackSwitch G8264 (Rear-to-Front) 7159G64 RackSwitch G8264 (Front-to-Rear) 715964F RackSwitch G8124E (Rear-to-Front) 7159BR6 RackSwitch G8124E (Front-to-Rear) 7159BF7 1Gb Ethernet RackSwitch G8052 (Rear-to-Front) 7159G52 RackSwitch G8052 (Front-to-Rear) 715952F

Table 1: Network Switch Options

Note

These configurations may change over time, so please contact [email protected] for any update.

(28)

4.3 SAP HANA Optimized Hardware Configurations

SEO models exist for certain configurations, please see the E: Lenovo X6 Server MTM List & Model Overview on page 214 for more details.

4.3.1 System x3850 X6 Single Node Configurations

SAP Models 128 256 384 512 256 512

Product x3850 X6

Type/Model 6241–AC3

CPU 2 ×Intel Xeon® E7-8880v2/v3 4 ×Intel Xeon® E7-8880v2/v3

Memory 128GB 256GB 384GB 512GB 256GB 512GB

Disk 6×1.2TB HDD 2×400GB SSD

Controller 1 ×M5210

Disk Layout 3.6 TB RAID5 for SAP HANA data/log

Network 2 × Dual-Port 10GbE_{1 × Quad-Port 1GigE}

Table 2: System x3850 X6 Single Node Configurations

4.3.2 System x3950 X6 Single Node Configurations

SAP Models 256 512 768 1024 1536 2048

CPU 4 ×Intel Xeon® E7-8880v2/v3

Memory 256GB 512GB 768GB 1024GB 1536GB 2048GB

Disk 6×1.2TB HDD 2×400GB SSD 12×1.2TB HDD 4×400GB SSD

Controller 1 ×M5210 2 ×M5210

Disk Layout 3.6 TB RAID5 for 9.6 TB RAID5 for

SAP HANA data/log SAP HANA data/log

Network 2 × Dual-Port 10GbE

1 × Quad-Port 1GigE

Table 3: IBM System x3950 X6 Single Node Four Socket Configurations

SAP Models 512 1024 1536 2048

Memory 512GB 1TB 1.5TB 2TB

Disk 6×1.2TB HDD 2×400GB SSD 12×1.2TB HDD 4×400GB SSD

Controller 1 ×M5210 2 ×M5210

(29)

4.3.3 System x3850 X6 Single Node Four Socket Configurations with Storage Expansion

SAP Models 768 1024 1536* 2048*

Memory 768GB 1TB 1.5TB 2TB

Disk 15×1.2TB HDD & 4×400GB SSD

Controller 1 ×M5210 & 1 ×M5120/M5225

Table 5: System x3950 X6 Single Node Four Socket Configurations with Storage Expansion

* For Suite on HANA only, not Datamart and BW

4.3.4 System x3950 X6 SAP ERP on SAP HANA Single Node Configurations

SAP Models 3TB 4TB 6TB

Memory 3TB 4TB 6TB

Disk 21×1.2TB HDD & 6×400GB SSD 30×1.2TB HDD & 8×400GB SSD

Controller 2 ×M5210 & 1 ×M5120/M5225

Disk Layout 19.2 TB RAID5 for SAP HANA data/log 28.8 TB RAID5 for SAP HANA data/log

Table 6: System x3950 X6 SAP ERP on SAP HANA Single Node Configurations

4.3.5 System x3850 X6 Cluster Node Configurations with Storage Expansion

SAP Models 256 512 1024

CPU 2 ×Intel Xeon® E7-8880v2 4 ×Intel Xeon® E7-8880v2/v3

Memory 256GB 512GB 1TB

Disk 15×1.2TB HDD & 4×400GB SSD

Controller 1 ×M5210 & 1 ×M5120/M5225

(30)

4.3.6 System x3950 X6 Cluster Node Configurations

SAP Models 512 1024 1024 2048

CPU 4 ×Intel Xeon® E7-8880v2/v3 8 ×Intel Xeon® E7-8880v2/v3

Memory 512GB 1TB 1TB 2TB

Disk 12×1.2TB HDD 21×1.2TB HDD

& 4×400GB SSD & 6×400GB SSD

Controller 2 ×M5210 2 ×M5210 & 1 ×M5120/M5225

Disk Layout 9.6 TB RAID5 19.2 TB RAID5

for SAP HANA data/log for SAP HANA data/log

Table 8: System x3950 X6 Cluster Node Configurations

4.4 Card Placement

Attention

You need to make sure, that the cards are placed in the correct PCI slot. Please refer to the tables below for the assignment regarding in which slot a certain card should be. This step must be done before the installation. Please be aware, that only with the correct card layout your machine is supported by Lenovo.

Depending on having two, four or eight socket machines, there is a different card placement. Please refer to figure 5 and table 10 two socket machines, figure 6 and table 11 on page 17 regarding four socket machines and figure 8 and table 12 on page 19. Concerning the numbering of the slots please note that PCI slots 11 and 12 are located in the Storage Book, see figure 7. A x3950 X6 machine has an additional Storage Book containing PCI slots 43 and 44. The Storage Books are accessible from the front.

4.4.1 Network Interface Cards

The x3850 X6 machine comes with two Mellanox Connect X-3 10GbE adapters that provide two 10GbE ports or two Mellanox ConnectX-3 FDR IB VPI adapters that provide two QSFP ports. With QSA adapters the QSFP ports support SFP+ transceivers for 10GbE connectivity. A quad port Intel I-350 provides four 1GbE ports and is placed in slot 10. In a x3950 X6 an additional I-350 card can be placed in slot 42. Intel I-340 PCI cards is available optionally, if more 1GbE ports are needed.

Please see the tables and figures below regarding the assignment regarding in which slot a certain card should be, depending on your machine type and configuration.

4.4.2 Slots for additional Network Interface Cards

If the customer needs more network ports, the PCI slots shown in table 9: Slots which may be used for additional NICs on page 16 may be used for additional NICs.

(31)

Machine PCI Slots

x3850 X6 two sockets 9, 10

x3850 X6 four sockets 2, 3, 5, 6, 10 x3950 X6 four sockets 9, 10, 41, 42 x3950 X6 eight sockets 5, 6, 10, 37, 38, 42

Table 9: Slots which may be used for additional NICs

4.4.3 RAID Adapter Cards

The internal RAID adapter is a ServeRAID M5210 which resides in slot 12 in the Storage Book. Regarding the x3950 X6, there are two internal RAID adapter used, residing in slot 12 and 44.

The first external RAID adapter (ServeRAID M5120 or M5225) in a x3850 X6 will be placed in slot 8, the second in slot 7 and then slot 9 for the third. Regarding a x3950 X6 machine, placement will start in slot 40, then 39, then 41 and finally 7 and 8, refer to table 13 for details.

Card Port Label Slot Ethernet

Device

ServeRAID M5210 (internal) – 12 –

Intel I-350 1GbE quad port

E 10 eth4 F eth5 G eth6 H eth7

Intel I-340 1GbE quad port *

– 9 eth8 – eth9 – eth10 – eth11

Mellanox ConnectX-3 (10GbE or FDR IB VPI) A 8 eth0

B eth1

Mellanox ConnectX-3 (10GbE or FDR IB VPI) C 7 eth2

D eth3

100MbE internal Ethernet Adapter for System

Management via the IMM I – –

Table 10: Card assignments for a two socket x3850 X6 * This card is optional

(32)

Figure 5: Workload Optimized System x3850 X6 2 Socket Rear View

Card Port Label Slot Ethernet

Device

ServeRAID M5210 (internal) – 12 –

E 10 eth4 F eth5 G eth6 H eth7 ServeRAID M5120/M5225 (external) * – 9 – ServeRAID M5120/M5225 (external) * – 8 – ServeRAID M5120/M5225 (external) * – 7 –

– 5 eth8 – eth9 – eth10 – eth11

D eth3

B eth1

Table 11: Card assignments for a four socket x3850 X6

(33)

Figure 6: Workload Optimized System x3850 X6 4 Socket Rear View

Figure 7: Workload Optimized System Storage Book. This contains slots 11, 12 and slots 43, 44 on x3950 X6 in an additional Storage Book

(34)

Card Port Label Slot Ethernet Device

E 10 eth4 F eth5 G eth6 H eth7

D eth3

B eth1

Management via the IMM J – –

K 42 e.g. eth8 L e.g. eth9 M e.g. eth10 N e.g. eth11

– 5 e.g. eth8 – e.g. eth9 – e.g. eth10 – e.g. eth11

Table 12: Network interface card assignments for an eight socket x3950 X6 * This cards is optional, please refer to table 13 for details

4 processors 8 processors Slot 512GB 4S 1TB 4S 1TB 2TB 4TB 6TB* 12TB* 4 MLNX MLNX MLNX MLNX MLNX 7 MLNX MLNX S/C M5120/ M5225 8 S/C M5120/ M5225

10 I350 I350 I350 I350 I350 I350 I350

12 M5210 M5210 M5210 M5210 M5210 M5210 36 MLNX MLNX MLNX MLNX MLNX 39 MLNX MLNX S/C M5120/ M5225 S/C M5120/ M5225 C M5120/ M5225 40 C M5120/ M5225 C M5120/ M5225 C M5120/ M5225 C M5120/ M5225 C M5120/ M5225 S/C M5120/ M5225 41 C M5120/ M5225 C M5120/ M5225

42 I350 I350 I350 I350 I350 I350 I350

44 M5210 M5210 M5210 M5210 M5210 M5210 M5210

(35)

(36)

5 Networking

5.1 Networking Requirements

The networking for the Lenovo Solution, the Integrated Management Module (IMM) and the correspond-ing switches should be set up and integrated into the customer network environment accordcorrespond-ing to the customer’s requirements and the recommendations from SAP. SAP currently recommends that individual workloads are separated by either physical or virtual LAN addresses or subnets.

The individual workloads described by SAP are:

• SAP HANA internal communication via SAP HANA private networking • Customer access to the SAP HANA appliance via:

– SAP Landscape Transformation Replication (LT) – Sybase Replication (SR)

– SAP Business Objects Data Services (DS) – Business Objects XI, Microsoft Excel, etc. – Server data management tools for:

∗ System/DB backup and restore operations

– Logical server application management (can be partially accomplished via Integrated

Manage-ment Module)

∗ SSH access, VNC access, SAP Support access

We strongly recommend that the following SAP Workloads are dedicated and distinct subnets using separate Ethernet adapters (NICs). If not, the network setup will become more complicated.

• SAP HANA client access • Server data management • Server application management

Additionally to the SAP workloads the Lenovo Solution defines two additional workloads: • IBM clustered files system communications for GPFS

• Physical server management via the Integrated Management Module

– Hardware support, console web access and SSH access

It is necessary to separate the IBM GPFS and SAP HANA internal networks from all other networks as well as from each other. Servers being configured in a clustered scenario require two dedicated high speed NICs (e.g. 10GbE) with separate physical private LANs for the internal communication of GPFS and SAP HANA. In addition external networks, e.g. for SAP Client/BW and SAP management communication should be separated as well. If not, SAP HANA performance may be compromised and the system is not supported by SAP nor Lenovo.

5.2 Jumbo Frames

It is possible and allowed to activate the usage of so-called jumbo frames for the HANA and GPFS networks. Jumbo frames are Ethernet frames with a Maximum Transmission Unit (MTU) up to 9000 bytes. The standard MTU is 1500.

(37)

The advantage of jumbo frames is less overhead for headers and checksum computation. This can lead to a better network performance on the HANA and GPFS networks.

Attention

Jumbo frames can only be used, if all network components (for example networking adapters and switches) that have to process these jumbo frames support the usage.

If erroneously activated, jumbo frames cause the loss of network connectivity.

The switches G8264, G8272, G8296 and G8124E are certified for the usage in the Lenovo Solution appliance with jumbo frames. In a standard cluster setup jumbo frames can be activated. In DR11 _or

High Availability setups the HANA and GPFS networks may communicate via non-Lenovo customer switches that cannot handle jumbo frames, therefore it is recommended to not use jumbo frames in these setups.

To change this behaviour, you have to change the MTU size. This can be done like the following: • SUSE: in the YaST module for networking: General tab in the configuration of the network

de-vice/bond

• Red Hat: changing the MTU size in the file /etc/sysconfig/network-scripts/ifcfg-* of the interface/bond

Warning

Jumbo frames are activated during the installation phase forbond0 and bond1. You may have to deactivate the usage of jumbo frames in certain scenarios.

5.3 Network Configuration

Before you configure the server and install the Lenovo Solution, please gather the following network information from your network administrator where indicated with theb symbol. Please use only IPv4 addresses.

Note

In case the customer plans to install a single node configuration, but would like to scale it out to a cluster by adding more severs: plan the network configuration for the GPFS and HANA networks as if the cluster would be already existing to simplify a later scale out.

IP Address _b

Default Network Prefix _b Default Netmask _b Default Gateway _b Primary DNS IP _b Secondary DNS IP _b Domain Search b NTP Server b

Table 14: Customer infrastructure addresses

(38)

Port Label

Network Single Cluster IP Address Hostname Netmask Gateway

Server Node01(Worker/Stand-By/Single) IBM GPFS Private Net-work (prede-fined) any A / C 127.0.1.1 (default) 192.168.10.101 (ex-ample) b gpfsnode01 (mandatory) b 255.255.255.0 (recommended) b None (recom-mended) SAP HANA Private Net-work (prede-fined) any B / D 127.0.2.1 (default) 192.168.20.101 (ex-ample) b hananode01 (mandatory) b 255.255.255.0 (recommended) b None (recom-mended) Customer Network

Any of the

re-maining NIC

ports

b b b b

IMM I b b b b

Server Node02(Worker/Stand-By) IBM GPFS Private Net-work (prede-fined) any A/C 127.0.1.1 (default) 192.168.10.102 (ex-ample) gpfsnode02 (mandatory) 255.255.255.0 (recommended) None (recom-mended) SAP HANA Private Net-work (prede-fined) any B/D 127.0.2.1 (default) 192.168.20.102 (ex-ample) hananode02 (mandatory) 255.255.255.0 (recommended) None (recom-mended) .. .

for all other nodes ..

.

Table 15: IP address configuration

5.4 Network Switch Configuration For Clustered Installations

In a clustered configuration with high availability, the internal networks of the appliance for GPFS and HANA are set up with redundant links. These connect to redundant G8264, G8272, G8296 or G8124E 10GigE switches. Both switches are connected with a minimum of two ISL ports. It is recommended to use the 40GbE ports for the ISLs. On host side the two corresponding ports of each network are configured as Linux bond devices. The data replication connection to the primary data source can also be set up in a redundant fashion and connects directly to the appliance internal 10GigE HANA network. The details for this setup depend strongly on the customers network infrastructure and need to be planned accordingly. Details to the exact configuration can be found in chapter 5.6.7: Network Configurations in a Clustered Environment on page 30.

Warning

When connecting the data replication network directly to the internal 10GigE network, an ACL needs to be configured on the uplink port to isolate the internal networks (e.g. 127.0.n.24) from the customer network.

If a network adapter or one of the switches fail, the SAP HANA network and the GPFS network are taken over by the remaining switch and network adapter.

It is recommended to establish redundant network connections for the other networks (e.g. client network) as well. This setup is similar to the internal networks and requires two identical 1GigE or 10GigE switches

Implementation Guide X6 1.9.96 13

Implementation Guide for System x

X6 Servers

X6 Systems Solution for SAP HANA Platform Edition

Copyrights and Trademarks

Contents

List of Figures

List of Tables

List of Listings

List of Abbreviations

1 Abstract

1.1 Preface & Scope

1.2 Acknowledgements

1.3 Feedback

1.4 Disclaimer

1.5 Support

2 Introduction

2.1 Purpose

2.2 Applicability

2.3 Exclusions and Exceptions

2.4 Conventions

3 Solution Overview

3.1 The SAP HANA Appliance Software

3.2 Definition of SAP HANA

Corporate Business Intelligence (BI)

…

4 Hardware Configurations

4.1 SAP HANA Platform Edition T-Shirt Sizes

4.2 Single Node versus Clustered Configuration

4.3 SAP HANA Optimized Hardware Configurations

4.4 Card Placement

5 Networking

5.1 Networking Requirements

5.2 Jumbo Frames

5.3 Network Configuration

5.4 Network Switch Configuration For Clustered Installations

_{X6 Servers}