Best Practice Test Management for Business Suite on HANA Migration Projects using SAP Solution Manager 7.1

(1)

November 2013

Marcus Wefers, Chief Support Architect, ALM Solution Management, AGS, SAP AG

Sebastian Geissler, Senior ALM Consultant, SAP Deutschland AG & Co. KG

Best Practice

Test Management for Business Suite on HANA

(2)

Agenda: Test Management for Suite on HANA migration projects

Introduction and Project Phases

Phase 1: Test Preparation



Business Blueprint



Test Scope Identification



Functional Regression Tests



Performance and Load Tests



Service and Interface Tests

Phase 2: Test Execution

Phase 3: Test Activities after Go-Live

Key Take Aways

(3)

(4)

Test Management for Suite on HANA migration projects

HANA migration project phases and involvement of test management

Preparation phase Suite on HANA migration project after go-live

Suite on HANA Migration Project

Test Management

Status assessment Tool & approach

decisons Test management setup Test planning Test execution Issue resolution Retest and sign-off

Regression tests for HANA revision

(5)

(6)

Test Management for Suite on HANA migration projects

Test Management status assessment and setup activities (1)

Workstream Business Blueprint Workstream Test Scope Identification Workstream Test Planning

• Coverage of business processes documented

• Identification of Business Analysts

Assessment & Decision Setup

Interviews with business teams to identify prio 1

business processes

Business processes and involved executables • RBPD service or

• SAP SolMan: Solution Documentation Assistant or • SAP SolMan: Scope and Effort Analyzer (Q1 2014)

Metrics for test scope identification - examples:

• priority or criticality of business process

• # of affected users

• usage frequency of business process

Functional Test Plan 1 Test case selection from business processes with

class A metrics

Functional Test Plan 2 test case selection from business processes including custom code / modifications via

• ABAP Test Cockpit + UPL

• SQL Monitor

• BPCA

• Status test system (PRD copy, test data, …) • Availability of business users for tests

• Reporting requirements

Test system setup

• Definition of Test Packages from Test Plans

• Tester assignment Optional: metric assignment to business processes (ABC classification) Workstream Functional Regression Tests

• Coverage assessment of manual and automated tests

• Target level for test automation, e.g. 30% • Test tool evaluation and tool selection +. ROI

expectations

Optional: Implementation project for selected test

automation tool

Test case definition

• selected functional tests for execution by QA teams • E2E integration tests for execution by Business Analysts

(7)

Test Management for Suite on HANA migration projects

Test Management status assessment and setup activities (2)

Workstream Performance Tests Load Tests Workstream Service & Interface Tests Workstream

Assessment & Decision Setup

Metric selection for load profile definition

• Top (15 ) transactions, batch jobs, reports creating 80% of system load

• performance critical transactions

• Throughput of critical transactions

• peak hours and maximum number of users

• Transactions with high number of affected users

• Processes with custom code or modifications

Definition of Load Profile scenarios – examples:

• high number of online users performing critical transactions in parallel

• throughput of processes (with 3rd party interfaces)

• critical processes including CC / modifications Performance Baseline

define baseline for comparisons • Workload Monitor (ST03N) • SAP Workload (STAD) • Performance Analysis (ST05)

Creation of Performance Test Scripts Definition of scaling factor

for PRD versus TST system using SAPS ratio

Optional: DB Stress Tests: test programs and queries to perform high volume DB access and postings

Identification of non-assigned systems

• Identification of SAP and non-SAP systems required, but not connected to HANA test system

• Tool selection for Service (system) Virtualization Identification of critical Services and APIs

• Identification of (web) services and APIs which require tests without UI (e.g. w/o calling App) and multiple use cases / data variants

• Tool selection for service testing

Service (system) Virtualization

• SAP recommendation: connect required system if possible and avoid system virtualization • Optional: virtualization of not connected, but required systems using system virtualization SW

Service Testing

Creation of service test cases to perform automated tests for (web-) services or APIs without application. Data variation to perform various regression tests of the service / API under test.

• Definition of Reporting Requirements

(8)

Phase 1: Test Preparation

(9)

SAP Solution Manager

Solution Documentation of System Landscape and Business Processes

System landscape documentation

SAP ERP DEV TST PRD TST DEV PRD Single source of truth Infrastructure SAP CRM

Business process documentation

Non-SAP business apps

(10)

Business Blueprint

E2E Business Processes and automated generation of SAP Module oriented Blueprint

Business Blueprint with E2E business processes

Alternatives to create E2E Business Blueprint

1. File upload using existing process documentation

2. Solution Documentation Assistant (tool) and Reverse Business Process Documentation (service)

3. Data transfer from ARIS or HP QC (tool by ALM COE EMEA) 4. Manual documentation supported by BPR (SAP content)

Automated generation of SAP Module oriented Blueprint 1. Application: Scope and Effort Analyzer

 Availability: SAP Solution Manager 7.1 SP11 (Q1 2014)

 Blueprint items generated in background based on usage statistics of executables in PRD systems

 Grouping by SAP Application Component Hierarchy (ACH)

Business Blueprint with process steps by SAP Modules Sales Order to Cash Quotation Sales Order Delivery Goods Issue Billing Procurement Procure to Pay LE Logistic Execution LE-SHP Shipping Delivery (VL01N) Goods Issue (VL02N) SD Sales & Distribution

SD-BIL Billing

Billing (VF01) SD-SLS Sales

Quotation (VA21) Sales Order (VA01)

E2E business process _{Process steps}

grouped by SAP Modules

(11)

Business Blueprint

Creation of Business Blueprint using Solution Document Assistant of SAP Solution Manager 7.1

SAP Solution Manager

Projects

SAP

CRM SolMan SAP Other Other

Solution

Solution Documentation Assistant

SAP Solution Landscape

SAP ERP

Content Interface

1

3

4

1 - Upload business process structure and rules

2

2 - Analyze managed systems

3 - Create and complete project (custom objects) 4 - Hand over to solution

RBPD Content

(12)

Business Blueprint

Automated generation of SAP-Module oriented Blueprint - Result

Result

• 3 level Blueprint hierarchy

• Level 1: SAP Module

• Level 2: ACH L2

• Level 3: Transaction or Report

• Tab „Transaction“ with

assigned executable Custom Code

• Node „Customer“

• Custom Code transactions and reports are grouped by development class

(13)

Business Blueprint

Composition of E2E Business Processes based on SAP-Module oriented Process Step Library

From Process Step Library ….

… to E2E Business Processes

Automatic generation

(14)

Phase 1: Test Preparation

(15)

Test Scope Identification

Functional Test Plan #1: Metrics used for identification of important business transactions

Step 1: Identification of metrics

• Prioritiy / criticality of business process

• Number of affected users of business process • Usage frequency of business process

• …

Step 2: BPx guidance

• ABC classification of business process priorities through interviews with Business process Experts (BPx) of all business areas

Step 3: Usage frequency analysis

• Assess the usage frequency with SAP tools like Workload Monitor (ST03N)

• ABC classification for top n business transactions

Step 4: ABC classification for Bus. Blueprint

• Definition of custom attributes for Business Blueprint in SAP Solution Manager from step 1 • Assign attribute value for business processes with

„A“ classification (from step 2 and 3), e.g.

• high priority / criticality • high number of users • high usage frequency

(16)

Test Scope Identification

Functional Test Plan #1 for important business transactions

Test Option 1:

SAP Solution Manager with integrated partner products

Business Blueprint _{Manual and automated Tests}

Manual Test Automated Test

Test Plan generation - alternatives:

1. filter all available test cases by Blueprint process attribute 2. filter Blueprint by keywords which are assigned to test cases

Functional Test Plan #1

Test Option 2:

SAP Quality Center by HP plus SAP TAO

Business Blueprint

QC Requirements Module

SAP Solman Adapter: transfer processes with

Class A classification

QC Test Plan Module

• Manual tests

• HP QTP and SAP TAO tests

QC Test Set generation

(17)

Optimize custom code and modifications

Motivation

Main drivers for optimizing custom code for the Suite

on HANA migration:

• Discover the potential and take the opportunity to

drastically improve the performance of long running

transactions using HANA functionality

• Restore the performance of transactions due to

performance impacts of HANA migration

• Fix functional degradation due the HANA migration

There is no need to adapt everything!

Identify what needs to be done and where

improvement potential is available

(18)

Test Scope Identification

Functional Test Plan #2 for Custom Code

ABAP Test Cockpit (ATC) Usage and Procedure Logging (UPL)

Approach to identify Custom Code with potential performance improvements

SQL Monitor (SQL-M)

SAP recommendation: define Regression Test Plan #2 for adjusted code objects

• find violating SQL statements with static code analysis

• fix functional & performance degradation by adaption of

ABAP/OpenSQL Statements, like

 SELECT for Pool/Cluster-Tables w/o ORDER BY

 Native SQL code (Critical Statements)  DB hints (Critical Statements)

 ATC results: functional prio 1 issues

• UPL logs all called and executed ABAP units like programs, function modules down to classes, methods and

subroutines

ATC results + UPL:

• exclude unused custom code from analysis

• limit analysis to high usage areas

• Dynamic runtime analysis of SQL statements

• Aggregated logging of all ABAP-based SQL executions

• Ability to identify custom defined tables with high access frequency  Performance issues CC tables  Narrow down ATC results

(19)

Test Scope Identification

Functional Test Plan 2 for custom code: ATC

(20)

Test Scope Identification

Functional Test Plan 2 for custom code: ATC

(21)

Test Scope Identification

Functional Test Plan #2 for custom code: Usage and Procedure Logging (UPL)

Usage and Procedure Logging (UPL)

 UPL is a kernel based logging technology providing runtime usage information of ABAP procedure units like methods, function modules, subroutines and much more…

 UPL complements the standard ST03N workload statistics of ABAP executables

 UPL provides 100 % reliable usage analysis without measurable performance impact

 UPL is available as of SAP Netweaver 7.01 SP10 with Kernel 720 Patch 94

1

2

Evaluate usage Execute business transaction SAP Kernel Load ABAP procedure units

Log ABAP usage

UPL

3

4

SAP Business Suite System like SAP ERP

(system role: PRD)

SAP Solution Manager

(22)

Test Scope Identification

Functional Test Plan #2 for custom code: Usage and Procedure Logging (UPL)

(23)

Test Scope Identification

Functional Test Plan #2 for custom code: customer example (modified)

ATC scan pattern Findings Fixed

SELECT for Pool/Cluster-Tables w/o ORDER BY >10 >10

Native SQL code (Critical Statements) >10 >10

DB hints (Critical Statements) <10 <5

Analysis of WHERE Condition for SELECT 1-10 <5 Analysis of WHERE Condition in UPDATE and DELETE 0 0 Changing Database Accesses in Loops >100 <20 EXIT or no statement in SELECT...ENDSELECT loop 1 0 SELECT Statements with Subsequent CHECK 0 0 Incomplete evaluation. <x>% of fields used <10 0 SELECT Statements That Bypass the Table Buffer >10 <10

Table Attributes Check 0 0

SAP recommendation: define Regression Test Plan #2 for adjusted code objects

1

2

Customer example (modified)

• The company activated UPL well in advance to the Suite on SAP HANA migration project • > 1000 used Custom Code objects detected

by UPL

• ABC classification for CC usage (3 months)

• Class A: > 100 mio executions

• Class B: > 1 mio executions

• ATC code scans results limited to Custom Code of usage frequency class A and B

ABC classification of CC usage frequency via UPL

(24)

Test Scope Identification

Functional Test Plan 2 for custom code: SQL Monitor

SQL Monitor

# executions # records selected

table name

Entry point

High # executions Zero # records

High # records High mean time

(25)

Test Scope Identification

Functional Test Plan #2 for Custom Code

ABAP Test Cockpit (ATC)

Approach to identify Custom Code with potential performance improvements

SQL Monitor (SQL-M)

SQL Performance Tuning Worklist

•

Combines ATC and SQL-M in one application

•

Transaction code: SWLT

(26)

Test Scope Identification

Functional Test Plan 2 for custom code: SQL Performance Tuning Worklist (T-Code: SWLT)

SQL Performance Tuning Worklist

(27)

Test Cope Identification

Functional Test Plan 2 for custom code: SQL Performance Tuning Worklist (T-Code: SWLT)

SQL Performance Tuning Worklist



correlates

• Static Code Checks covering golden rules from

ABAP Test Cockpit

• Runtime information (numbers of execution, execution time, fetch count,…) from SQL

Monitor



Create a weighted work list of code

occurrences that need attention

(28)

Test Cope Identification

Example for prioritization and resolution using SQL Monitor

SQL Performance Tuning

Worklist provides information

about

• Runtime information (from SQL Monitor)

• Static Checks from ABAP Test Cockpit (ATC)

… and includes

• starting point of executable, URL, RFC,… (e.g. transaction VA01)

• detailed information on how to improve the ABAP statement

(29)

Test Scope Identification

Functional Test Plan 2 for custom code: Business Process Change Analyzer (BPCA)

SAP Application

Goal

Activity

Optimal performance after HANA migration

• ABAP Test Cockpit (ATC)

• Usage and Procedure Logging (UPL)

• SQL Monitor

• SQL Performance Tuning Worklist (SWLT)

• Identification of Custom Code and Modifications with problematic ABAP statements

• Adjust code based on priority and usage

No negative impact on critical business

processes in combination with

effort optimized regression test scope

• Business Process Change Analyzer (BPCA) - SAP Solution Manager

• SAP and partner test tools

• Identification of

business processes impacted by CC adjustments

• Test scope optimization

(30)

Test Scope Identification

Functional Test Plan 2 for custom code: Business Process Change Analyzer (BPCA)

1 ₂

3

4

Business Blueprint _{Custom Code and Modification adjustments}

Transport

adjusted Custom Code / Modifications

BPCA

TBOM (*) Tests (**)

(*) Semi-dynamic TBOMs generation fully automated with SAP SolMan 7.1 SP10 (**) Test cases for HP QC integrated with Business Blueprint and BPCA via SAP-HP interfaces

Change impact analysis

• selection of adjusted custom code / modifications from transports

• Selection of business processes, TBOMs, test cases

• BPCA change impact analysis plus test scope optimization

(31)

Phase 1: Test Preparation

(32)

Test Option 3

SAP Connector, Quality Manager, Requirements Composer, Team Concert, ...

BPCA

Test Planning

Test System Setup Test Execution

Change Deployment Change Impact Analysis

SAP Solution Update New SAP

Solution

Test Management

Products and capabilities integrated with SAP Solution Manager 7.1

Test Workbench Test Automation Framework

HP QTP

CBTA 3rd_Party

Quality Center Business Process Testing

HP QTP SAP TAO Test Automation Test Management Test Automation Test Management Test Option 2

(33)

Phase 1: Test Preparation

1c - Functional Regression Tests

(34)

Test Management – Test Option 1 using SAP Solution Manager 7.1

Capabilities of Test Option 1 with SAP Solution Manager 7.1 (SP07 and beyond)

Business Process Change Analyzer

(BPCA)

WoC Test Management

• Manual Tests • Automated Tests

SAP TDMS

Test Planning

eCATT

WoC Test Managment Test Workbench

BW Reporting Partner Logs and Reports

Test Status Reporting

Test script creation Test system assignment Test data assignment

Test Automation Framework

Test Case Assignment

other 3rd party test automation tools

Test Scope Optimization

Test Execution

HP QTP

SAP Solution Manager 7.1

Lean test system setup

CBTA (Component based Test Automation)

Test Planning

Test System Setup Test Execution and Reporting

WS Certify

WoC Test Management

• Test Plan generation • Tester assignment

(35)

Manual Testing

Manual Test Case for business transaction with description of execution steps and access to test data

(36)

Manual Testing

Convinient access for manual Testers

1

2

3

Tester Worklist

1) Easy access for

manual testers from

Work Center Test

Management

2) Test Packages

assigned to Tester

including status

3) Test Cases of selected

Test Package

- Start of execution

- Test status

- Test Notes

- Incidents

1 2 3

(37)

Manual Testing

(38)

Manual Testing

Test Execution: user views test data to be used during test execution

Second window with view of available test data of TDC stays open during test execution

(39)

Manual Testing

Test Execution: guided access to System Under Test (SUT) to start execution of business transaction

Screen 1: test script _{Screen 2: test data}

Test execution

Screen 3: automatic launch of transaction that shall be tested by user in preselected test system

(40)

Test Automation with SAP Solution Manager 7.1

Test Design

 Seamless integration between SAP Solution Manager and test tools from SAP and Partners  Test Data Container

(TDC) provide test data for test script parameters  Central management of

System under Tests

Test Automation Framework

Test Execution

 Start of automated tests from Workcenter Test Management

 Ad-hoc start or scheduler for unattended execution  Integrated test execution

logs from partner tools

Test Result Analysis

 Test Manager can view test results from manual and automated tests in one environment

 Status and progress resports

 Integrated partner reports

Accelerated Repair

 Workflow between Tester and Test Engineer to trigger test case repair  Rich environment for Test

Engineer to analyze problems and perform repair activities

(41)

Test Automation Framework in SAP Solution Manager 7.1

Automated tests assigned to process steps in Business Blueprint

Test Script Business Process

HP QTP

System under Test Test Data

Test Configuration

other ISV test automation tools

Order Type Customer Product

OR C1000 P-100 OR C2000 P-200

Test Data Container

SAP customers have a choice of test automation tools integrated with SAP Solution

Manager 7.1 through the Test Automation Framework

CBTA, eCATT

 Component Based Test Automation (CBTA) is shipped with SAP Solution Manager 7.1 SP07

 Customers with SAP Enterprise Support contract can download 2 HP QTP licenses from SMP (https:/service.sap.com/testing)  Test Automation Framework provides 2 interfaces to integrate ISV test automation tools. SAP ICC offers certification services

(42)

Component-based Test Automation (CBTA)

Test Automation of E2E Business Processes using SAP Solution Manager - CBTA

Component-based Test Automation (CBTA)

Business Analyst executes business

transaction via CBTA

Test Composition Environment (TCE)

Business Process “Order to Cash” Quotation Sales Order Delivery Goods Issue Billing

CBTA creates test script based on test components during execution

Attributes • Business Transaction _{• Systems under Test}

Parameters • Parameters for test data inputs _{• Dynamic handover between scripts} Script

Components

• SAP delivered test components • On the fly generated components

(43)

Automated Tests

Flow to create new automated test using Component-based Test Automation (CBTA)

Business Process Test Composition Environment (TCE) create new Test Configuration

Component Based Test Automation (CBTA)

CBTA – Test Creation Wizard CBTA – process flow analysis

CBTA – Test Components / Test Script

Quotation Sales Order Delivery

Test Composition Environment (TCE) finalize Test Configuration

Test Data Container

Test Data Assignment Wizard assign test data

Store Test Configuration at process step or

(44)

Functional Regression Tests

Test data planning

2 1 Test Data Container (TDC): Data Structure

Test Data Container(s) act as central repository for your test data.

Customer benefit: test data changes can be done in one central location leading to a

significant lower maintenance effort and faster availability of test data

 How: create TDC parameter for all input fields of

your business process used in test script  Who: Test Engineer

TDC: test data planning

test data entry test data upload

 Who: Business Analysts

 Use: Test data stored in a TDC can be consumed

(45)

Test Configuration

Test Script

SAP CBTA, HP QTP, …

System under Test

Test Data Container (TDC)

Logon and automated

business process execuction

Test data selection

TDC

Test execution:

handover of Test Script + Test Data + SUT info User starts

test execution

Functional Regression Tests

Test execution of automated tests

Test Script Test Data System Data Tester Worklist

in Workcenter Test Management

Test results and logs

1

2

3

5

(46)

Functional Regression Tests

Test Reporting with SAP Solution Manager: Gap, Status and Progress Analysis

Test Status Reports _{Trend and Progress Reports} Test Tool Logs

Gap Analysis _{Quick Status Overview}

Test Result Report

Key benefits: Leveraging SAP Solution Manager for all status and results reporting, limiting investment in 3rd party tools

(47)

Phase 1: Test Preparation

1c - Functional Regression Tests

Test Ooption2 using SAP Solution Manager 7.1,

HP Quality Center and SAP TAO

(48)

Functional Regression Tests

Products and capabilities of Test Option 2 with SAP Solution Manager 7.1

Quality Center

• Requirement

• Test Plan

• Business Process Tests

SAP TDMS

Test Option 2

Test Planning Test System Setup

Test Execution Reporting

Solution Manager Business Blueprint • Process documentation • Business requirements Solution Manager BPCA

• Change Impact Analysis • Test Scope Optimization

Adapter Quality Center • Test Lab • Sprinter • QTP • Defect •Blueprint

•Defects / test results

SAP TAO

• Optimized SAP test scripts

• Accelerated maintenance

• Load and

performance testing LoadRunner

(49)

Functional Regression Tests

Manual Testing: HP Sprinter



Floating windows with test step

guidance for Tester



Smart Defects



Recording of User Actions (list) and

Story Board (screenshots)



Full integration with Quality Center for

Test Script selection and results

(50)

Functional Regression Tests

Test Automation of Business Processes using SAP TAO

SAP TAO Process Flow Analyzer

Business Analyst executes

business transaction from SAP TAO

Test Case* composed out of Test Components with

parameters for all input fields Test Components for each SAPGUI subscreen with parameters for all fields

File with Test Data for each input parameter

SAP Quality Center by HP

upload

Business Scenario “Order to Cash” Quotation Sales Order Delivery Goods Issue Billing

(*) Note: Further test case refinement might be needed e.g. Test engineer may add more validation steps and consolidate the test case for performance improvement.

(51)

SAP TAO

Test Execution with SAP TAO built test scripts

SAP Quality Center by HP

SAP Test System

During test execution test data is entered into input fields of

the SAP business process Test Script execution

Test data file

Iteration Result

# 1 Passed # 2 Passed

(52)

SAP TAO

Test Automation of E2E Business Processes

Variant 1

Quotation _{Sales Order V1}

C1 C2 C3

SAP TAO Test Components

Test data

Delivery Goods Issue Billing

Variant 2

C1 C2 C3

Delivery Goods Issue Billing C4

Test data

SAP TAO test scripts

 Test script variants can be built easily via additional Test Components, Flows and Test Data

(53)

SAP TAO Test Automation of E2E Business Processes

Variant 1

C1 C2 C3

SAP TAO Test Components

Test data

Delivery Goods Issue Billing

Variant 2

C1 C2 C3

Delivery Goods Issue Billing C4

Test data

Typical

software changes

affecting automated tests

1. Customizing changes for process step (optional field mandatory field, ...)

2. Custom Code changes for process step (additional mandatory fields, additional screens, ...)

Customer benefit: SAP TAO provides accelerated repair functionality through rapid and semi-automated exchange of central test components

Test execution fails

X

(54)

Goal

Test Execution of manual and automated tests

* additional license required

Test Lab – execution and documentation of test results

Defects – managing incidents

Functional Regression Tests

(55)

Functional Regression Tests

SAP Quality Center by HP – Dashboards and Reporting

Goal

Reporting of test status and progress suitable for different types of users

Scope

 Custom and role-based reports  Coverage graphs with drilldown

Benefits

 Comprehensive and holistic overview about test progress

 Out of the box reports

 Various reporting types usable for a wide range of users from manual testers to senior management

(56)

Phase 1: Test Preparation

(57)



Level 1 – Performance & Load Test of key business processes

 End-to-end execution time including specific known queries

 Resource consumption of end-to-end scenarios including long running queries

 CPU consumption

 Memory consumption

 Degree of parallelization

 Throughput

 Volume restrictions



Level 2 - Runtime behavior and resource consumption of HANA DB Statements

 Execution Time of specific known queries (SQLs / HTTPs)

 Resource Consumption of long running queries

 Measure CPU and memory consumption, and degree of parallelization

Performance & Load Tests

Definition of KPIs

(58)

Goal: Identification of Top 15 Processes causing ca. 80% of system load

Approach 1



ST03N: Workload Monitor – what’s used in the system, elapsed time



Download ST03N table to MS Excel

 Selection of Top Transactions causing ca. 80% of system load

 Double-Click: How many users



STAD: Business Transaction Analysis – what‘s happening in the system right now



ST05: Performance Analysis



SAT: Runtime Analysis

Performance and Load Tests

Definition of load profiles (1)

Definition of Load Profiles

(59)

Goal: Identification of Top 15 Processes causing ca. 80% of system load

Approach 2

 Interviews



Frequently used business processes



Business-critical business processes



Custom developed business transactions



„Long Runners“, but rarely executed (MRP Run, Fixed Asset Depreciation)



How may users execute those processes in parallel (Average/Peak)?



How often are they executed (Throughput: 100.000 Sales Order / year, 10.000 / month)?

Performance and Load Tests

Definition of load profiles (2)

Definition of Load Profiles

(60)

Load & Performance Tests

E2E Test script example for CRM on HANA using SAP Loadrunner by HP

E2E Script – CRM on HANA Single Actions – CRM on HANA Runtime Settings – CRM on HANA Paramters– CRM on HANA

(61)

Load & Performance Tests

Example results for CRM on HANA using SAP Loadrunner by HP

Execution of the SAP LoadRunner by HP scripts in accordance to the load profiles defined:

• here: 1000 Vusers (= processes) • here: step wise ramp-up of Vusers • Think Times: 5 s

Online resource consumption monitor during execution for defined monitoring point in SAP LoadRunner by HP

• here: Transaction Reponse Times

Test Results:

• here: SQLs / hour

• here: HTTPs steps / hour

(62)

Load & Performance Tests

(63)

Load & Performance Tests

Runtime behavior & resource consumption of HANA DB

 Measuring the HANA Engines portion of the overall query response time

 Option 1: Expensive SQL statements  Option 2: SQL trace

 Analyzing SQL execution of pure SQL statement in detail

 Analyzing SQL execution with the profile trace

 Capturing performance traces for Attribute, Analytic & Calculation Views

 Capturing a performance trace from the HANA Studio

 X-Server connection to the HANA database server and starting the HDB admin tool  Capturing a performance trace with the HDB admin tool

 Finding and displaying a specific SQL query in the HDB admin tool  Understand which engine executes which plan operations

 Understand the degree of parallel execution  Drill down into single plan operations

 Analyzing the performance trace of an Analytic View

 Analyzing the performance trace of a Calculation View

SAP Recomendation: Drill Down Analysis on SAP HANA DB based on results of E2E Performance & LoadTests executed with SAP LoadRunner by HP

(64)

Phase 1: Test Preparation

(65)

Service and Interface Tests

Approach

UI Layer / Application Layer Custom Application A DB Layer Custom Application B Custom API Custom Table 1 Custom Table 2 SAP Table … HANA

Headless testing

API test without application or UI

Test Tool:

SAP Quality Center by HP – Service

Test Module

Relevance for SAP HANA:

Ability to test multiple use cases of custom-built interfaces with read / write access to custom and

SAP defined tables residing in the SAP HANA database layer after technical migration

multiple use cases multiple use cases

(66)

Service and Interface Tests

Tool: SAP Quality Center by HP – Service Test Module

SAP Quality Center by HP

- Service Test Module

(67)

(68)

Test Management for Suite on HANA migration projects

Test execution during migration project

Q Gate Functional

Tests Functional Tests

by QM teams

Reporting (test progress reports, status reports: open problems / resolution status)

Q Gate Performance Load Tests Q Gate Interface Tests Sign-Off • incidents • corrections • retest Q Gate IntegrationTests by Business Analysts Functional Tests

Automated Regression Tests for E2E Business Processes

Performance Tests

for mission critical processes

Optional: Load Tests and DB Stress Tests

for large systems with very high throughput requirements

Performance & Load

Tests

Service and Interface Tests

Functional tests of APIs / Services / Interfaces without UI

Service & Interface

Tests

Comparison actual vs. baseline performance

(69)

(70)

Test Management for Suite on HANA migration projects

Part 3: Regression Tests for HANA revision deployments

 Error corrections for SAP HANA via SAP HANA Revisons

Situation

 Regression tests required at technical and / or process level

 Complete regression test not feasable due to time and effort constrains

Regression Tests

 Business Processes: Identify subset of functional tests for mission critical processes

 Technical level: Identify important business operations implemented via DB Views or Stored Procedures

 Define automated tests via functional test automation and service tests

 Execute automated tests as minimum regression tests to verify HANA revision

(71)

(72)

Key Take Away

 Use SAP Solution Manager capabilities, in case your blueprint is incomplete

 Functional test cases shall be assigned to business processes or process steps

Test Scope 1 – critical business processes

 Identify Business Process Experts

 Identify priority 1 business processes and assign custom attribute in your blueprint

 ABC classification of frequently used business transactions

System preparation

 Activate UPL as early as possible in your PRD systems

Test Scope 2 - Custom Code

 ABAP Test Cockpit (ATC) plus UPL for scope limitation

 SQL Monitor to identify important custom tables

Functional Regression Tests

 Consider test automation

 Setup automated tests for all prio 1 business processes

 Prepare tests for CC adjustments

Test Execution

(73)

Test Management

Overview and Detail Presentations and Offline Demos

SAP Service Marketplace:

http://service.sap.com/testing

Information

 Presentations

 Offline Demos

(74)

Best Practices / Next Practice:

Regression Testing of SAP-centric Business Processes

Leverage the experience of other SAP customers and find

out what works best for them.

Content

 Regresstion tests as integral part of the software change process  9 Recommendations for regression testing

 Customer use cases from Colgate-Palmolive, Ferrero, Sara Lee, ConAgra Foods, Soka-Bau

Best Practice available at

SDN: http://www.sdn.sap.com/irj/sdn/alm-test-management

(75)

Best Practices: Regression testing of SAP-centric Business Processes

9 Recommendations

Test System Business Blueprint Creation of Regression Tests Governance

1 _{Lean documentation of important business processes}

2 Setup of lean test systems

3 _{Test automation for critical business processes}

4 Comprehensive test data for automated regression tests 5 Provisioning of easy to use manual tests

7

Manual test execution with guidance for business users 9 Management of the change process

Change Impact

Analysis 6 Change Impact Analysis: Risk-based test scope identification

Test Execution

8

(76)

© 201

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power

Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries. Oracle and Java are registered trademarks of Oracle and its affiliates.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc.

HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®_,

World Wide Web Consortium, Massachusetts Institute of Technology.

Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc. IOS is a registered trademark of Cisco Systems Inc.

RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited.

Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc.

INTERMEC is a registered trademark of Intermec Technologies Corporation. Wi-Fi is a registered trademark of Wi-Fi Alliance.

Bluetooth is a registered trademark of Bluetooth SIG Inc.

Motorola is a registered trademark of Motorola Trademark Holdings LLC. Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.