Using SQL Server 2014 In-Memory Optimized Columnstore with SAP BW

(1)

Using SQL Server 2014 In-Memory Optimized

Columnstore with SAP BW

Applies to:

SAP Business Warehouse 7.0 and higher running on Microsoft SQL Server 2014 and higher

Summary

SQL Server 2014 In-Memory Optimized Columnstore dramatically decreases disk space usage and can improve BW query performance by factors. You can use the columnstore index for SAP BW f-fact and e-fact tables, for SAP BW basic cubes, and aggregates. SQL Server 2014 columnstore support has been integrated into SAP BW and was down-ported to all common SAP BW releases:

 SAP BW 7.0 (SAP NetWeaver 7.0)

 SAP BW 7.01 (SAP enhancement package 1 for SAP NetWeaver 7.0)  SAP BW 7.02 (SAP enhancement package 2 for SAP NetWeaver 7.0)  SAP BW 7.11 (SAP enhancement package 1 for SAP NetWeaver 7.1)  SAP BW 7.3 (SAP NetWeaver 7.3)

 SAP BW 7.31 (SAP enhancement package 1 for SAP NetWeaver 7.3)  SAP BW 7.4 (SAP NetWeaver 7.4)

Author:

Martin Merdes

Company:

Microsoft

Created on:

February 2015

Version:

1.01

Author Bio

Martin Merdes is a Senior Software Engineer at Microsoft. Since 1996 he has been working in different technical functions for Microsoft at SAP Germany: Support Engineer, Senior Consultant, Technical Account Manager, Premier Field Engineer and Software Development Engineer. Since 2009 he is responsible for the porting of SAP BW on Microsoft SQL Server.

(2)

Overview

SQL Server 2012 introduced an additional storage format for data. Besides the normal row-oriented store, there is a column-oriented store as well. To make it as transparent as possible for database administrators and applications, the new columnstore was implemented as a new index type: nonclustered columnstore index. Therefore, a single table can use both stores side-by-side. The nonclustered columnstore index has one major drawback: It is a read-only index. Therefore an additional columnstore index type was introduced in SQL Server 2014: the writeable clustered columnstore index. In SQL Server 2014 you can use all index types: rowstore, clustered and nonclustered columnstore indexes.

The columnstore is often called In-Memory Optimized DW in Microsoft Knowledge Base Articles and BLOGs. Here you often see the acronym NCCI for the (read-only) Nonclustered Columnstore Index and

CCI for the (writeable) Clustered Columnstore Index. Benefits of Using the Columnstore

Space Usage

Indexes can be compressed much better when stored by columns rather than by rows. Furthermore, the number of indexes for the fact tables is reduced when using the columnstore. Therefore, we see

consistent space savings when using the columnstore in SAP BW. The space savings are dependent from the number of BW dimensions, number of key figures, and the data distribution.

The following example shows the total disk space needed for a 100,000,000 row cube having 11

dimensions and 8 key figures. The total space contains the disk space for the f-fact, e-fact and dimension tables. Note that the dimension tables still use rowstore indexes in SAP BW on SQL Server 2014.

Query Performance

The column-oriented store is optimized for aggregating mass data, while the row-oriented store can be used optimally when accessing just a single row. Query performance is typically by factors better when using the columnstore compared with the rowstore. However, the performance improvements depend on the individual query. BW aggregates also benefit from using the columnstore.

The columnstore is optimized for mass aggregation. Therefore, BW cubes and aggregates having only a few thousand rows hardly benefit from the columnstore. On the other hand, query performance is typically not an issue anyway for such small cubes.

0 MB 5000 MB 10000 MB 15000 MB 20000 MB 25000 MB 30000 MB 35000 MB 40000 MB SQL 2005 SQL 2008 PAGE compression SQL 2012 NCCI SQL 2014 CCI SQL 2005 SQL 2008 PAGE

compression SQL 2012 NCCI SQL 2014 CCI

(4)

In SAP NetWeaver (and BW) systems you typically configure the intra-query parallelism to 1 by setting the SQL Server configuration option “max degree of parallelism”. In SAP BW you can overwrite this global setting for SAP BW queries. You simply have to set the SAP RSADMIN parameter

MSS_MAXDOP_QUERY in SAP report SAP_RSADMIN_MAINTAIN. This results in adding a SQL

Server optimizer hint MAXDOP for all BW queries. The default value of MSS_MAXDOP_QUERY is 2 for SQL Server 2012 and 4 for SQL Server 2014. We do not recommend changing the default value on a productive server with 16 CPU threads or less.

For powerful DB servers you may increase the value of MSS_MAXDOP_QUERY, but you should never set it to MAXDOP=1. We typically do not see a benefit for the columnstore when increasing

MSS_MAXDOP_QUERY above 8. Increasing the parallelism too much is not a good idea. This might result in a resource bottleneck during high workload. In this case, SQL Server might be forced using actually only MAXDOP =1 for some concurrently running queries. Furthermore, some columnstore optimizations only work with MAXDOP >1 in SQL Server 2012. However, with SQL Server 2014 these optimizations also work with MAXDOP =1.

The overall runtime of an SAP BW query consists of the time consumed on the database server, the SAP BW application server and the client. The columnstore is an optimization on the database server.

Therefore the columnstore has no impact on the runtime consumed on the SAP BW application server, which can be a significant part of the overall BW query runtime

An SAP BW query may run more than one SQL query. This is always the case when using BW

Semantically Partitioned Cubes or BW Multi-Provider. Both are logical cubes, which consist of

several physical cubes. When running a BW query against the logical cube, separate SQL queries run against each physical cube. Afterwards, the result sets of all queries have to be aggregated on the SAP BW application server. However, the SAP BW application server does not use parallelism. Each SAP work process is executed single-threaded. Furthermore, aggregation on the application server requires much more rows, which have to be transferred from the database to the application server.

Faster BW Processes

Many BW processes like DTPs, BW cube compression and DB index creation are much faster when using the writeable columnstore. One reason is the reduced number of indexes you have to maintain during data load. Re-creating a columnstore index (or converting a cube to columnstore) can use SQL Server parallelism much better than rowstore indexes.

SAP BW uses a separate RSADMIN parameter to configure parallelism for index creation:

MSS_MAXDOP_INDEXING. The default value is 8. Index creation scales quite well even with a higher degree of parallelism. Therefore, you can increase MSS_MAXDOP_INDEXING as long as sufficient CPU and memory resources are available.

Restrictions

SAP BW does not allow the creation of a writeable columnstore index on the f-fact table of a Real-Time cube. A Real-Time cube does not close a data load request before running queries against the cube. Therefore loaded data may still be in the SQL Server Delta Store, which then results in even worse query performance (compared with the rowstore).

In SQL Server 2014 you cannot create an additional b-tree index, unique index or primary key constraint when already having a writeable columnstore index. Therefore, the writeable columnstore is restricted to the f-fact and e-fact tables of SAP BW for SQL Server 2014. Additional support for other SAP BW tables, like master data tables, dimension tables and DSO tables is planned for future versions of Microsoft SQL Server.

(5)

Prerequisites

The writeable columnstore index of SQL Server 2014 cannot be used out of the box with SAP BW. It requires additional SAP BW code. SAP Note 2114876 always contains the up-to-date prerequisites.

As at December 2014, you have to install the following software components:

SQL Server

The minimum required SQL Server version is Cumulative Update 5 (CU5) of SQL Server 2014. However, we recommend installing the newest service pack, once available. You can download the latest SQL Server patches from http://technet.microsoft.com/en-us/sqlserver/ff803383.aspx.

SAP BW

To use the columnstore on SQL Server 2014, you have to apply the following SAP Support Package Stacks (SPS). Afterwards you should apply SAP Note 1949486, which will contain necessary bug fixes in the future. SAP BW NW SPS BW SP 7.0 32 *) 34 7.0 EHP1 17 17 7.0 EHP2 17 17 7.1 EHP1 14 14 7.3 13 13 7.3 EHP1 15 15 7.4 10 10

* NW SPS 32 includes SAP Basis SP 32 and SAP BW SP 34

Alternatively, you may apply an older SAP SPS and additional correction instructions. For more information, see SAP Note 2114876.

The required SAP BW code for using SQL Server 2014 writeable columnstore also contains many improvements for SQL Server 2012 read-only columnstore. Therefore, we recommend applying the new code also for SQL Server 2012. When discussing features of SQL Server 2012 below, we assume that the newest SAP BW code is already applied.

(6)

Applying the Columnstore

Applying the columnstore is a two-step process. In the first step you configure the BW cube for using the columnstore in the SAP report MSSCSTORE. This step only changes the property of the cube in the SAP BW dictionary. In the second step you convert the cube to fit the columnstore configuration. In this step the indexes are created on the database. You can perform the second step either in report

MSSCSTORE or run an index repair in SAP transaction RSA1 or a BW process chain (see section Data Load below).

Step 1: Configure Cube

To configure a single cube, start report MSSCSTORE, enter the name of the cube, choose the desired cube type, make sure that the checkbox “change cube definition” is set, and start the report by pressing

Execute (F8).

Configure Cube Type for a Single Cube

You can also configure all BW cubes at the same time. For this you have to set a global cube type in the tab “Process all cubes” and start the report by pressing execute (F8). Note that this is not simply a default for new cubes (as it used to be in older versions of report MSSCSTORE). By setting a global cube type, the cube type of all existing cubes is changed and you will not be able any more to set the cube type individually for a single cube. Once you turn the global cube type off, all existing cubes keep their cube type. Afterwards you can change the cube type individually in tab “Process single cube”.

(7)

Step 2: Convert Cube

You can convert a cube from rowstore to columnstore (and vice versa) by repairing the database indexes of this cube in SAP BW. This is done automatically the next time you run a BW process chain, which contains the process chain step Index Repair for this particular cube. However, the easiest way to repair a cube is using SAP report MSSCSTORE. Here you can either configure and repair the cube in separate steps or perform both steps together.

Configure and Repair a Cube at the Same Time

The columnstore configuration is stored in the SAP BW dictionary independently of the installed SQL Server release. However, when repairing a cube, the SQL Server release is taken into account. The following table shows the index types on the database for e-fact and f-fact table of a cube after it has been repaired:

Row-Store (no CS) Basic CS Column-Store (CS)

SQL Server 2008 f-fact table b-tree b-tree b-tree

and lower e-fact table b-tree b-tree b-tree

SQL Server 2012 f-fact table b-tree b-tree b-tree

e-fact table b-tree read-only CS read-only CS

SQL Server 2014 f-fact table b-tree b-tree writeable CS

and higher e-fact table b-tree read-only CS writeable CS

Indexes on DB Dependent on Cube Configuration and SQL Server Release

There are some exceptions to the above rules: The f-fact table of a real-time cube is always using b-tree indexes. A cube on SQL Server 2012, which has a high precision key figure, and all APO cubes always use b-tree indexes.

All b-tree indexes of the fact-tables are dropped on the database when converting a BW cube from rowstore to writeable columnstore (on SQL Server 2014). However, these indexes might still be visible in the SAP data dictionary for the fact tables (with the flag “not for Microsoft SQL Server”). Therefore always check the index state on the database, not in the SAP data dictionary.

(8)

Running MSSCSTORE as a Batch Job

Once you have chosen your options, you can run report MSSCSTORE interactively by pressing the F8 button. By pressing F9 you can schedule a batch job instead. This is recommended when repairing the indexes or when processing all cubes rather than a single cube. You can check the job logs in SAP transaction SM37 or by pressing the SAP function key “SM37” (Ctrl+F1). Report MSSCSTORE also writes a detailed log into the SAP application log. This is the case when running in batch and when running interactively. To check the application log, press the SAP function key “Logs” (Ctrl+F2).

Running MSSCSTORE in Batch

Aggregates

MSSCSTORE always configures the aggregates of a cube the same way as the cube itself. When repairing a cube all aggregates are also repaired. When configuring or repairing a partition of a semantically partitioned cube, all partitions are processed at the same time. In many cases BW query performance is fast enough even without aggregates, once you have applied the columnstore. Therefore, we implemented the option to deactivate aggregates in report MSSCSTORE. However, by default we recommend keeping the aggregates. Once you have deactivated them, you can Activate and Fill them in report MSSCSTORE again. Activating simply creates the database tables. You cannot use aggregates in BW queries without filling them. Simply activating the aggregates makes sense, if you already have a BW process chain for filling the aggregates.

(9)

Checking Cube Status

You can check the status of a single cube by pressing the SAP function key “Status of cube” (Ctrl+F5) in report MSSCSTORE. This performs a full index check of all tables of the cube and its aggregates. In column “DDIC” you can see the BW cube configuration, which is always the same for all tables. In column “DB Expected” you can see the expected indexes on the database according to the cube configuration and the SQL Server release. Column “DB Check” shows whether all indexes are correctly created on the database. You can also see this in column “Exception” as an icon. If it is not green, you should run an index repair as described above.

Status of a Single Cube

You can get an overview of all cubes when using the F4 help in the cube input field (“Configure single cube:”). In the column “Exception” you can easily see all cubes for which the database indexes do not match their cube definition. In the example below, cube 0CCMDAHGR has a red status, because the cube is defined as columnstore while the indexes on the database are still using b-tree indexes. For performance reasons, the F4 help does not perform a full index check. It only checks the columnstore index. Therefore, a single missing b-tree index of a rowstore cube is not detected here. The status is then still green.

(10)

Loading Data

SAP BW uses Data Transfer Processes (DTPs) for loading data into a cube. If the cube is using a writeable columnstore index, the loaded data is not fully in columnstore format after the DTP has finished. Some rows might still be in the Delta Store. The procedure of moving all rows from the Delta Store to the columnstore is called rowgroup compression. SAP has integrated the rowgroup

compression into the BW Index Repair (also known as “Create Index” or “Repair Index”) function for cubes. For best query performance you have to run the BW Index Repair after each DTP.

SAP recommends running a database Update Statistics (also known as “Refresh Statistics”) after each DTP. This recommendation is not only related to writeable columnstore indexes. It is also valid for read-only columnstore or rowstore indexes. However, Update Statistics is not necessary when running a BW cube compression after the DTP. The BW cube compression automatically performs an Update Statistics.

Manually Running a DTP

Typically you are using SAP BW process chains for running DTPs. However, you might start a single DTP in SAP transaction RSA1. In this case you should also run an Index Repair and an Update Statistics once the DTP has finished.

Transaction RSA1 → Modeling → InfoProvider → Manage → Performance Running DTPs in BW Process Chains

You should always include the process chain steps Index Repair and Update Statistics in all your SAP BW data load process chains. The process chain step Index Delete is only used for rowstore indexes (of the BW f-fact table and the BW dimension tables). Therefore, it only has an impact on cubes using the rowstore or the read-only columnstore (for e-fact table). Index Delete has no impact on cubes using the writeable columnstore. Therefore, you do not need the process chain step Index Delete once you convert the cube to writeable columnstore.

(11)

Index Repair (“Create Index”) and Update Statistics (“DB Statistics”) within a BW Process Chain BW Cube Compression

BW cube compression (also known as “Condense” or “Collapse”) has several benefits. It reduces

the total number of rows in a BW cube by aggregating the request ID. This also results in reduced space consumption on the database. Cube compression reduces the number of database partitions in the f-fact table. For inventory cubes it creates reference points, which results in much faster inventory queries. For cubes using the read-only columnstore index, cube compression is mandatory. Without cube

compression all data is stored in the f-fact table, which cannot use the read-only columnstore index at all. However, cube compression also has some drawbacks. It is an additional step during data load, which takes some time. BW queries cannot run during BW cube compression of the cube or aggregate used by the query. Once a faulty loaded request has been compressed, it cannot be easily deleted any more. For Microsoft SQL Server SAP recommends regularly compressing BW cubes. This could be part of the daily DTP process chain. If the daily process chain runs concurrent with SAP BW queries, you can only include the BW cube compression in those process chains, which are running at night or weekend. Dependent on the cube type, we recommend the following:

Inventory Cubes

Compress most of the requests of an inventory cube, ideally all of them. This may significantly increase query performance on inventory cubes. In SAP BW process chains you can specify that all requests are

(12)

Property of BW Process Chain Step Collapse

Aggregates

Always fully compress all aggregates. This is done automatically during aggregate load or rollup, once you have defined the relevant cube property. A typical scenario is keeping all requests loaded within the last few days or weeks uncompressed in the basic cube, while compressing all requests in its

aggregates. In this case you can easily delete recent faulty loaded requests. Recent requests are not compressed in the basic cube. During request deletion, SAP BW automatically rebuilds (the fully compressed) aggregates.

Setting Cube Property: Always Fully Compress all Aggregates

Basic Cubes

Each uncompressed BW request uses its own database partition in the f-fact table. Although you can have up to 15,000 partitions, we recommend not having more than 1,000 partitions. Even a few hundred partitions might result in reduced BW query performance. The only way to get rid of these partitions is to compress the BW requests. Therefore, we strongly recommend keeping the number of uncompressed requests low. This is independent from the cube type and columnstore type.

(13)

For cubes using the rowstore you might still have a few hundred uncompressed requests. Cube

compression takes quite a long time for rowstore cubes. Furthermore, cube compression might not result in the expected space savings: For rowstore cubes, the e-fact table has no clustered index at all. This results in less space efficiency compared with tables having a clustered index (like the f-fact table). Inventory cubes store additional rows in the e-fact table, the reference points. This results in faster BW queries on BW compressed inventory cubes. On the other hand it increases the size of the e-fact table. For all these reasons many customers perform BW cube compression seldom (or not at all) when they are using rowstore cubes. However, in any case you should make sure that the number of partitions in the f-fact table does not exceed about 1,000.

For cubes using the read-only columnstore, you have to compress as many requests as possible. BW cube compression moves data from the f-fact table (having rowstore indexes) to the e-fact table (having only columnstore index). Without BW cube compression you simply cannot benefit from the read-only columnstore.

For cubes using the writeable columnstore, you can also use the columnstore on the f-fact table. However, you still should not have more than a few hundred uncompressed requests. Cube compression on a writeable columnstore cube is much faster compared with cube compression on a rowstore cube. When using cube compression, you do not need to include the process chain step Update Statistics in your DTP process chains (because update statistics is performed during BW cube compression). A typical process chain looks like this:

(14)

Best Practices

For using the columnstore with SAP BW we recommend the following:

 Keep the database index statistics up-to-date. You can achieve this by adding one of the following process chain types at the end of your SAP BW cube load (DTP) process chain: “Compression of the InfoCube” or “Construct Database Statistics”. (You can manually update the database index statistics of a cube using SAP transaction RSA1 –> Modeling –> InfoProvider –> Manage Cube –> Performance –> Refresh Statistics)

 Ensure that the database options Auto Create Statistics and Auto Update Statistics are turned on. For SAP BW we typically turn off the database option Auto Create Statistics Async. Keep in mind that all index statistics are dropped once you convert a table from rowstore to columnstore. They are automatically created once the first queries are running (because of the database option Auto Create Statistics). Therefore the queries may take longer than expected, when running immediate after the conversion to columnstore.

 Make sure that all columnstore rowgroups are compressed after each data load. Therefore, you have to add the process chain type “Generate Index” after a DTP in a process chain. (You can manually compress all columnstore rowgroups of a cube using SAP transaction RSA1 –> Modeling –> InfoProvider –> Manage Cube –> Performance –> Repair DB Indexes)

 Run BW Cube Compression regularly. You can keep those requests uncompressed, which were loaded within the last few weeks. However, for best query performance, the majority of all requests in a cube should be compressed. For inventory cubes, ideally all requests should be compressed.

 BW Partitioning of the e-fact table by time dimension can further increase query performance.

However, these performance improvements are often overrated.

 BW Aggregates can increase query performance to an optimum. However, the administrative

overhead of aggregates can be significant. Therefore keep the number of aggregates low. Configure all cubes to always use fully compressed aggregates (option “Compress After Rollup”)  Use BW Exceptional Aggregations only when there is a business need. Exceptional

aggregations are typically performed on the BW application server (which is single-threaded). This might consume a significant proportion of the whole BW query time and cannot be accelerated by any database technology – like the Columnstore.

 You can further decrease the needed disk space for the columnstore by changing the data compression type COLUMNSTORE_ARCHIVE using report MSSCOMPRESS. This results in additional disk space savings of roughly 20%. However, this also might result in slower query performance of 20%. Therefore we do not recommend using COLUMNSTORE_ARCHIVE for SAP BW.

 Initially keep the default values of the RSADMIN parameter MSS_MAXDOP_QUERY (2 for SQL Server 2012, 4 for SQL Server 2014). Increasing this value can result in a memory bottleneck, which is counterproductive. On database servers having plenty of CPU threads and RAM you may increase MSS_MAXDOP_QUERY to 8.

 You can speed-up Data Transfer Processes (DTPs) into a cube by setting the RSADMIN parameter RSDD_ENABLE_MASS_CUBE_WRITE=X (for SAP BW 7.30 and higher).

 Make sure that the recommended trace flags for SQL Server are set, in particular trace flag 9481 for SQL Server 2014.

 Configure sufficient memory for SQL Server and make sure that Locked Pages In Memory is used for SQL Server

 Create many tempdb data files (at least 16) and store them on a fast disk (SSD). A single tempdb log file is sufficient.

 For SAP BW, use the SQL Server recovery model BULK LOGGED whenever possible (as long as your backup strategy allows it and you are not using database mirroring). Converting the different cube types and any other index operation can be significant faster when using BULK LOGGED.

(15)

Copyright

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE.

The information contained herein may be changed without prior notice.

Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.

These materials are provided by SAP SE and its affiliated companies (“SAP SE Group”) for informational purposes only, without representation or warranty of any kind, and SAP SE Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

SAP SE and other SAP SE products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE in Germany and other countries.

Please see

http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark

Using SQL Server 2014 In-Memory Optimized Columnstore with SAP BW