Deutsche Bank – Finance IT
Migration Oracle Exadata
1
Motivation
Migration (Phase 2)
3
2
PoC (Phase 1)
4
Observations
Agenda
5
Volker Bettag, Architect
Dr. Michael Dreier, Infrastructure Manager
Randolf Geist, Oracle Specialist
Erwin Heute, Oracle Specialist
Jens Koch, MicroStrategy Infrastructure/Project Manager
Deutsche Bank Data Centre
Contact
Dr. Marcus Prätzas, Program Manager
Deutsche Bank AG
Wilhelm-Fay-Str. 31-37
D-65936 Frankfurt
Topic Area
RWA (Basel I / II)
EC / EL / GVA
Output / Activity
RWA calculations, Monthly driver analysis, Quarterly COREP reporting
Monthly Basel II reporting, EPE, MR-RWA
EC / EL / Average Active Equity calculation and reporting
GVA for not impaired corporate credit exposure
Others
Group Derivative Bookings, Global Securities Netting, Banking bookcollateral, Country risk, ...
Disclosure
20-F Item 11 (Risk Section, 37 pages), Footnotes, Annual ReportAnalyst presentations, Interim Report, Financial Data SupplementGerman Regulatory
KWGCapital, KWG 13 / 14, Financial Conglomerate disclosureInfluence rule making and interpretation
Daily Derivatives
Daily derivatives counterparty riskProvide EPE calculationsBusiness Background – Drivers
Motivation – Daily Processing
Performance Demand 2010
Core Process* Run Times by Quarter
Q1 2010
SAS deployed on AMD CPUs with internal PCIe SSD storage
Q2 2010
InfiniBand private interconnect, Enhanced parallel processing
Q3 2010
Datawarehouse Infrastructure PoC using Oracle Exadata and SSD based storage severs. Oracle Infrastructure setup.
Q4 2010 – Exadata
Oracle Infrastructure go-live
Q1-Q3 2011 – Exadata
Migration of full environment
Target was ~10h
i.e. 50% reduction in non-calculation steps required
*
Technical Process
Distributed Engines Netting Expected Loss EC Basel II KWG RWACredit Risk Engines
Data Warehouse Disclosure Process Control B/S Netting GVA EL / EC Country Risk KWG 13 / 14 Principle I/II Basel II Regional QA Regional QA Daily QA Monthly Source Monthly Source Monthly Source Monthly Source Daily Source Daily Source Daily Source Ext. Calc SAS
PoC
– Testpoints
Input Area Master Area Reporting Area
Data
Delivery Input Master Report
Calculation View, Extract
1
2
3
4
5
Five key production processes have been chosen
A full set of production data ist used for testing
The tests were executed in DB datacenter
The requirement has been set to 50% performance increase compared to the
monthly production setup at the time.
PoC – Testpoint Characteristics
Testpoint 1 – Integration Function (TP1)
Data Transformation between two Oracle schemas. CPU power consumption (e.g.
currency conversion) as well as large sequential IO operations. The IO is done in parallel and includes substantial DML.
Testpoint 2/3 – SAS Engine Interface (TP2/3)
Perform a data down- and upload to the SAS Engine environment. As this is not a core database functionality rather than a regression test of the InfiniBand connection not further listed here.
Testpoint 4 – Starbuilder (TP4)
Large single threaded operation, where CPU and IO performance are equally essential. Compared to TP1 these are far less complex operations.
Testpoint 5 – Microstrategy Reporting (TP5)
Random IO and massive parallel execution. Representative set of 110 and 470 reports from production
.
Testpoint 1 Results – Integration Function
54% performance gain on Exadata (V2)
About 25 test-runs with different Oracle / System configuration settings have been executed for each environment. Minor application changes.
The maximum parallelism causes internal Oracle contention issues. 5 compute nodes show best performance.
TP1
Oracle Exadata Scalability
With the exception of some parts that are executed across all available nodes the scalability has been tested using a variable number of compute nodes
The optimum is reached with 5 nodes. Beyond that no improvement has been observed.
Oracle Exadata Scalability – Data Volume
When doubling the data volume the runtime increases by 7%, for a factor of three the runtime increases by 19%, with a factor of 4 the runtime gets 34% longer.
Disaster Recovery – Active Data Guard
1. Disaster recovery solutions utilizing Oracle Data-Guard for replication.
Result
Oracle Exadata achieves an overall better performance improvement of ~55%
In particular the better reporting performance of the Oracle Exadata adds significant more value.
The feature of hybrid column compression (available on Exadata only) enables a data reduction for historical data down to ~25%.
Lower cost than the previous solution (traditional SAN based)
Observation
The PoC showed contention-issues effecting the achievable performance and scalability of Oracle RAC on the Exadata V2. This occurs in particular when heavily using DDL like truncating partitioning and rebuilding indexes on other partitions in parallel.
Conclusion
Migration of full environment using V2-8
Architecture Solution (2011)
Oracle Exadata V28 Datacentre #1
Full Rack #1 (45 TB available for data + FRA)
Monthly Prod.
(10 TB)
Daily Prod. (Data Guard Copy)
(6 TB)
Flash Recovery Area (all databases - 22 TB)
Cluster Filesystem (Buffer, etc. 4 TB)
Monthly Production (Data Guard Copy)
(10 TB)
Daily Prod.
(6 TB)
Flash Recovery Area (all databases – 22TB) Cluster Filesystem (Buffer, etc. 4 TB) Oracle Exadata V28 Datacentre #2 Monthly UAT (10 TB) Daily UAT (6 TB)
Flash Recovery Area (all databases - 22 TB) Cluster Filesystem (Buffer, etc. 4 TB) Oracle Exadata V28 Datacentre #2 Oracle Exadata V2 Datacentre #1 INT(10 TB) DEV(3 TB)
Flash Recovery Area
(all databases - 22 TB)
Cluster Filesystem (Buffer, 4 TB)
DR (Data Guard Copy) Clone (Snapshot Copy)
Full Rack #2 (45 TB available for data + FRA)
Full Rack #3 (45 TB available for data + FRA)
Full Rack #4 (45 TB available for data + FRA) (existing system)
Regional QA
(2 TB)
Regional OA. (Data Guard Copy)(2 TB) QA UAT (2 TB) QA INT (1 TB) QADEV (0.5 TB) Contingency (7 TB)
Migration started Q1 – 2011
Core functionality was proven & further performance gains indentified (index
usage on ODM) in the PoC
PoC complete and (daily) system live
Oracle Support for go live, environment review, tuning tips. All 12 findings
during POC had been resolved in < 3 weeks and addressed by patch bundle
sets.
Migration Log Book Q2 – Q3 2011
May
3 ODMs have been delivered and
handed over from Oracle to Data
Centre
HW & SW install in ~10 days
(Oracle)
ODMs have been handed over
from data centre to project 2
weeks later
June / July
First full environment (incl. SAS,
NFS, etc.) established
Migration rehearsal & testing
cycles
Integration testing in Jul
August
Dress rehearsal
September
DataGuard lines established
Improved performance with 10G line to
be compared with Q1 POC on 1GB
October / November
Last cell patches applied on all 3 ODMs
Final test cycles
Summary
More than one year experience with the software stack on Oracle Exadata
processing data on a daily, weekly and monthly data
Performance, cost and storage objectives have been met
No Hardware failures detected so far, important patches applied
Exadata v2.8 configuration is to be rated above commodity level (using SAS
disk only)
Two powerful database nodes proves higher performance & stability vs. a
smaller node
Effizienteres Kreditrisikoreporting dank
optimierter Data Warehouse Infrastruktur
FAZIT
Der Einsatz der Oracle Exadata Database Machine für das Data Warehouse für das Kreditrisiko-reporting steht für 50% weniger Laufzeit sowie 75% geringeres
Datenvolumen – und das bei rund 20% niedrigeren Kosten.
DAS UNTERNEHMEN
• Die Deutsche Bank ist eine führende globale Investmentbank mit einem bedeutenden Privatkundengeschäft sowie sich gegenseitig verstärkenden Geschäftsfeldern.
• Branche: Finanzdienstleistungen
• Mitarbeiter: > 100.000
DIE HERAUSFORDERUNG
• Die Analyse von Kreditrisiken und zeitnahes Reporting gewinnt immer größere Bedeutung.
• Die gestiegenen Datenvolumina sowie die umfangreichen Berechnungen stellen eine Herausforderung für das zeitnahe Reporting dar. Dem zu begegnen erfordert den Aufbau einer zukunftsorienterten, performanteren Infrastruktur.
• Mehr als 500 Benutzer greifen aktiv auf die verschiedensten Aspekte im DWH zu. Tausende Abnehmer werden weltweit mit Informationen in unterschiedlichen Formaten versorgt.
ORACLE PRODUKTE & SERVICES
• Oracle Exadata Database Machine
• Oracle Linux
• Oracle Customer Support
DIE LÖSUNG
• Quartalsweise, monatliche, wöchentliche bzw. tägliche Bereitstellung der Berichte mit massiv verbesserter Performance
• Laufzeit zur Generierung der täglichen Reports um 50% verkürzt
• Dank der Storage-Kompression wurde das Datenvolumen um 75% reduziert
• Kosteneinsparungen von etwa 20%, reduzierte Platzanforderungen und weniger Stromverbrauch
