• No results found

HP D3600 Disk Enclosure 4,000 Mailbox Resiliency Exchange 2013 Storage Solution

N/A
N/A
Protected

Academic year: 2021

Share "HP D3600 Disk Enclosure 4,000 Mailbox Resiliency Exchange 2013 Storage Solution"

Copied!
31
0
0

Loading.... (view fulltext now)

Full text

(1)

Technical white paper

HP D3600 Disk Enclosure 4,000

Mailbox Resiliency Exchange

2013 Storage Solution

Table of contents

Overview ... 2

Disclaimer... 2

Features of the tested solution ... 2

HP D3600 Disk Enclosure ... 3

HP ProLiant DL360 Gen9 Server ... 3

HP Smart Array P441 Controller ... 4

Solution description ... 5

Targeted customer profile ... 5

Storage configuration ... 5

Mailbox configuration – primary site operational ... 7

Mailbox configuration – primary site offline and single server failure at secondary data site (planned

or unplanned)... 8

Tested deployment ... 9

Best practices ... 10

Test results summary ... 11

Storage performance results ... 12

Database backup / recovery performance ... 12

Conclusion ... 12

Appendix A – stress testing ... 13

Appendix B – performance testing ... 18

Appendix C – streaming backup testing ... 23

Appendix D – soft recovery testing ... 25

(2)

Overview

This document provides information on the HP D3600 Disk Enclosure and HP ProLiant DL360 Gen9 Server solution for

Microsoft® Exchange Server 2013 (Exchange), based on the Microsoft Exchange Solution Reviewed Program (ESRP) –

Storage program. For any questions or comments regarding the contents of this document, see the For more information

section.

The ESRP – Storage program was developed by Microsoft Corporation to provide a common storage testing framework for

vendors to provide information on its storage solutions for Microsoft Exchange Server software. For more details on the

Microsoft ESRP – Storage program, see: http://technet.microsoft.com/en-us/exchange/ff182054.aspx

Tested with: ESRP – Storage Version 4.0

Test Date: May 2015

Disclaimer

HP produced this document independently of Microsoft Corporation. Microsoft Corporation expressly disclaims

responsibility for, and makes no warranty, express or implied, with respect to the accuracy of the contents of this document.

Features of the tested solution

This solution implements the Database Availability Group (DAG) mailbox resiliency feature found in Microsoft Exchange

Server 2013 (Exchange). This feature is the base component of the high availability and site resilience framework built into

Exchange. A DAG is a group of up to 16 mailbox servers that hosts a set of databases and provides database-level recovery

from failures that affect individual servers or databases.

This solution brief describes a tested configuration for deploying Microsoft Exchange Server 2013 (Exchange) in a 4,000-

user, mailbox-resilient environment. The performance results and best practices outlined in this document provide tested

guidelines for configuring Exchange with the HP D3600 disk enclosure, HP ProLiant DL360 Gen9 Server, and HP Smart Array

P441 controller in a direct-attached storage (DAS) solution.

This solution used a single DAG to support 4,000 mailboxes, each with a 10GB mailbox capacity and a work profile of 200

messages sent and received for each user, each day.

To provide mailbox resiliency, there are four copies of each database in an active/active environment; two copies located at

the primary site and two copies located at the secondary site.

Microsoft’s preferred architecture for Exchange 2013 is an Active/Active site deployment with active and passive databases

equally distributed across the primary and secondary data centers. In this four-copy configuration, the recommendation is

the Primary Data Center and Secondary Data Center balance the user access equally across sites during normal operations.

All four servers in the DAG host active databases supporting 4,000 users across both sites. We tested worst case scenario

with one server running all active databases.

The performance results outlined in this paper are from the testing of a single building block at the secondary site. See the

section Test results summary for more information.

Note

HP has developed the HP Sizer for Microsoft Exchange Server 2013 to assist customers with proper server and storage

sizing for their Exchange Server 2013 deployments. The algorithms developed and implemented in this tool originate from

extensive testing of Exchange Server 2013 server roles and technologies, HP’s depth of experience with Microsoft

Exchange, and intricate knowledge of server CPU, memory, and I/O subsystems. Designed to simplify configuration across a

broad range of deployments, this tool is free to download and use. For more information, visit:

hp.com/solutions/microsoft/exchange2013/sizer

In addition to the HP Sizer, HP recommends that customers use the Microsoft Exchange Mailbox Server Role Requirements

calculator. If this calculator is used as a starting point, it can be imported or read into the HP Sizer using the Load Exchange

calculator button.

(3)

HP D3600 Disk Enclosure

Overview

Manage your increasing small and midrange business storage needs by deploying the HP D3600 disk enclosure, the next

generation 12Gb SAS low cost, flexible, and tiered external storage system. The HP D3600 disk enclosures are ideal for

small application environments in small and medium businesses, remote offices, and departmental locations.

New 12Gb SAS enclosures – Large Form Factor (LFF) D3600 with 12 drive bays – offer modular solutions to simplify the

capacity expansion of HP ProLiant server environments to external storage, without moving to SAN or NAS. Buy what is

needed today and purchase additional capacity as data storage needs grow.

The D3600 enclosures support direct attach storage to ProLiant Servers with the HP Smart Array P441 Controllers.

Figure 1: Front view of the HP D3600 Dual I/O Module Disk Enclosure

Key features and benefits

Affordable Storage for Capacity Demand

The LFF D3600 disk enclosure supports up to 12 (twelve) LFF SAS drives for a maximum capacity of 7.2TB with 12G

600GB SAS drives or 96TB with 12G 8TB SAS MDL or 6G 8TB SATA MDL drives. Eight D3600 large form factor (LFF)

enclosures can be connected together from a single controller port.

HP D3000 Enclosures’ 12Gb SAS enables a higher/faster data transfer rate doubling the current transfer rate of 6Gb

solutions providing crucial bandwidth.

Provides investment protection by allowing you to purchase HP D3600 storage based on current needs, and plan for the

future without hidden management charges. Achieve increased data speed with enterprise-class dual-port SAS drives.

Provides a higher level of redundancy and reliability, reducing single points of failure within the storage network.

Flexible and Scalable with Room to Grow

Flexibility to mix and match SAS, SATA, and Solid State drives in the same enclosure, and extend the benefit of tiered

storage to direct attach customers.

Designed to give room to grow as your storage demands increase, the D3600 LFF can grow up to 96 drives.

Easy to Manage

Supports HP SmartDrive Carrier and Gen8 ProLiant Drives, enabling a single set of common drives for Gen8 ProLiant

servers and D3000 Enclosures.

Uses familiar HP ProLiant management tools, including HP Systems Insight Manager and HP Array Configuration Utility

(ACU) software.

Compatible with selected HP Smart Array SAS Controllers and HBAs.

For more information on the D3600 Disk Enclosure, visit hp.com/go/d3000.

HP ProLiant DL360 Gen9 Server

The HP ProLiant DL360 Gen9 Server delivers a 1U chassis, with up to two processors delivering an optimal unit that

combines high-performance, low energy consumption, improved uptime, and increased density. Intel® Xeon® E5 2600 v3

processors deliver up to 70% performance gain and 36% efficiency over the Gen8 family.

The HP ProLiant DL360 Gen9 Server, combined with HP DDR4 SmartMemory, supports up to 1.5TB while delivering up to

14% performance improvements over previous generations. Manage your DL360 Gen9 Server in any IT environment by

automating the most essential server lifecycle management tasks: deploy, update, monitor, and maintain with ease.

The HP ProLiant DL360 Gen9 Server is dedicated to meet the most demanding dynamic computing workloads.

(4)

Key features of the HP ProLiant DL360 Gen9 Server

Provides flexibility of networking bandwidth and fabric to adapt and grow to changing business needs.

Achieve greater

capacity with flexible drive configuration options, with up to ten Small Form Factor (SFF) or four Large Form Factor (LFF)

HP SmartDrives, delivering optimal performance, capacity, and reliability. Meet various customer segments and workload

requirements affordably.

The HP ProLiant DL360 Gen9 Server supports improved ambient temperature standards with HP Extended Ambient

Operating Support (ASHRAE A3 and A4), helping to reduce cooling expenses.

With HP ProLiant DL360 Gen9 Server, HP OneView provides converged management for automation simplicity across

servers, storage, and networking.

Embedded management can deploy, monitor, and support DL360 Gen9 Server remotely, out of band with HP iLO.

Optimize firmware and driver updates and reduce downtime with HP Smart Update, consisting of HP SUM (Smart Update

Manager) and SPP (Service Pack for ProLiant).

HP ProLiant DL360 Gen9 Server comes with a complete set of HP Technology Services, delivering confidence, reducing

risk, and helping customers achieve agility and stability.

Figure 2: HP ProLiant DL360 Gen9 Server

HP Smart Array P441 Controller

The HP Smart Array P441 Controller is a low-profile, PCIe3 x8, 12Gb/s Serial Attached SCSI (SAS) RAID controller that

provides enterprise-class storage performance, increased external scalability, and data protection for HP ProLiant Gen9

servers. It features eight external physical links and delivers increased server uptime by providing advanced storage

functionality. This includes online RAID level migration (between any RAID levels) with flash backed write cache (FBWC),

global online spare, and pre-failure warning. Key features include:

Increased Performance for Applications

The HP Smart Array P441 Controller delivers 12Gb/s SAS connectivity on HP ProLiant Gen9 Servers when connected to

12Gb/s external storage devices for increased performance in messaging, database, or general server applications.

Offers transportable 4GB DDR3-1866 Flash Backed Write Cache, providing up to 14.9GB/s maximum cache bandwidth to

reduce latency in write-intensive applications that require heavy logging, such as databases.

Optional HP SmartCache provides read and write acceleration for workloads such as databases or web pages.

Data Protection with Encryption Capability

Provides long-term data retention with 4GB Flash Backed Write Cache (FBWC) for improved data reliability.

The HP Smart Array P441 Controller offers optional HP Secure Encryption capability that protects data at rest on any bulk

storage attached to the controller.

Configuration Is Efficient and Simple

The HP Smart Array P441 Controller and the attached storage devices are configured and managed by the easy–to-use

HP Smart Storage Administrator software, which is included in the server intelligent provisioning.

Data compatibility within the HP Smart Array Controller family allows simple and easy migration at any time. When needs

for higher performance, capacity, and availability increase, such as when migrating an existing just a bunch of disks

(JBOD) to a more advanced disk array configuration, tasks can be performed efficiently and reliably.

HP SSD Smart Path feature

The HP SSD Smart Path feature included in the Smart Array software stack improves Solid State Disk (SSD) read

performance. With up to 4x better SSD read performance, HP SSD Smart Path chooses the optimum path to SSD and

accelerates reads for all RAID levels and RAID0 writes.

HP SmartCache

(5)

For more information on HP Smart Array P441 Controller, visit:

http://www8.hp.com/in/en/products/iss-controllers/product-detail.html?oid=6995461

Figure 3: HP Smart Array P441 Controller

Solution description

This solution was designed with two HP D3600 Enclosures for a 4,000 user Exchange 2013 deployment, with high

availability through both Smart Array RAID0 and Exchange 2013 DAG copies, using four active servers in the solution.

Mailbox resiliency data protection is provided by configuring multiple database copies on a minimum of two HP D3600

Enclosures. Only one HP D3600 was included in the testing described in this document. This document provides best

practices to help administrators implement an Exchange deployment with the following requirements.

Targeted customer profile

A single Database Availability Group deployment with two mailbox servers in the primary site and two mailbox servers in

the secondary site, for a total of four mailbox servers across the entire solution.

4,000 mailbox users (1,000 user mailboxes per server)

Exchange 2013 profile of 0.16 IOPS workload per user (200 messages sent and received per user, per day, with an

additional 20% headroom added to base IOPS)

10GB mailbox size

Four copies of each mailbox database (one active / three passive)

Twenty Two databases per server (eleven active and eleven passive)

Background Database Maintenance (BDM) enabled in testing

Designed for server and site resiliency

11 6TB 12G SAS 7.2K LFF (3.5-inch) Dual Port MDL Hard drives per array for the Exchange databases and transaction logs

Storage deployed as single drive RAID0. See the following Storage configuration section for more information

Storage configuration

This solution uses the D3600 Disk Enclosure, attached to a Smart Array P441 controller and configured using HP Smart

Storage Administrator. For storage performance and redundancy, the HP D3600 and P441 are cabled as a dual-domain

configuration. This configuration uses independent SAS data paths from the servers to the storage. The dual-domain

communication path prevents interruption of disk I/O in the event of a data path failure. The Exchange database and

transaction logs are stored on the HP D3600 array; the Exchange restore volume are located on the server.

Multiple databases per volume are available in a Smart Array RAID0 scenario in Exchange 2013 allowing for active and

passive copies (including lagged copies) to be mixed on a single disk, enabling better disk utilization. See Microsoft’s

(6)

A single “Array X” is created for each physical drive. Then a single Logical Drive is created from all free space in the array

using a 256KB or larger strip size and RAID0. Each RAID0 disk group consists of one 6TB disk, for a total of eleven RAID

groups. Databases and logs reside on the same volumes since Exchange was configured for multiple database copies.

HP also recommends designating one extra disk per server as a spare in the case of disk failure. Refer to

http://blogs.technet.com/b/pfemsgil/archive/2013/03/24/exchange-2013-auto-reseed.aspx to configure spare volumes

for automatic reseed.

HP Smart Storage Administrator

HP SSA is the main tool for configuring arrays on HP Smart Array controllers. It exists in three interface formats: the HP SSA

GUI, the HP SSA CLI, and HP SSA Scripting. All formats provide support for configuration tasks.

HP Smart Storage Administrator Scripting Cookbook

http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c03924344

HP Smart Array Controllers User Guide for HP ProLiant Gen9 Servers

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04441382

Figure 4 shows the logical and physical view of the storage from the HP Smart Storage Administrator (SSA) utility.

Figure 4: HP Smart Storage Administrator interface

For more details on implementing a dual-domain SAS configuration with the D3600, see

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04219600-1.pdf

(7)

Figure 5: Rack mounted view of the D3600 Disk Enclosure and the HP ProLiant DL360 Gen9 Server, cabled in a dual-domain configuration

1.

HP ProLiant DL360 Gen9 Server

2.

HP D3600 Drive Enclosure

3.

Primary path

4.

Secondary path

Mailbox configuration – primary site operational

Figure 6 shows a fully functional scenario using four servers in the DAG hosting active and passive databases from two

different HP D3600 enclosures, one per data center. In a data center failover scenario, the two servers in the SDC will be

able to serve all the databases

4,000 mailbox users

0.16 IOPS workload per user (200 messages sent and received per user, per day, with an additional 20% headroom)

10GB mailbox size

Four copies of each mailbox database (one active / three passive)

Twenty Two databases per server (eleven active and eleven passive)

11 6TB 12G SAS 7.2K LFF (3.5-inch) Dual Port MDL hard drives

Designed for server and site resiliency

(8)

Figure 6: Database and copy layout creating Mailbox resiliency.

Mailbox configuration – primary site offline and single server failure at secondary data

site (planned or unplanned)

Figure 7 shows the tested mailbox layout for active database copies when the primary site and one of the servers in the SDC

site are unavailable due to a planned or unplanned event. This solution demonstrates the worst-case scenario with one host

in the SDC hosting 22 active databases, with the following requirements:

4,000 mailbox users

0.16 IOPS workload per user (200 messages sent and received per user, per day, with an additional 20% headroom)

10GB mailbox size

Primary site’s database copies and one of the two servers at the Secondary site are unavailable due to a planned or

unplanned event

The mailbox server in the Secondary site hosts 22 database copies all active

(9)

Figure 7: Primary site unavailable and single server failure at Secondary site.

Tested deployment

The following tables summarize the tested environment.

Simulated Exchange configuration

Number of Exchange mailboxes simulated

4,000

(10GB mailbox size)

Number of Database Availability Groups (DAGs)

1

Number of active mailbox servers per DAG

4 (1 hosting active mailboxes in test)

Number of active mailboxes per server

4,000 (tested scenario)

Number of databases per server

22

Number of copies per database

4 (2 at primary / 2 at secondary)

Number of active mailboxes per database

182

Simulated profile: I/O operations per second per mailbox

(IOPS, include 20% headroom)

0.16

Database and Log LUN size

2.79TB

Total database size for performance testing

40TB

Background database maintenance (BDM)

Tested with 24/7 BDM enabled

% formatted storage capacity used by Exchange database

65.12%

40.03TB / 61.47TB

(10)

Storage hardware

Storage Connectivity

12Gb/s SAS

Storage model and firmware version

HP D3600 Dual I/O Module Disk Enclosure

Firmware Version: 1.00

Number of storage controllers per server

1

Number of SAS ports on storage

2

Maximum bandwidth of storage connectivity to host

24Gb/s

(2 ports x 12Gb/s)

Storage Controller and Firmware Version

HP Smart Array P441

Firmware Version: 2.14

Controller Cache

4GB Flash-Backed Write Cache (FBWC , 25/75 read/write)

Host server type

HP ProLiant DL360 Gen9

Total number of disks tested in solution

11

Maximum number of disks that can be hosted in the tested

configuration

96 LFF drives (12 per D3600 x8)

Storage software

Smart Array P441 Driver

63.8.0.64

Host OS

Microsoft Windows Server® 2012 R2

ESE.dll file version

15.00.1076.009

Storage disk configuration (mailbox and log disks)

Disk type, speed, and firmware version

HP 6TB 12G SAS 7.2K LFF (3.5in) Dual Port MDL

Firmware Version: HPD2

Raw capacity for each disk (TB)

6TB

Number of physical disks in test

11

Total raw storage capacity (TB)

66TB**

RAID level & Set Size

RAID0

Stripe Size (KiB)

256

Total formatted capacity (TB)

61.47TB

Storage capacity utilization

93.13%

61.47TB /66TB

(Formatted capacity/ Total Raw capacity)

Database capacity utilization

60.65%

40.03TB /66TB

(Total database size / Total Raw capacity)

** Capacity of drives, and therefore the raw data capacity of the disk array, based on 1KB = 1000 bytes, not 1024 bytes. This means that

1GB = 1,000,000,000 bytes, not 1,073,741,824 bytes

Best practices

When deploying Exchange, it is important to not only size for storage capacity, but to also consider the storage performance

demands. Validating the storage subsystem is capable of supporting the anticipated and future workload minimizes the risk

of the disk subsystem becoming an I/O bottleneck.

Exchange Server is a disk-intensive application. For Exchange 2013 best practices on storage design from Microsoft, see

http://technet.microsoft.com/en-us/library/ee832792(v=exchg.150).aspx.

(11)

Drivers and Firmware

Ensure that firmware and drivers for all hardware in the solution are up to date. For more information, see:

hp.com/go/support.

Physical Drive Write Cache

Unlike controller-level cache, individual drive write caches are not protected. If the server loses power, or a hot plug drive

is physically removed from the array during operations, unwritten data in the physical drives’ write caches will be lost. HP

recommends disabling Physical Drive Write Cache on the HP Smart Array P441 Controller. For more information, see:

http://h20565.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c02278484

Accelerator Ratio and Flash-Backed Write Cache (FBWC)

HP recommends setting 75 percent write cache and 25 percent read cache (battery-backed cache).

Jetstress 2013

Before installing Exchange Server 2013, test your storage solution with Jetstress to ensure it is configured properly and

meets your requirements. For more information on Jetstress, see:

microsoft.com/en-us/download/details.aspx?id=36849.

Mailbox Database Copies

When deploying a JBOD configuration, ensure that there are at least three copies for each Exchange mailbox database in

the solution. This solution was tested with four database copies.

Stripe Size

For Exchange deployments, HP recommends using the largest strip size for your selected RAID level. The strip size was

configured for 256KB. For more information, see: http://technet.microsoft.com/en-us/library/ee832792.aspx.

RAID Level

Database volumes used in this solution were configured as single drive Smart Array RAID0. Provisioning of other RAID

types is supported but not recommended for this Exchange Server solution.

BitLocker Drive Encryption

BitLocker Drive Encryption provides considerable protection of data volumes. For more information on Windows®

BitLocker, see: https://technet.microsoft.com/en-us/library/c61f2a12-8ae6-4957-b031-97b4d762cf31

In this solution, data volumes are encrypted using a command-line tool that can be used in place of BitLocker Drive

Encryption. For more information on the command-line tool, see:

https://technet.microsoft.com/en-us/library/ff829849.aspx

Windows Volumes

For best performance, format the Windows volumes with 64KB allocation unit size.

Test results summary

This section provides a high-level summary of the test data from the ESRP and a reference to the detailed HTML reports,

generated by the ESRP testing framework.

For solutions that use a uniform building block style, in which each building block is independent and isolated, ESRP requires

testing only one building block. For this solution, a building block consisted of one HP ProLiant DL360 Gen9 Server and one

HP D3600 Disk Enclosure, with 11 x 6TB LFF SAS drives hosting all 4,000 mailboxes.

Reliability

Reliability testing exercises the storage with the maximum sustainable Exchange I/O load for 24 hours. The goal is to verify

that the storage can handle the maximum I/O load for an extended amount of time. Both log and database files are

analyzed for integrity after the stress test to confirm that no database or log corruption occurred.

After running the 24-hour test, we found no relevant errors reported in the event log for the storage reliability testing, and

no errors reported for the checksum process.

(12)

Storage performance results

Storage performance testing exercises the storage with the maximum sustainable Exchange I/O load for two hours. The

test measures storage I/O response times under the maximum sustainable Exchange I/O load. The following data is the

sum of all logical disk I/Os and an average of all logical disk I/O latency in the two-hour test duration.

This solution design is an Exchange deployment with a single DAG. The DAG has two mailbox servers in the primary site and

two mailbox servers in the disaster recovery site. Ideally, the two servers in the disaster recovery site should host all active

mailboxes when the primary site is offline (planned or unplanned.)

The performance results table below shows the worst-case scenario when one server is running all four databases.

Database I/O

Needed Disk Transfers/sec*

640

Database Disk Transfers/sec

1019

Database Disk Reads/sec

713

Database Disk Writes/sec

305

Average Database Disk Read Latency (ms)

17.98

Average Database Disk Write Latency (ms)

1.848

Transaction Log I/O

Log Disk Writes/sec

73.34

Average Log Disk Write Latency (ms)

<1 ms

* This row represents the total IOPS necessary to satisfy the IOPS for all mailboxes in the solution.

Database backup / recovery performance

This section contains two test reports. The first report measures the sequential read rate of the database files; the second

report measures the recovery/replay performance (playing transaction logs into the database). The tests measure the read

rates and log replay rates with all mailboxes under concurrent or simultaneous load.

Database Read-Only Performance

This test measures the maximum backup rate for Volume Shadow Copy Service (VSS) aware backup applications. The

following table shows the average and sum of the database reads.

Average MB read/sec per database

48.48

MB read/sec total

1066.57

Transaction Log Recovery / Replay Performance

This test measures the maximum rate that log files can play against the databases. The following table shows the average

rate for log files played. Each log file is 1 MB in size.

Average number of log files played

506

Average time to play one log file (sec)

7.24

Conclusion

The information discussed in this solution brief highlights the tested performance results and configuration best practices

for an Exchange solution supporting 4,000 10GB mailboxes and a 0.16 IOPS workload for each user (200 messages sent

and received per user, per day with additional 20% headroom). The tested solution included an HP D3600 Disk Enclosure

with 11 6TB 7.2K LFF SAS drives for the mailbox database storage and transaction logs and 22 Database volumes

(13)

to initialization. Testing demonstrated that the storage subsystem was capable of an aggregate 1019 disk transfers per

second and contains the necessary capacity to support 4,000 Exchange users with 10GB mailboxes and provides additional

performance headroom.

The test results and data presented in this document are from the ESRP test framework. Test results are only a guide;

customers should test their specific solution to make sure it meets their requirements.

The ESRP program is not a benchmarking program. The tests do not measure the maximum throughput for a given

solution. Rather, program focus is on producing recommendations from vendors for Exchange Server storage solutions.

Data presented in this document should not be used for direct comparisons among solutions.

Appendix A – stress testing

Microsoft Exchange Jetstress 2013

Stress Test Result Report Test Summary

Overall Test Result Pass Machine Name INDIA-DL360-G9 Test Description

Test Start Time 7/17/2015 3:09:35 AM Test End Time 7/18/2015 3:17:49 AM Collection Start Time 7/17/2015 3:17:43 AM Collection End Time 7/18/2015 3:17:39 AM Jetstress Version 15.00.0995.000 ESE Version 15.00.1076.009

Operating System Windows Server 2012 R2 Datacenter (6.2.9200.0)

Performance Log C:\Program Files\Exchange Jetstress\24HR_TEST\Bitlocker_24hr_test\Stress_2015_7_17_3_10_20.blg

Database Sizing and Throughput

Achieved Transactional I/O per Second 993.117 Target Transactional I/O per Second 640

Initial Database Size (bytes) 41943845306368 Final Database Size (bytes) 41972827947008 Database Files (Count) 22

Jetstress System Parameters

Thread Count 33 Minimum Database Cache 704.0 MB Maximum Database Cache 5632.0 MB Insert Operations 40% Delete Operations 20% Replace Operations 5% Read Operations 35% Lazy Commits 70% Run Background Database Maintenance True Number of Copies per Database 4 Database Configuration

Instance4788.1 Log path: C:\dblg1

Database: C:\dblg1\Jetstress001001.edb Instance4788.2 Log path: C:\dblg2

Database: C:\dblg2\Jetstress002001.edb Instance4788.3 Log path: C:\dblg3

Database: C:\dblg3\Jetstress003001.edb Instance4788.4 Log path: C:\dblg4

Database: C:\dblg4\Jetstress004001.edb Instance4788.5 Log path: C:\dblg5

Database: C:\dblg5\Jetstress005001.edb Instance4788.6 Log path: C:\dblg6

Database: C:\dblg6\Jetstress006001.edb Instance4788.7 Log path: C:\dblg7

(14)

Instance4788.9 Log path: C:\dblg9

Database: C:\dblg9\Jetstress009001.edb Instance4788.10 Log path: C:\dblg10

Database: C:\dblg10\Jetstress010001.edb Instance4788.11 Log path: C:\dblg11

Database: C:\dblg11\Jetstress011001.edb Instance4788.12 Log path: C:\dblg12

Database: C:\dblg12\Jetstress012001.edb Instance4788.13 Log path: C:\dblg13

Database: C:\dblg13\Jetstress013001.edb Instance4788.14 Log path: C:\dblg14

Database: C:\dblg14\Jetstress014001.edb Instance4788.15 Log path: C:\dblg15

Database: C:\dblg15\Jetstress015001.edb Instance4788.16 Log path: C:\dblg16

Database: C:\dblg16\Jetstress016001.edb Instance4788.17 Log path: C:\dblg17

Database: C:\dblg17\Jetstress017001.edb Instance4788.18 Log path: C:\dblg18

Database: C:\dblg18\Jetstress018001.edb Instance4788.19 Log path: C:\dblg19

Database: C:\dblg19\Jetstress019001.edb Instance4788.20 Log path: C:\dblg20

Database: C:\dblg20\Jetstress020001.edb Instance4788.21 Log path: C:\dblg21

Database: C:\dblg21\Jetstress021001.edb Instance4788.22 Log path: C:\dblg22

Database: C:\dblg22\Jetstress022001.edb

Transactional I/O Performance MSExchange Database ==> Instances I/O Database Reads Average Latency (msec) I/O Database Writes Average Latency (msec) I/O Database Reads/sec I/O Database Writes/sec I/O Database Reads Average Bytes I/O Database Writes Average Bytes I/O Log Reads Average Latency (msec) I/O Log Writes Average Latency (msec) I/O Log

Reads/sec I/O Log Writes/sec I/O Log Reads Average Bytes I/O Log Writes Average Bytes Instance4788.1 17.636 0.711 31.663 13.552 33216.317 36111.988 0.000 0.286 0.000 3.348 0.000 20659.190 Instance4788.2 17.424 0.713 31.637 13.584 33225.312 36110.495 0.000 0.225 0.000 3.357 0.000 20678.245 Instance4788.3 17.666 0.949 31.628 13.502 33249.287 36103.180 0.000 0.289 0.000 3.338 0.000 20653.714 Instance4788.4 17.050 0.929 31.650 13.545 33232.300 36126.303 0.000 0.222 0.000 3.341 0.000 20697.165 Instance4788.5 17.860 1.175 31.527 13.444 33228.692 36123.422 0.000 0.288 0.000 3.339 0.000 20764.607 Instance4788.6 17.023 1.171 31.626 13.523 33238.333 36132.324 0.000 0.223 0.000 3.343 0.000 20660.879 Instance4788.7 16.527 1.390 31.582 13.501 33227.031 36114.939 0.000 0.224 0.000 3.323 0.000 20819.073 Instance4788.8 17.186 1.392 31.645 13.588 33236.965 36116.735 0.000 0.225 0.000 3.363 0.000 20645.682 Instance4788.9 16.600 1.598 31.590 13.501 33230.618 36124.670 0.000 0.221 0.000 3.340 0.000 20739.733 Instance4788.10 17.264 1.608 31.648 13.531 33242.404 36108.462 0.000 0.222 0.000 3.347 0.000 20639.763 Instance4788.11 17.316 1.844 31.648 13.571 33233.520 36098.377 0.000 0.225 0.000 3.347 0.000 20599.210 Instance4788.12 17.916 1.840 31.631 13.550 33226.626 36125.313 0.000 0.225 0.000 3.357 0.000 20625.084 Instance4788.13 16.863 2.048 31.632 13.576 33230.671 36116.315 0.000 0.225 0.000 3.359 0.000 20672.309 Instance4788.14 17.570 2.047 31.610 13.519 33241.869 36126.349 0.000 0.225 0.000 3.351 0.000 20627.691 Instance4788.15 16.092 2.282 31.586 13.484 33248.985 36107.326 0.000 0.225 0.000 3.330 0.000 20758.875 Instance4788.16 18.786 2.247 31.593 13.522 33220.125 36145.707 0.000 0.285 0.000 3.353 0.000 20725.660 Instance4788.17 15.751 2.504 31.606 13.499 33246.881 36145.374 0.000 0.226 0.000 3.351 0.000 20675.448 Instance4788.18 18.472 2.472 31.559 13.534 33212.557 36115.797 0.000 0.223 0.000 3.370 0.000 20688.510 Instance4788.19 16.023 2.725 31.642 13.594 33238.761 36133.550 0.000 0.225 0.000 3.355 0.000 20699.887 Instance4788.20 18.817 2.679 31.585 13.549 33230.796 36117.828 0.000 0.222 0.000 3.361 0.000 20682.671 Instance4788.21 15.864 2.865 31.559 13.435 33219.250 36131.140 0.000 0.222 0.000 3.334 0.000 20733.042 Instance4788.22 18.629 2.879 31.646 13.516 33235.323 36106.717 0.000 0.224 0.000 3.347 0.000 20585.007 Background Database Maintenance I/O Performance

MSExchange Database

==> Instances Database Maintenance IO Reads/sec Database Maintenance IO Reads Average Bytes Instance4788.1 8.994 261929.502

Instance4788.2 9.148 261916.895 Instance4788.3 8.773 261898.094 Instance4788.4 9.029 261918.250

(15)

Instance4788.5 8.761 261899.865 Instance4788.6 9.051 261914.392 Instance4788.7 9.148 261920.289 Instance4788.8 9.146 261930.182 Instance4788.9 9.149 261894.067 Instance4788.10 9.149 261893.172 Instance4788.11 9.149 261904.489 Instance4788.12 9.148 261925.495 Instance4788.13 9.140 261900.059 Instance4788.14 9.148 261923.317 Instance4788.15 9.149 261892.544 Instance4788.16 8.834 261926.875 Instance4788.17 9.149 261892.493 Instance4788.18 8.752 261902.556 Instance4788.19 9.148 261916.374 Instance4788.20 8.739 261905.459 Instance4788.21 9.149 261915.900 Instance4788.22 8.746 261902.478 Log Replication I/O Performance

MSExchange Database ==>

Instances I/O Log Reads/sec I/O Log Reads Average Bytes Instance4788.1 0.876 114157.977 Instance4788.2 0.881 114847.291 Instance4788.3 0.873 113616.415 Instance4788.4 0.876 114063.884 Instance4788.5 0.878 114272.424 Instance4788.6 0.876 114007.202 Instance4788.7 0.878 114393.093 Instance4788.8 0.881 114789.771 Instance4788.9 0.879 114472.033 Instance4788.10 0.875 113901.442 Instance4788.11 0.875 114024.698 Instance4788.12 0.878 114348.637 Instance4788.13 0.881 114759.934 Instance4788.14 0.877 114106.816 Instance4788.15 0.876 114045.088 Instance4788.16 0.881 114679.995 Instance4788.17 0.878 114310.321 Instance4788.18 0.884 115193.023 Instance4788.19 0.881 114717.001 Instance4788.20 0.882 114841.475 Instance4788.21 0.877 114225.765 Instance4788.22 0.874 113819.426

Total I/O Performance MSExchange Database ==> Instances I/O Database Reads Average Latency (msec) I/O Database Writes Average Latency (msec) I/O Database Reads/sec I/O Database Writes/sec I/O Database Reads Average Bytes I/O Database Writes Average Bytes I/O Log Reads Average Latency (msec) I/O Log Writes Average Latency (msec) I/O Log

Reads/sec I/O Log Writes/sec I/O Log Reads Average Bytes I/O Log Writes Average Bytes Instance4788.1 17.636 0.711 40.657 13.552 83811.569 36111.988 0.394 0.286 0.876 3.348 114157.977 20659.190 Instance4788.2 17.424 0.713 40.785 13.584 84519.422 36110.495 0.223 0.225 0.881 3.357 114847.291 20678.245 Instance4788.3 17.666 0.949 40.401 13.502 82901.935 36103.180 0.395 0.289 0.873 3.338 113616.415 20653.714 Instance4788.4 17.050 0.929 40.679 13.545 83990.027 36126.303 0.217 0.222 0.876 3.341 114063.884 20697.165 Instance4788.5 17.860 1.175 40.288 13.444 82956.584 36123.422 0.405 0.288 0.878 3.339 114272.424 20764.607 Instance4788.6 17.023 1.171 40.677 13.523 84121.794 36132.324 0.219 0.223 0.876 3.343 114007.202 20660.879 Instance4788.7 16.527 1.390 40.730 13.501 84591.862 36114.939 0.248 0.224 0.878 3.323 114393.093 20819.073 Instance4788.8 17.186 1.392 40.791 13.588 84516.149 36116.735 0.225 0.225 0.881 3.363 114789.771 20645.682

(16)

Instance4788.12 17.916 1.840 40.779 13.550 84528.992 36125.313 0.223 0.225 0.878 3.357 114348.637 20625.084 Instance4788.13 16.863 2.048 40.772 13.576 84493.509 36116.315 0.250 0.225 0.881 3.359 114759.934 20672.309 Instance4788.14 17.570 2.047 40.758 13.519 84569.388 36126.349 0.225 0.225 0.877 3.351 114106.816 20627.691 Instance4788.15 16.092 2.282 40.736 13.484 84602.911 36107.326 0.218 0.225 0.876 3.330 114045.088 20758.875 Instance4788.16 18.786 2.247 40.427 13.522 83195.656 36145.707 0.390 0.285 0.881 3.353 114679.995 20725.660 Instance4788.17 15.751 2.504 40.755 13.499 84574.397 36145.374 0.233 0.226 0.878 3.351 114310.321 20675.448 Instance4788.18 18.472 2.472 40.311 13.534 82862.851 36115.797 0.211 0.223 0.884 3.370 115193.023 20688.510 Instance4788.19 16.023 2.725 40.791 13.594 84525.938 36133.550 0.234 0.225 0.881 3.355 114717.001 20699.887 Instance4788.20 18.817 2.679 40.324 13.549 82790.524 36117.828 0.228 0.222 0.882 3.361 114841.475 20682.671 Instance4788.21 15.864 2.865 40.707 13.435 84616.321 36131.140 0.243 0.222 0.877 3.334 114225.765 20733.042 Instance4788.22 18.629 2.879 40.393 13.516 82748.565 36106.717 0.215 0.224 0.874 3.347 113819.426 20585.007 Host System Performance

Counter Average Minimum Maximum % Processor Time 0.904 0.473 1.887 Available MBytes 121017.526 120928.000 121400.000 Free System Page Table Entries 16449764.119 16449027.000 16450225.000 Transition Pages RePurposed/sec 0.000 0.000 0.000 Pool Nonpaged Bytes 498827896.523 493182976.000 504635392.000 Pool Paged Bytes 208223088.375 191639552.000 225783808.000 Database Page Fault Stalls/sec 0.000 0.000 0.000

Test Log 7/17/2015 3:09:35 AM -- Preparing for testing ... 7/17/2015 3:09:57 AM -- Attaching databases ...

7/17/2015 3:09:57 AM -- Preparations for testing are complete. 7/17/2015 3:09:57 AM -- Starting transaction dispatch ..

7/17/2015 3:09:57 AM -- Database cache settings: (minimum: 704.0 MB, maximum: 5.5 GB) 7/17/2015 3:09:57 AM -- Database flush thresholds: (start: 56.3 MB, stop: 112.6 MB)

7/17/2015 3:10:20 AM -- Database read latency thresholds: (average: 20 msec/read, maximum: 200 msec/read). 7/17/2015 3:10:20 AM -- Log write latency thresholds: (average: 10 msec/write, maximum: 200 msec/write).

7/17/2015 3:10:24 AM -- Operation mix: Sessions 33, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%. 7/17/2015 3:10:24 AM -- Performance logging started (interval: 15000 ms).

7/17/2015 3:10:24 AM -- Attaining prerequisites:

7/17/2015 3:17:43 AM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 5318984000.0 (lower bound: 5315022000.0, upper bound: none) 7/18/2015 3:17:44 AM -- Performance logging has ended.

7/18/2015 3:17:44 AM -- JetInterop batch transaction stats: 94815, 94815, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814, 94814 and 94814.

7/18/2015 3:17:44 AM -- Dispatching transactions ends. 7/18/2015 3:17:44 AM -- Shutting down databases ...

7/18/2015 3:17:49 AM -- Instance4788.1 (complete), Instance4788.2 (complete), Instance4788.3 (complete), Instance4788.4 (complete), Instance4788.5 (complete), Instance4788.6 (complete), Instance4788.7 (complete), Instance4788.8 (complete), Instance4788.9 (complete), Instance4788.10 (complete), Instance4788.11 (complete), Instance4788.12 (complete), Instance4788.13 (complete), Instance4788.14 (complete), Instance4788.15 (complete), Instance4788.16 (complete), Instance4788.17 (complete), Instance4788.18 (complete), Instance4788.19 (complete), Instance4788.20 (complete), Instance4788.21 (complete) and Instance4788.22 (complete)

7/18/2015 3:17:49 AM -- C:\Program Files\Exchange Jetstress\24HR_TEST\Bitlocker_24hr_test\Stress_2015_7_17_3_10_20.blg has 5747 samples. 7/18/2015 3:17:49 AM -- Creating test report ...

7/18/2015 3:20:49 AM -- Instance4788.1 has 17.6 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.1 has 0.3 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.1 has 0.3 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.2 has 17.4 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.2 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.2 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.3 has 17.7 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.3 has 0.3 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.3 has 0.3 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.4 has 17.1 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.4 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.4 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.5 has 17.9 for I/O Database Reads Average Latency.

(17)

7/18/2015 3:20:49 AM -- Instance4788.5 has 0.3 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.6 has 17.0 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.6 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.6 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.7 has 16.5 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.7 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.7 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.8 has 17.2 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.8 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.8 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.9 has 16.6 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.9 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.9 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.10 has 17.3 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.10 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.10 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.11 has 17.3 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.11 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.11 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.12 has 17.9 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.12 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.12 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.13 has 16.9 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.13 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.13 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.14 has 17.6 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.14 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.14 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.15 has 16.1 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.15 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.15 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.16 has 18.8 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.16 has 0.3 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.16 has 0.3 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.17 has 15.8 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.17 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.17 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.18 has 18.5 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.18 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.18 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.19 has 16.0 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.19 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.19 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.20 has 18.8 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.20 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.20 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.21 has 15.9 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.21 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.21 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.22 has 18.6 for I/O Database Reads Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.22 has 0.2 for I/O Log Writes Average Latency. 7/18/2015 3:20:49 AM -- Instance4788.22 has 0.2 for I/O Log Reads Average Latency. 7/18/2015 3:20:49 AM -- Test has 0 Maximum Database Page Fault Stalls/sec.

7/18/2015 3:20:49 AM -- The test has 0 Database Page Fault Stalls/sec samples higher than 0.

(18)

Appendix B – performance testing

Microsoft Exchange Jetstress 2013 Performance Test Result Report Test Summary

Overall Test Result Pass

Machine Name INDIA-DL360-G9 Test Description

Test Start Time 7/19/2015 7:53:30 PM Test End Time 7/19/2015 10:01:49 PM Collection Start Time 7/19/2015 8:01:43 PM Collection End Time 7/19/2015 10:01:31 PM Jetstress Version 15.00.0995.000 ESE Version 15.00.1076.009

Operating System Windows Server 2012 R2 Datacenter (6.2.9200.0)

Performance Log C:\Program Files\Exchange Jetstress\2HR_TEST\Bitlocker_2hr_test\Performance_2015_7_19_19_54_17.blg

Database Sizing and Throughput

Achieved Transactional I/O per Second 1018.693 Target Transactional I/O per Second 640

Initial Database Size (bytes) 41972827947008 Final Database Size (bytes) 41975394861056 Database Files (Count) 22

Jetstress System Parameters

Thread Count 33 Minimum Database Cache 704.0 MB Maximum Database Cache 5632.0 MB Insert Operations 40% Delete Operations 20% Replace Operations 5% Read Operations 35% Lazy Commits 70% Run Background Database Maintenance True Number of Copies per Database 4

Database Configuration

Instance5276.1 Log path: C:\dblg1

Database: C:\dblg1\Jetstress001001.edb Instance5276.2 Log path: C:\dblg2

Database: C:\dblg2\Jetstress002001.edb Instance5276.3 Log path: C:\dblg3

Database: C:\dblg3\Jetstress003001.edb Instance5276.4 Log path: C:\dblg4

Database: C:\dblg4\Jetstress004001.edb Instance5276.5 Log path: C:\dblg5

Database: C:\dblg5\Jetstress005001.edb Instance5276.6 Log path: C:\dblg6

Database: C:\dblg6\Jetstress006001.edb Instance5276.7 Log path: C:\dblg7

Database: C:\dblg7\Jetstress007001.edb Instance5276.8 Log path: C:\dblg8

Database: C:\dblg8\Jetstress008001.edb Instance5276.9 Log path: C:\dblg9

Database: C:\dblg9\Jetstress009001.edb Instance5276.10 Log path: C:\dblg10

Database: C:\dblg10\Jetstress010001.edb Instance5276.11 Log path: C:\dblg11

Database: C:\dblg11\Jetstress011001.edb Instance5276.12 Log path: C:\dblg12

(19)

Instance5276.13 Log path: C:\dblg13

Database: C:\dblg13\Jetstress013001.edb Instance5276.14 Log path: C:\dblg14

Database: C:\dblg14\Jetstress014001.edb Instance5276.15 Log path: C:\dblg15

Database: C:\dblg15\Jetstress015001.edb Instance5276.16 Log path: C:\dblg16

Database: C:\dblg16\Jetstress016001.edb Instance5276.17 Log path: C:\dblg17

Database: C:\dblg17\Jetstress017001.edb Instance5276.18 Log path: C:\dblg18

Database: C:\dblg18\Jetstress018001.edb Instance5276.19 Log path: C:\dblg19

Database: C:\dblg19\Jetstress019001.edb Instance5276.20 Log path: C:\dblg20

Database: C:\dblg20\Jetstress020001.edb Instance5276.21 Log path: C:\dblg21

Database: C:\dblg21\Jetstress021001.edb Instance5276.22 Log path: C:\dblg22

Database: C:\dblg22\Jetstress022001.edb

Transactional I/O Performance MSExchange Database ==> Instances I/O Database Reads Average Latency (msec) I/O Database Writes Average Latency (msec) I/O Database Reads/sec I/O Database Writes/sec I/O Database Reads Average Bytes I/O Database Writes Average Bytes I/O Log Reads Average Latency (msec) I/O Log Writes Average Latency (msec) I/O Log

Reads/sec I/O Log Writes/sec I/O Log Reads Average Bytes I/O Log Writes Average Bytes Instance5276.1 16.773 0.728 32.284 13.842 33226.059 35686.588 0.000 0.286 0.000 3.371 0.000 20860.237 Instance5276.2 19.524 0.720 32.313 13.836 33234.404 35715.351 0.000 0.223 0.000 3.365 0.000 20712.814 Instance5276.3 18.552 0.968 32.483 13.982 33236.405 35647.932 0.000 0.285 0.000 3.346 0.000 20543.257 Instance5276.4 17.397 0.933 32.346 13.628 33188.544 35665.763 0.000 0.222 0.000 3.251 0.000 20685.157 Instance5276.5 18.991 1.198 32.536 14.208 33241.560 35672.844 0.000 0.284 0.000 3.386 0.000 20714.678 Instance5276.6 17.801 1.191 32.422 14.001 33187.628 35732.529 0.000 0.220 0.000 3.349 0.000 20832.876 Instance5276.7 17.186 1.405 32.440 13.787 33159.987 35678.879 0.000 0.219 0.000 3.349 0.000 20345.849 Instance5276.8 17.851 1.405 32.391 13.753 33208.158 35703.849 0.000 0.222 0.000 3.317 0.000 20629.904 Instance5276.9 17.308 1.643 32.494 13.991 33224.200 35659.474 0.000 0.220 0.000 3.365 0.000 20602.219 Instance5276.10 17.991 1.623 32.393 13.725 33234.516 35742.627 0.000 0.222 0.000 3.297 0.000 20591.544 Instance5276.11 16.868 1.869 32.371 13.659 33194.238 35690.886 0.000 0.222 0.000 3.276 0.000 20564.374 Instance5276.12 19.662 1.839 32.182 13.561 33241.970 35736.034 0.000 0.223 0.000 3.253 0.000 20928.735 Instance5276.13 17.604 2.070 32.389 13.880 33207.861 35661.580 0.000 0.223 0.000 3.315 0.000 20857.372 Instance5276.14 18.245 2.084 32.435 13.904 33195.755 35696.855 0.000 0.223 0.000 3.320 0.000 20874.148 Instance5276.15 16.758 2.323 32.537 14.094 33207.967 35658.369 0.000 0.223 0.000 3.390 0.000 20669.068 Instance5276.16 19.628 2.296 32.615 13.984 33218.020 35638.283 0.000 0.283 0.000 3.333 0.000 20477.168 Instance5276.17 16.402 2.544 32.539 13.881 33210.156 35700.753 0.000 0.221 0.000 3.354 0.000 20483.268 Instance5276.18 19.245 2.523 32.414 13.880 33219.920 35635.390 0.000 0.221 0.000 3.329 0.000 20694.792 Instance5276.19 16.689 2.750 32.418 13.830 33233.124 35615.138 0.000 0.222 0.000 3.325 0.000 20732.608 Instance5276.20 19.575 2.732 32.490 14.008 33190.634 35648.835 0.000 0.219 0.000 3.345 0.000 20507.027 Instance5276.21 17.016 2.896 32.447 13.928 33249.254 35676.932 0.000 0.221 0.000 3.373 0.000 20796.720 Instance5276.22 18.596 2.926 32.495 13.895 33225.260 35636.706 0.000 0.222 0.000 3.336 0.000 20654.767 Background Database Maintenance I/O Performance

MSExchange Database ==>

Instances Database Maintenance IO Reads/sec Database Maintenance IO Reads Average Bytes Instance5276.1 9.149 261906.075 Instance5276.2 8.716 261875.982 Instance5276.3 8.792 261919.554 Instance5276.4 9.149 261857.826 Instance5276.5 8.801 261935.257 Instance5276.6 9.150 261848.932

(20)

Instance5276.9 9.146 261994.710 Instance5276.10 9.147 261928.843 Instance5276.11 9.149 261941.287 Instance5276.12 8.710 261945.854 Instance5276.13 9.146 261985.900 Instance5276.14 9.142 261850.303 Instance5276.15 9.150 261877.135 Instance5276.16 8.815 261908.599 Instance5276.17 9.148 261937.642 Instance5276.18 8.733 261957.693 Instance5276.19 9.149 261851.160 Instance5276.20 8.711 261877.534 Instance5276.21 9.147 261961.912 Instance5276.22 8.976 261928.482 Log Replication I/O Performance

MSExchange Database ==> Instances I/O Log Reads/sec I/O Log Reads Average Bytes Instance5276.1 0.884 115056.880 Instance5276.2 0.880 114567.276 Instance5276.3 0.873 113588.068 Instance5276.4 0.850 110677.568 Instance5276.5 0.888 115546.483 Instance5276.6 0.884 115056.880 Instance5276.7 0.861 112119.257 Instance5276.8 0.869 113626.948 Instance5276.9 0.884 115056.880 Instance5276.10 0.858 111629.653 Instance5276.11 0.858 111629.653 Instance5276.12 0.865 112608.861 Instance5276.13 0.877 114077.672 Instance5276.14 0.880 114567.276 Instance5276.15 0.888 115546.483 Instance5276.16 0.865 112635.983 Instance5276.17 0.873 113588.068 Instance5276.18 0.873 113588.068 Instance5276.19 0.873 113588.068 Instance5276.20 0.873 113588.068 Instance5276.21 0.884 115585.363 Instance5276.22 0.873 113588.068 Total I/O Performance

MSExchange Database ==> Instances I/O Database Reads Average Latency (msec) I/O Database Writes Average Latency (msec) I/O Database Reads/sec I/O Database Writes/sec I/O Database Reads Average Bytes I/O Database Writes Average Bytes I/O Log Reads Average Latency (msec) I/O Log Writes Average Latency (msec) I/O Log Reads/sec I/O Log Writes/sec I/O Log Reads Average Bytes I/O Log Writes Average Bytes Instance5276.1 16.773 0.728 41.434 13.842 83723.090 35686.588 0.445 0.286 0.884 3.371 115056.880 20860.237 Instance5276.2 19.524 0.720 41.029 13.836 81805.490 35715.351 0.266 0.223 0.880 3.365 114567.276 20712.814 Instance5276.3 18.552 0.968 41.275 13.982 81947.315 35647.932 0.471 0.285 0.873 3.346 113588.068 20543.257 Instance5276.4 17.397 0.933 41.495 13.628 83606.051 35665.763 0.213 0.222 0.850 3.251 110677.568 20685.157 Instance5276.5 18.991 1.198 41.337 14.208 81934.583 35672.844 0.455 0.284 0.888 3.386 115546.483 20714.678 Instance5276.6 17.801 1.191 41.572 14.001 83517.076 35732.529 0.246 0.220 0.884 3.349 115056.880 20832.876 Instance5276.7 17.186 1.405 41.590 13.787 83480.383 35678.879 0.215 0.219 0.861 3.349 112119.257 20345.849 Instance5276.8 17.851 1.405 41.542 13.753 83571.695 35703.849 0.207 0.222 0.869 3.317 113626.948 20629.904 Instance5276.9 17.308 1.643 41.640 13.991 83474.624 35659.474 0.206 0.220 0.884 3.365 115056.880 20602.219

(21)

Instance5276.11 16.868 1.869 41.519 13.659 83598.150 35690.886 0.280 0.222 0.858 3.276 111629.653 20564.374 Instance5276.12 19.662 1.839 40.892 13.561 81954.507 35736.034 0.256 0.223 0.865 3.253 112608.861 20928.735 Instance5276.13 17.604 2.070 41.536 13.880 83585.761 35661.580 0.204 0.223 0.877 3.315 114077.672 20857.372 Instance5276.14 18.245 2.084 41.578 13.904 83473.902 35696.855 0.223 0.223 0.880 3.320 114567.276 20874.148 Instance5276.15 16.758 2.323 41.688 14.094 83399.062 35658.369 0.249 0.223 0.888 3.390 115546.483 20669.068 Instance5276.16 19.628 2.296 41.430 13.984 81875.462 35638.283 0.431 0.283 0.865 3.333 112635.983 20477.168 Instance5276.17 16.402 2.544 41.687 13.881 83404.883 35700.753 0.294 0.221 0.873 3.354 113588.068 20483.268 Instance5276.18 19.245 2.523 41.147 13.880 81767.125 35635.390 0.272 0.221 0.873 3.329 113588.068 20694.792 Instance5276.19 16.689 2.750 41.567 13.830 83551.875 35615.138 0.317 0.222 0.873 3.325 113588.068 20732.608 Instance5276.20 19.575 2.732 41.201 14.008 81542.930 35648.835 0.275 0.219 0.873 3.345 113588.068 20507.027 Instance5276.21 17.016 2.896 41.594 13.928 83546.927 35676.932 0.287 0.221 0.884 3.373 115585.363 20796.720 Instance5276.22 18.596 2.926 41.470 13.895 82724.140 35636.706 0.225 0.222 0.873 3.336 113588.068 20654.767

Host System Performance

Counter Average Minimum Maximum % Processor Time 0.934 0.443 1.727 Available MBytes 120889.685 120841.000 121285.000 Free System Page Table Entries 16449523.672 16449007.000 16449954.000 Transition Pages RePurposed/sec 0.000 0.000 0.000 Pool Nonpaged Bytes 516321150.924 515551232.000 517091328.000 Pool Paged Bytes 244748916.168 243228672.000 246247424.000 Database Page Fault Stalls/sec 0.000 0.000 0.000

Test Log 7/19/2015 7:53:30 PM -- Preparing for testing ... 7/19/2015 7:53:53 PM -- Attaching databases ...

7/19/2015 7:53:53 PM -- Preparations for testing are complete. 7/19/2015 7:53:53 PM -- Starting transaction dispatch ..

7/19/2015 7:53:53 PM -- Database cache settings: (minimum: 704.0 MB, maximum: 5.5 GB) 7/19/2015 7:53:53 PM -- Database flush thresholds: (start: 56.3 MB, stop: 112.6 MB)

7/19/2015 7:54:17 PM -- Database read latency thresholds: (average: 20 msec/read, maximum: 100 msec/read). 7/19/2015 7:54:17 PM -- Log write latency thresholds: (average: 10 msec/write, maximum: 100 msec/write).

7/19/2015 7:54:20 PM -- Operation mix: Sessions 33, Inserts 40%, Deletes 20%, Replaces 5%, Reads 35%, Lazy Commits 70%. 7/19/2015 7:54:20 PM -- Performance logging started (interval: 15000 ms).

7/19/2015 7:54:20 PM -- Attaining prerequisites:

7/19/2015 8:01:43 PM -- \MSExchange Database(JetstressWin)\Database Cache Size, Last: 5338505000.0 (lower bound: 5315022000.0, upper bound: none) 7/19/2015 10:01:44 PM -- Performance logging has ended.

7/19/2015 10:01:44 PM -- JetInterop batch transaction stats: 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8346, 8345, 8345 and 8345.

7/19/2015 10:01:44 PM -- Dispatching transactions ends. 7/19/2015 10:01:44 PM -- Shutting down databases ...

7/19/2015 10:01:49 PM -- Instance5276.1 (complete), Instance5276.2 (complete), Instance5276.3 (complete), Instance5276.4 (complete), Instance5276.5 (complete), Instance5276.6 (complete), Instance5276.7 (complete), Instance5276.8 (complete), Instance5276.9 (complete), Instance5276.10 (complete), Instance5276.11 (complete), Instance5276.12 (complete), Instance5276.13 (complete), Instance5276.14 (complete), Instance5276.15 (complete), Instance5276.16 (complete), Instance5276.17 (complete), Instance5276.18 (complete), Instance5276.19 (complete), Instance5276.20 (complete), Instance5276.21 (complete) and Instance5276.22 (complete)

7/19/2015 10:01:49 PM -- C:\Program Files\Exchange Jetstress\2HR_TEST\Bitlocker_2hr_test\Performance_2015_7_19_19_54_17.blg has 505 samples. 7/19/2015 10:01:49 PM -- Creating test report ...

7/19/2015 10:01:58 PM -- Instance5276.1 has 16.8 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.1 has 0.3 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.1 has 0.3 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.2 has 19.5 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.2 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.2 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.3 has 18.6 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.3 has 0.3 for I/O Log Writes Average Latency.

(22)

7/19/2015 10:01:58 PM -- Instance5276.4 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.5 has 19.0 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.5 has 0.3 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.5 has 0.3 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.6 has 17.8 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.6 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.6 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.7 has 17.2 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.7 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.7 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.8 has 17.9 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.8 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.8 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.9 has 17.3 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.9 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.9 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.10 has 18.0 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.10 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.10 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.11 has 16.9 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.11 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.11 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.12 has 19.7 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.12 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.12 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.13 has 17.6 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.13 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.13 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.14 has 18.2 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.14 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.14 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.15 has 16.8 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.15 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.15 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.16 has 19.6 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.16 has 0.3 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.16 has 0.3 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.17 has 16.4 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.17 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.17 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.18 has 19.2 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.18 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.18 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.19 has 16.7 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.19 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.19 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.20 has 19.6 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.20 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.20 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.21 has 17.0 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.21 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.21 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.22 has 18.6 for I/O Database Reads Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.22 has 0.2 for I/O Log Writes Average Latency. 7/19/2015 10:01:58 PM -- Instance5276.22 has 0.2 for I/O Log Reads Average Latency. 7/19/2015 10:01:58 PM -- Test has 0 Maximum Database Page Fault Stalls/sec.

(23)

7/19/2015 10:01:58 PM -- C:\Program Files\Exchange Jetstress\2HR_TEST\Bitlocker_2hr_test\Performance_2015_7_19_19_54_17.xml has 475 samples queried.

Appendix C – streaming backup testing

Microsoft Exchange Jetstress 2013 Database backup Test Result Report Database Backup Statistics - All

Database Instance Database Size (MBytes) Elapsed Backup Time MBytes Transferred/sec

Instance3628.1 1819576.03 06:21:36 79.47 Instance3628.2 1819592.03 03:54:20 129.41 Instance3628.3 1819568.03 10:08:31 49.84 Instance3628.4 1819568.03 12:23:01 40.81 Instance3628.5 1819576.03 12:28:01 40.54 Instance3628.6 1819576.03 11:43:32 43.11 Instance3628.7 1819576.03 15:19:05 33.00 Instance3628.8 1819576.03 13:55:17 36.31 Instance3628.9 1819584.03 14:21:51 35.19 Instance3628.10 1819568.03 13:40:36 36.96 Instance3628.11 1819568.03 15:02:05 33.62 Instance3628.12 1819576.03 13:35:08 37.20 Instance3628.13 1819576.03 11:08:19 45.38 Instance3628.14 1819584.03 10:08:18 49.85 Instance3628.15 1819568.03 15:09:25 33.35 Instance3628.16 1819584.03 16:55:32 29.86 Instance3628.17 1819576.03 06:49:47 74.00 Instance3628.18 1819592.03 05:49:14 86.83 Instance3628.19 1819576.03 16:30:12 30.63 Instance3628.20 1819576.03 18:10:35 27.81 Instance3628.21 1819576.03 10:02:27 50.34 Instance3628.22 1819568.03 11:44:00 43.08 Avg 48.48 Sum 1066.57

Jetstress System Parameters Thread Count 33 Minimum Database Cache 704.0 MB Maximum Database Cache 5632.0 MB Insert Operations 40% Delete Operations 20% Replace Operations 5% Read Operations 35% Lazy Commits 70% Database Configuration

Instance3628.1 Log path: C:\dblg1

Database: C:\dblg1\Jetstress001001.edb Instance3628.2 Log path: C:\dblg2

Database: C:\dblg2\Jetstress002001.edb Instance3628.3 Log path: C:\dblg3

Database: C:\dblg3\Jetstress003001.edb Instance3628.4 Log path: C:\dblg4

Database: C:\dblg4\Jetstress004001.edb Instance3628.5 Log path: C:\dblg5

Database: C:\dblg5\Jetstress005001.edb Instance3628.6 Log path: C:\dblg6

Database: C:\dblg6\Jetstress006001.edb Instance3628.7 Log path: C:\dblg7

References

Related documents

This solution brief provides information on a Microsoft® Exchange Server 2010 (Exchange) storage solution based on HP ProLiant DL380p Gen8 servers utilizing internal server

- HP Virtual Connect for cClass BladeSystem Setup and - This 3day course provides instruction on HP BladeSystem - HP ProLiant BL460c Gen9 Server Blade HewlettPackard - HP

The paver type must comply with the requirements for application 2, residential driveways, light traffic (NZS 3116, Clause 302 Paver Selection). The pavers must be laid in

Standard Features HP ProLiant Server Support HP ProLiant BL260c G5 HP ProLiant BL2x220c G5 HP ProLiant BL2x220c G6 HP ProLiant BL280c G6 HP ProLiant BL460c HP ProLiant BL460c G5

The D6000 can also be directly connected to selected ProLiant Gen8 servers using the Smart Array P421 and P822 Controllers using a standard 2M SAS cable; Single and Dual Domain

6 Gb/s front-end SAS connectivity using 6 Gb/s SAS BL switches in the interconnect bays, and HP Smart Array P721m controllers in each BladeSystem server ( NOTE: D6000 support

The HP StorageWorks 60 Modular Smart Array enclosure is a 2U Serial Attach SCSI (SAS) disk drive storage enclosure supporting 3.5&#34; SAS or Serial ATA (SATA) drives.. This

For ProLiant servers and storage systems, this service covers HP-branded hardware options qualified for the server, purchased at the same time or afterward, internal to the