Manager 10g Grid Control Handbook

(1)

ORACLE

Oracle

Press

Oracle

_Enterprise

Manager 10g

Grid

Control

_Handbook

Werner

De

_Gruyter

Matthew

Hart

Daniel

_Nguyen

Mc

Grauu

Hill

TECHNISCHE

INFORMATIQNSBIBUOTHEK

UNIVERSITATSBIBUOTHEK

HANNOVER

New York

Chicago

San Francisco

Lisbon

London

Madrid

Mexico

_City

Milan

New Delhi San

Juan

Seoul

_{Singapore Sydney}

Toronto

TIB/UB Hannover 133 380 939

(2)

Acknowledgments

Xlx

Introduction XXMI

PART 1

Installing

and

_Deploying

Grid

Control

1 What Is Grid Control?

Enterprise Manager Concepts

3

Enterprise Management

Software 4 The VariousManagementTools 6

DBControl 6

ASControl 7

GridControl

_Concepts

7

GridControlvs.

Enterprise

Manager 7

Acronym

vs. Idiom 8

Metricsvs.

Targets

3

Administratorvs.User 9

Grid Control

_Components

10

Grid Control Console 11

TheGrid Control Agent 11

The

_Management

Server(OMS) 13

TheGrid Control

_Repository

13

The Flow of Information 14

Agent

and OMS Communication 14

OMS-to-Repository

Communication 16

User(Web-Client)toOMS Communication 19

TCP PortsUsed

_by

_{Enterprise Manager}

20

The Different

_Types

of

_Targets

21

Summary

22

(3)

X Oracle

_{Enterprise Manager 10g}

Grid

Control Handbook

23

2

_Preparing

and

_Installing

the Environment

24

Preparing

for the Install

25

Phased Rollout

2&

Topology

ofthe

_Management

Framework

Topology

of the

Managed

Targets

^

Preparing

forFutureGrowth

NamingConventionsfor

Physical

Hosts 28

Using

a Different Hostnamefor the Install 29

Preinstallation Checks for the Installer and the Database 30

OT

Modifying

the

_Response

Files

Hardware Requirements 33

Installing

GridControl 34

Enterprise Manager

1Og

Grid Control

_Using

a New Database 34

Enterprise

Manager

1Og Grid Control

Using

an

Existing

Database 35

Additional

_Management

Server 36

Additional

_{Management Agent}

37

Postconfiguration

onthe RepositoryServer 37

Postconfiguration

onthe OMS Server 38

Loader_{Setup and Shared File}

_System

Loader 38 Out-Of-BandNotifications 39

Log

and Trace Filesand Rotation 41

Oracle HTTPServer

_{Configuration}

45

SMTP Server 45

Securing

the Environment 45

Setting Up

Advanced

_Networking

_Option

_(ANO) 46

Configuring

HTTPServertoUse SSLforConsole Ul 46 EMKEY

_Operations

48

Locking

and

_Unlocking

theOMSServer 48

Securing

the Databaseand the

_Application

Server 50

Using

Firewalls ₅₁

Postconfiguration

for

_Patching

and

_Provisioning

54

Software

_Library

₅₄

Proxy

Server

_Settings

₅₅

My Oracle

Support Setup

56

EMCLI ₅₇

Installing

EMCLI ₅₈

Setting

Up EMCLI ₅₉ Files Used

by

EMCLI ₆₀

Synchronizing

EMCLI Software ₆₁

CommandOverview ₆₁

Setting Up

aTestEnvironment ₆₂

Installationand

_Deployment

Best Practices ₆₂

CommonMistakes ₆₂

Best Practices ₆₃

(4)

Contents

xi

3 Grid Control

_{and Maximum}

_Availability

81

Why High

Availability?

₈₂

HA:

_Eliminating

the

_Single

Point ofFailure ₈₄

Making

GridControl Disaster

_{Recovery-Ready}

₉₅

Backup

and

_Recovery

99

Strengthening

the Extended Infrastructure ₁₀₆

Summary—EM

HA in the RealWorld ₁₀₇ 4

_{Deploying Agents}

_{into the}

_Enterprise

109

Preparing

the ServerMachine ₁₁₀

Hostnames and Hostname Resolution ₁₁₀ User Accounts ₁₁₀ Time Zone and OSClock ₁₁₁

Available Disk_Space ₁₁₂

Different

_Types

of Install ₁₁₃

Standard OUI Interactive Install ₁₁₃ Silent OUI Install ₁₁₃

Agent

Download ₁₁₄

Agent

Push _{(aka Ul-Based Push)} ₁₁₄

Shared

_Agent

(aka

_{State-Deployed}

_Agenl/NFS

_Agent) ₁₁₅

Using

the

_Agent

_{Download Install}

116

The DefaultInstalled Software ₁₁₆

Customizing

the Agent Download_Script ₁₁₉

Using

astaticports.ini File ₁₂₂

Command-Line Optionsfor the_Agent Download

_Script

₁₂₂

Using

the

_Agent

Push Install ₁₂₄

ThePrerequisitesfor Push Installs ₁₂₄

Installation

_Steps

₁₂₄

Using the Shared

Agent

Install ₁₂₅

The

_{Prerequisites}

for Shared

_Agent

Installs ₁₂₅ Installation _Steps ₁₂₅

Postinstallation and

_{Configuration Steps}

₁₂₆

Enabling

OOB

_Messages

₁₂₇

Configuring

the

_{Monitoring Space}

₁₂₈

Logging

and

_Tracing

₁₂₈

Timeout

_{Configuration}

Incoming

Connections ₁₃₅ Threads and Processes ₁₃₆

(5)

••

XII Oracle

_{Enterprise Manager}

_10g

Grid Control Handbook

Watchdog

Monitoring 137 Oracle Database Connections 138

PropertyTimeouts 139

OMS Server

_Request

Timeouts 140

Summary

141

5 Grid Control:

_Sizing

theEnvironmentand Other Best Practices 143

JustHow

_Many

OMS Servers Do I Need? 144

Adding

anAdditional OMS Server 145

XML Loaders 147

Metric Data

_Rollup

153

Agent

Heartbeating

154

Job Handling

155

Notification

Handling

157

How

Big

Doesthe Database NeedtoBe? 158

Installing

and

_Deploying

Grid Control:Common Mistakes 161

Summary:

Best Practices 162 6

_Maintaining

and

_Managing

the EMGC

Infrastructure

163

MonitortheMonitor 164

TheOMSand

_{Repository Target}

164

WhatlsMTM? 166

Health-Checks 176

Cleaning

UpFiles 183

Defining

Timeouts 185

Managing theCoreInfrastructure 187

XML Loaders 187

GridControl Jobs 193

Notification

_Dispatching

198

Agent Heartbeating

201

Repository

Metric Calculations 203 Data

_Rollup

206 Data

_Purging

207

Keeping Up

with

_Changes

in the Data Center 211 Infrastructure

_Changes

211 Server MachineOperations 217

Patching

and

_Upgrading

theInfrastructure 224

Keeping

Track of

_Changes

231

AdministratorManagement 234

Summary

238

The CommonMistakes 238 Best Practices ₂₃₉

(6)

Contents XI

ii

PART II

Grid

_{Control Common Tasks and Functions}

7

_Principles

_of

_{Target Monitoring}

243

Monitoring

and

_Managing

aTarget

244

Setting Up

aTargetforAdministration and

_Monitoring

₂₄₅ Credential Information ₂₄₅

Monitoring

Properties ₂₄₉

Target Properties

₂₅₀

Navigatingthe Ul Consolefor

_{Target Monitoring}

₂₅₁

Locating

theTargetin the Console ₂₅₁

Finding

theAgent fora

Target

₂₅₂

TargetState and

_Availability

₂₅₃

Availability

States ₂₅₃

Availability

Grouping ₂₅₄

Summary

₂₅₅

8

_Managing

EM Users and Audits ₂₅₇

Defining

Administrators ₂₅₈

GridControl

_Privileges

₂₅₉

Job

Privileges

₂₅₉

₂₆₃

CreatingNew Roles ₂₆₃

Deleting

aRole ₂₆₃

Set _{Up Centralized} Authentication ₂₆₃

User

_Management

₂₆₆

Auditing

₂₆₆

Enabling

Auditing

266

Audit

_Reporting

₂₆₇

Purgingthe Audit Data 268

Summary

269

9 Metrics and

Notifications

271

Maximizing

the Power ofGrid Control Metrics 272

Reviewing

Metrics intheGC Environment ₂₇₂ MetricThresholds 273

Setting

theMetric Thresholds ₂₇₇

(7)

XIV

_Oracle

_{Enterprise Manager}

_10g

Grid Control Handbook

Managing Alerts 282

User-DefinedMetrics 287

Policies in EM Grid Control 289

Suspend Monitoring with Blackouts 291

Corrective Actions 293

Getting

the Most (orShouldWeSay Least)Out ofNotifications 293

Understanding

and Utilizing Methods

Effectively

294 Notification Rules 301 What AreSchedules? 303

Summary 304

10

Jobs

and Task Automation 305

Jobs 306

Anatomy

ofaJob 306

Managed Targets

to ExecuteJobs

Against

307

Further Classification ofJobs, Job Executions, andJob Runs 310 TheJob

System

310

Using

and

_Defining

Jobs 316

Job

Library

320 A Noteon

Enterprise

Manager Command-LineInterface andJobs 321

JobsandGroups 322

Analyzing,

Diagnosing, and

_{Troubleshooting Jobs}

324

Searching

Jobs 327

Maintaining

the_{Job System} 328

Summary 331

11

_{Reporting and Dashboards}

333

Reports 334

Common Uses for

_Reports

334

How toGetaReport 335

The Anatomyofa

Report

336

CreatingaNew Report 337

Dashboards 343

Customizing

Dashboards 344

WritingYour Own Dashboards 345

AnEnd-to-EndView of

_{Deployed Applications: Topology}

Viewer 346

Summary 347

PART ill

Enterprise

Manager

Power User's

Guide

12 Host

_Management

351

Hosts ₃₅₂

Provisioning

Hosts ₃₅₂

(8)

Contents XV

WhatDo You

_Really

Know about Your Hosts? 356

Oracle,What Are

_{My Agents Monitoring?}

₃₆₆

Creating

and

_Enforcing

Standards forHost

_Management

₃₆₉

Summary 379

13

_{Systems, Services,}

and

_Croups

₃₈₁

Systems 382

Building

a

System

₃₈₃

BestPracticesforUsing Systems 389

Services ₃₈₉

Web

_Application

₃₉₀

Building

aService ₃₉₀

Best Practicesfor UsingServices 394

Groups 395

Redundancy

Groups 396

Building

aGroup 396

BestPractices for Using

Groups

396

Defining

the EM_{System and}Service 400

Summary 409

14

_{Database Management}

in Grid Control 411

Enterprise Manager DatabaseControl 412

Configuring

Databasesin Grid Control 416

Database Discovery 416

PreferredCredentials for Database Activities 416

Database Administration in Grid Control 419

Database

_Monitoring

419

Database Administration 419 The Database Home Page 420 Performance 423

Availability

429

Backupand Recovery 430

Database

_Backups

fromGrid Control 432

Performing

RecoveryinGrid Control 436

Backup Management

and Reporting 438

TheServer 438

Schema

_Monitoring

and

_Management

443

DataMovement 445 Grid Control and Managing the ProductionDatabase

_Ecosystem

446

Databases andConsolidated

_Targets

(Groups,

Systems,

Services) 447

Database Change

Management

449

Summary 451

15 RAC and Cluster

_Management

453

Managing

RAC Clusters in Grid Control 454

(9)

•

XVI

_Oracle

_{Enterprise Manager}

_lOg

_{Grid Control}

_Handbook

Navigating Through

the Cluster Resources 463

Cluster-Managed

Database Services 466

Listener Information 466 ASM

_Monitoring

467

Summary

472

16 Data Guard

_Management

473

The

_{High Availability}

Console 474 Data Guard Overview 475

Creating

a

Physical Standby

Database in Grid Control 476

Data Guard_{Management Operations:}An Overview 477

Data Guard

_Monitoring

andMetrics 485

Configuring

a

Physical

Standby

fora RAC Database 486

Summary 488

17

_Application

_Monitoring

489

The

_Complexity

of_{Online Operations} ₄₉₀

Servicing Applications

492

Justifying

Service

Availability

492

Working

with Beacons 492

Whereto Put Beacons 493

SSL

_Monitoring

with Beacons 496

Exposing Monitoring Configuration

Tasks 505

Setting Up

Watch Lists 513 Root Cause

_Analysis

515

Duality Explained

516

Summary

Target

Discovery Debugging

535

Target Monitoring Debugging

536

Agent

Metric

_Debugging

₅₃₇

Agent Upload

Debugging

542

Agent Job

Debugging

545

TimeZone

_Debugging

₅₄₆

(10)

Contents XVII

Diagnosing

OMSProblems • ••-• 549

Java

Debugging

549 OMS

_Target

_Debugging

• 551

XML Loader

Debugging

554 EMJob

Debugging

556

Repository

Validation 557

Using

EMDIAG ₅₅₉ WhatIs EMDIAG? 559

Installing

EMDIAG 561

Using

Verify

Commands 563

Dumping Reports 566

Reportingon EMDIAG

Findings

568

Summary

569

PARTIV

Appendixes

A

_{Configuration}

Files 573

Agent

574

OMS:

_Management

Server 576

B

_{Configuration}

Parameters 577

Agent:

Propertiesinthe

_{emd.properties}

File 578

OMS:_Propertiesin the_{emoms.properties}File 586

C

_{Log Files}

597

Agent

598

Agent Log andTraceFiles 598

Oracle NetLog Files 599

OMS:ManagementServer 599

Oracle HTTP ServerLogFiles 599

ASControl Agent

Log

Files 600

OC4J

Applications

600

OMS Server(OC4J_EM)

Log

and Trace Files 601

Agent

Push Installs_{(OC4J_EMPROV) Log} and Trace Files 601

DCM

_Log

Files 602

OPMN

_Log

Files 603 Webcache Server_LogFiles 603

Install,

Configure,

and Patch 603

Installer 604

Configuration

Tools 604

OPatch 604

D Environment Variables 605

Environment VariablesUsed

by

EMCTLonthe Agent 606

(11)

XVlii

Oracle

_{Enterprise Manager lOgGrid}

Control Handbook

E Standard

_Repository

F Command-Line Tool

_Options

623

EMCTL:

_Agent-Side

Commands 624 EMCTL: OMS Server-Side Commands 627 EMCTL: AS Control Commands 630

G

_Acronyms

631

Glossary

639