Installing and Configuring
Server Groups
If you have comments or suggestions about this documentation, contact Information Design and Development by email at
doc_feedback@bmc.com.
about the company, its products, corporate offices, special events, and career opportunities.
United States and Canada
Address BMC SOFTWARE INC
2101 CITYWEST BLVD HOUSTON TX 77042-2827 USA Telephone 713 918 8800 or 800 841 2031 Fax 713 918 8000
Outside United States and Canada
Telephone (01) 713 918 8800 Fax (01) 713 918 8000
© Copyright 2011 BMC Software, Inc.
BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners.
IBM, Informix, DB2, and AIX are registered trademarks of International Business Machines Corporation in the United States, other countries, or both.
Linux is the registered trademark of Linus Torvalds.
UNIX is the registered trademark of The Open Group in the US and other countries.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. BMC Software considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable End User License Agreement for the product and the proprietary and restricted rights notices included in this documentation.
Restricted rights legend
U.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to
restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC Software, Inc., 2101 CityWest Blvd., Houston, TX 77042-2827, USA. Any contract notices should be sent to this address.
Support by telephone or email. To expedite your inquiry, please see “Before Contacting BMC Software.”
Support website
You can obtain technical support from BMC Software 24 hours a day, 7 days a week at
http://www.bmc.com/support. From this website, you can:
■ Read overviews about support services and programs that BMC Software offers.
■ Find the most current information about BMC Software products. ■ Search a database for problems similar to yours and possible solutions. ■ Order or download product documentation.
■ Report a problem or ask a question.
■ Subscribe to receive email notices when new product versions are released.
■ Find worldwide BMC Software support center locations and contact information, including email addresses, fax numbers, and telephone numbers.
Support by telephone or email
In the United States and Canada, if you need technical support and do not have access to the Web, call 800 537 1813 or send an email message to customer_support@bmc.com. (In the Subject line, enter
SupID:<yourSupportContractID>, such as SupID:12345.) Outside the United States and Canada, contact your local support center for assistance.
Before contacting BMC Software
Have the following information available so that Customer Support can begin working on your issue immediately:
■ Product information
— Product name
— Product version (release number)
— License number and password (trial or permanent)
■ Operating system and environment information
— Machine type
— Operating system type, version, and service pack — System hardware configuration
— Serial numbers
— Related software (database, application, and communication) including type, version, and service pack or maintenance level
■ Sequence of events leading to the problem
■ Commands and options that you used
■ Messages received (and the time and date that you received them)
— Product error messages
— Messages from the operating system, such as file system full
License key and password information
If you have a question about your license key or password, contact Customer Support through one of the following methods:
■ E-mail customer_support@bmc.com. (In the Subject line, enter SupID:<yourSupportContractID>,
such as SupID:12345.)
■ In the United States and Canada, call 800 537 1813. Outside the United States and Canada, contact your local support center for assistance.
Chapter 1 Installing and configuring server groups 7
Overview . . . 7
Audience . . . 7
How this document relates to the installation documentation. . . 8
Server Groups Overview . . . 8
Implementing a server group . . . 9
Server group planning. . . 17
Example configurations . . . 17
BMC Software Installation Planning - Products and Components . . . 20
Installing a server group (new installation) . . . 20
Before starting the installation. . . 20
Server group installation overview . . . 22
Installing the first server and the database . . . 23
Configure the first server to be a server group member . . . 25
Test and confirm that the first server is working properly . . . 26
Installing the next server in the server group . . . 27
Configuring the next server for the server group. . . 28
Test and confirm that the current server is working properly . . . 29
Configure the server group check interval . . . 30
Configure the server group signaling option . . . 31
Set failover rankings for servers and operations . . . 33
Configure BMC Remedy Alert for the server group . . . 35
Configure full text search (FTS) for a server group . . . 36
Configure DSO for the server group . . . 37
Configure Email Engine for the server group . . . 38
Configure Flashboards for server groups . . . 38
Working with server groups . . . 39
Bypass the load balancer to work on a specific server. . . 40
Configure logging for server groups . . . 40
Installing and configuring server
groups
This white paper provides the following information:
An overview of what server groups are and how they work
How to plan a server group installation
Detailed server group installation and configuration instructions
Information on upgrading server groups
Testing and troubleshooting tips
Overview
This white paper provides detailed setup and configuration information on how to plan for, install and configure the components for a BMC Remedy Action Request System (AR System) server group. It supports any or all of the following BMC software components:
BMC Remedy AR System 7.6.04 or higher server
Server tier components including the mid tier, the Approval Server, the Assignment Engine, the Flashboards server, the Email Engine, DSO, and FTS
BMC Atrium Core 7.6.04 or higher
BMC Remedy ITSM Suite 7.6.04 or higher, including BMC Remedy Asset Management, BMC Remedy Change Management, and BMC Remedy Incident and Problem Management
BMC Remedy Service Request Management 7.6.04 or higher
BMC Remedy Service Level Management 7.6.04 or higher
Audience
This white paper is written for administrators who are responsible for setting up and maintaining the AR System in a multiple server environment. It is required that you are familiar with the AR System environment, components and
How this document relates to the installation documentation
This document is intended as a supplement to the standard AR System server and AR System component installation documentation. It does describe the full set of procedures for installing and configuring server groups, but doesn’t included detailed instructions on installing all components and options that are covered in the AR Server installation document or the installation documentation from any of the other components.
Server Groups Overview
The AR System server groups feature provides failover management for crucial operations, server scalability, application-level load balancing, and the sharing of floating licenses among the servers. A server group consists of two or more AR System servers that are managed as a single unit. The servers share the same AR System database, but perform workflow and database updates independently from each other.
Figure 1-1: AR System servers in a server group
A server group acts as a single AR System server to support applications running on multiple AR System servers. The individual servers in the server group are configured to spread the load of shared services, and to provide backup for each other to ensure that those services are always available. BMC applications that are based on AR System, such as the BMC Atrium CMDB and its related applications, as well as the BMC Remedy ITSM suite of applications can be installed and configured to operate within a server group.
To ensure high availability of AR System operations, a server group can be configured to provide failover protection by assigning rankings for specific AR System operations to the servers in the group.
Benefits of using a server group
The following are the benefits of configuring your AR System implementation using a server group:
Application-level software load balancing for heavy user traffic
Automatic server failover (if a server goes down the system seamlessly keeps running).
Ease of administration; it has only one database to manage and back up
AR System servers share all BMC software licenses except AR Server licenses
There is one server designated as an administrative server (When the workflow and applications are handled on the administrative server, the changes are automatically propagated to other servers in the group.)
Specific servers in the group can also be configured to handle reporting, reconciliation, and other tasks that can impact performance, freeing up the remaining servers in the group to handle user traffic
Server groups also work in conjunction with hardware load balancers that direct user traffic to some or all servers in the group. For best performance and reliability, BMC recommends that you use a load balancer when implementing server groups. For specific details on using a load balancer, see the Using a Hardware Load Balancer with AR System 7.6.04 white paper.
Recommendations on when to use a server group
Implement a server group if you require failover protection, or if you have a large environment that requires multiple servers. AR System can be setup to run on multiple servers without using a server group, and there may be some cases where that is the best solution, however the recommended best practice for running multiple servers is to install them as a server group.
NOTE
It is required to always use the exact same version and patch level for all BMC software applications on each server included within a server group. And, to always upgrade each application on each server within the server group at the same time.
Implementing a server group
Server groups are designed to work with AR System in any type of supported environment that has more than one server. This includes large distributed setups that make use of hardware load balancers and the Distributed Server Option (DSO). The following sections describe various configurations and optional components for using the server group feature.
Server Roles (page 10)
Licensing structure (page 12)
Server group naming (page 12)
Using a load balancer with server groups (page 13)
Using DSO with server groups (page 14)
Using FTS with server groups (page 15)
Using BMC Remedy Alert with server groups (page 16)
Using data archiving with server groups (page 16)
Server Roles
In a server group, each server is typically the primary owner of one or more specific roles. Each role represents a specific AR System application or component. In any server group implementation, no matter how simple, there is one server that is configured the administration role. This is typically the first server installed and is used to perform all administration operations for the server group. Because all of the servers share the same database, this allows the group to be managed as if it were a single server.
Other servers can be assigned specific primary roles. For example a server might be dedicated to just one specific primary task, such as Approval Server, while another server might be setup as a primary server to host a group of roles that might be closely related such Atrium CMDB and Atrium Integration Engine. The primary roles are configured on the AR System Server Group Operation Ranking form, and each server should have at least one other server configured for failover. The following is the complete list of server group roles for BMC software, as supported by the AR System Server Group Operation Ranking form.
Administration: Performs all Administration tasks for the entire server group.
Approval Server: The approval server provides the approval functionality within BMC Remedy applications. An approval represents the signature or acknowledgment of an individual where required in a process. The approval server records all necessary information to provide an audit trail and proof of authenticity of all approvals.
Archive: The archive feature of AR System provides a convenient way to periodically store data (not definitions) from a form to an archive form; this reduces the amount of data accessed during searches on the main form thus improving system performance. Archiving applies to all types of forms, except display-only forms.
Assignment Engine: Using processes instead of workflow, the Assignment Engine enables you to automatically assign requests to individuals. The
assignment method determines who is assigned to an issue when more than one person matches the qualification.
Atrium Integration Engine: The BMC Atrium Integration Engine (AIE) provides the hooks to enable data to pass between AR System and other systems, such as an Enterprise Resource Planning (ERP) system. AIE consists of the Data Exchange application and the AIE service as well as a configuration
Business Rules Engine: The business rules engine is a component of BMC Service Level Management that is used to interpret stored rules to construct the filters that are required to implement the rules. The main form that the business rules engine is the SLA:Business Rules form which contains references to objects required to create a filter.
CMDB: BMC Atrium Configuration Management Database. This is a core component of any IT Service Management (ITSM) environment and adds substantial value to a simple Incident Management environment. Specifically, it makes incident management more efficient by providing support staff and IT management an up-to-date image of their production IT environment.
DSO: The BMC Remedy Distributed Server Option (DSO) is used to build large-scale, distributed environments that behave like a single virtual system. DSO enables an organization to share common information among geographically distributed servers and to keep that information consistent. DSO enables you to transfer requests between servers and to keep copies of a request synchronized across multiple servers, even if those servers are geographically dispersed.
E-Mail Engine: A server-based module provided with AR System that communicates with both the AR System server and an email server. Email Engine receives email messages and can parse and interpret the messages to execute specific instructions on an AR System form. It also sends email messages to AR System and directs notifications as a result of filters and escalations.
Escalation: An escalation is an action or group of actions performed on the server at specified times or time intervals. Basically, an escalation is an automated, time-based process that searches for requests that match certain criteria at specified times and takes actions based on the results of the search. For example, an escalation can trigger AR System to notify the next level of
management if a problem is not assigned to a technician within one hour of submission.
Flashboards: A real-time visual monitoring tool that can show the state of service operations, warn about potential problems, and collect and display trend data.
Full Text Index: Full text index is the indexing feature for the full text search (FTS) method used in AR System to search for text in long text fields or attached documents. It is typically much faster than using the native database searching functionality
SLM Collector: The BMC SLM Collector is a component of BMC Service Level Management which is used for collecting metrics from external data sources. The collector manages the collection of performance-monitoring data and delegates the task of data collection to one or more collection points. The collection points can be on the same server as the collector, but are usually placed closer to the data sources.
Reconciliation Engine: The reconciliation engine is a patent-pending
component of the BMC Atrium CMDB. The reconciliation engine is based on business rules, which allows you to leverage existing data coming from third-party asset or discovery tools. The solution does not lock you into any one vendor’s discovery tools or existing asset repositories.
Licensing structure
Because servers in a server group use the same database, they share licenses. Each AR System server must have its own AR Server license key, but the server group feature shares all other BMC product licenses with all of the servers in the group. So for any product in a server group, when you install the license, since it gets stored in the database shared by all the servers, it only has to be installed one time. This registers it for all servers in the group.
To add a server license, follow the procedure in the Adding Licenses section of the BMC Remedy Configuration Guide.
All other license types, such as all types of Fixed and Floating user licenses and application licenses, are stored in the database and are therefore shared by all servers in the server group. You can add these other product licenses at any time. However, for all AR System servers, except the first server, the license must be added prior to installing the server.
Server group naming
For each server group, you define a common server name alias and apply it to each server in the group. The alias identifies the server group in workflow so that the workflow can run correctly on any server in the group. It is also used for other items such as notification shortcuts, and Web URLs used in notifications. You also define a unique name for each server in the group so that the servers in the group can identify each other and so that you can direct administrative or specialized operations to a specific server. Both the server alias name and the unique names must be able to resolve by DNS.
This document assumes that if there is more than one AR System Server pointing to the same database, that work is directed to the individual servers via some automated functionality, such as a hardware or software load balancer, or through manual configuration such as directing some users to one server and other users to another server. It is also possible that one server is used primarily for users while the other server is used for automations and integrations. In any case, the actual configurations for various settings, such as the server name alias, need to be considered for the specific environments.
The server name alias
All AR System servers in a group must have the same server name parameter. This is specified in the Administration Console as the Server Name Alias field. If your server group uses a load balancer, the common server name parameter should be the same as the short DNS name of the load balancer (which is a name that resolves to the load balancer IP address). Clients can therefore be directed to the load balancer using the server name parameter.
NOTE
If the server name alias setting is not the same as the load balancer short DNS name, ARTask email attachments generated by the server might not work. When generating an ARTask attachment, the server generates a reference to itself using the server name parameter with the domain name appended. Clients opening the ARTask will then use the fully qualified domain name to be routed to the server group through the load balancer.
Unique names for each server
Each server in a server group needs a uniquely defined name to identify itself to other servers in the group. Usually this is the host name of the computer where the server is installed.
To identify the unique name for each server in the group, the following line is added to each server's configuration file:
Server-Connect-Name: servername
DNS must be able to resolve this name, and it is used exactly as specified (that is, no domain name is appended). Each server uses this name to register as a server group member. Other servers in the group use the name when communication between servers is required. In addition, various external server components use the name when connecting to the local server. This name can be specified as either the short name or the fully qualified name.
The server group name
The server group name was used in some earlier releases in relationship to licensing, but it is no longer necessary to set this value. In release 7.5.00 and later, this setting is not used, however, there is still a field for it in the Administration Console. That field can be left blank.
Using a load balancer with server groups
The load balancer acts like a NAT device that routes any TCP or UDP traffic. Since the AR System server uses an ONC-RPC implementation that is layered on top of TCP/IP, AR System server traffic can be load balanced. Server groups are
independent of load balancing, but the concepts are complementary.
The example below uses a load balancer to direct traffic to a server group of three AR System servers. In Figure 1-2, the uppermost AR System server has the primary ranking for the Administration and Escalation operations. The other two AR System servers can be used to back up these operations, when the uppermost server is not running.
Figure 1-2: Basic load-balancer configuration with multiple AR System servers
For more information, see the BMC Remedy white paper, Using a Hardware Load Balancer with AR System 7.6.04. This white paper is posted on the Customer Support website (http://www.bmc.com/support).
NOTE
If the load balancer belongs to a different domain than the AR System Servers, then the fully qualified domain name of the Server-Name alias will be wrong. In this case, the domain name parameter should be specified in the ar.cfg file for each AR System server using the domain of the load balancer
Using DSO with server groups
The BMC Remedy Distributed Server Option (DSO) is used to build large-scale, distributed environments that behave like a single virtual system. DSO enables an organization to share common information among geographically distributed servers and to keep that information consistent. DSO enables you to transfer requests between servers and to keep copies of a request synchronized across multiple servers, even if those servers are geographically dispersed.
To configure DSO for load balancing, in addition to configuring the hardware load balancer, you must configure multiple servers to use a single database. To do this, a server group is used.
In a server group, you DSO is used to provide automatic fail-over capability for transfer operations typically restricted to one server. You do this by creating a distributed mapping and then specifying that transfers can be initiated from any server in the group. For detailed information on the DSO option, see the BMC Remedy Distributed Server Option Guide.
For example, as illustrated in Figure 1-3 on page 15, you can transfer copies of a request to other servers and ensure that any changes made to the copies are also made to the original request. The way that you define the processes for
transferring information is similar to the way that you define business processes for an application. First, managers at each site must agree on what information to transfer from one application to another, what conditions drive transfers, and which sites control the ability to update a record. An administrator at each site then uses DSO to implement these decisions.
Figure 1-3: Distributed AR System servers in a server group using DSO
Using FTS with server groups
If you use Full Text Search (FTS) in a server group, only one server in the group can index data at a time. In a server group, the server that owns the full text indexing operation processes all pending indexing tasks regardless of their server of origin. The other servers have read-only access to the index files.
If the server that owns the indexing operation fails, the ownership of the indexing fails over to the next highest ranked server that is running, as set on the AR System Server Group Operation Ranking form. The full text indexing operation can also fall back to a higher ranking server when that server comes back online. This fallback is delayed until in-progress indexing tasks are completed.
FTS is configured after all servers in the group have been installed and configured to run within a server group. It is recommended that the FTS collection directory and the FTS configuration directory be located on a separate computer in a fault tolerant location where both AR servers have network access with read and write permissions.
Using BMC Remedy Alert with server groups
When an Remedy alert client registers with an AR System server in a server group, it registers on only one of the servers. That registration is automatically broadcast to the other servers in the group, but some IP address configuration is required to properly register the client on the other servers. In a load-balanced environment, the registration information contains the load balancer IP address instead of an actual server IP address. Therefore, each server must have an IP address mapping from the load balancer IP address to its own IP address.
Remedy Alert is configured after all servers in the group have been installed and configured to run within a server group.
Using data archiving with server groups
By default, the data archiving feature is enabled on an AR System server. To disable archiving for all forms on a server, select the Disable Archive option on the Configuration tab of the AR System Administration: Server Information form. For a server group, you can disable archiving on all the servers except one if you want archiving to take place on only one server. To do this, configure the server group in the AR System Server Group Operation Ranking form to make sure that only one server performs the archiving operation.
Server group planning
There are many things to plan when first deciding to use server groups, and it’s important to map everything out ahead of time. The following are key planning details:
How many servers do you plan to use?
Which applications and components will run on which servers?
What database type are you using?
What hardware do you need to support the servers?
What type of network connection and communication is needed?
Will you be using a load balancer?
Are the servers at the same geographic location?
Are you going to use the Distributed Server Option (DSO)?
Do you have a staging and testing environment?
If you will be using FTS, do you have access to a shard file system and the necessary permissions configured?
It is recommended to have all of those questions answered before starting a server group setup. It is also recommended to create a list of licenses that will be needed for all products, including the AR System server licenses.
Example configurations
This section contains examples of a simple configuration and a complex configuration.
Simple example
In the simplest form, a server group can be setup with two AR System servers and a database server. Each of the AR System servers have all Remedy products installed.
In this example, the first server installed should be configured to be the primary administration server. That server will handle all of the server configuration tasks. Then, using the AR System Server Group Operation Ranking form, the
applications should be distributed evenly across both servers, so that each server handles about half of the load, and each server has the other server configured for failover on each of its applications.
The exception to this is the email engine and the flashboards server. In a simple configuration, those two items are only installed and configured on one server. Configuring failover for those operations can be complex and may not be necessary in a simple configuration.
The full text search feature should be installed on each server. Each server has the ability to read from FTS, but only one server can be set to write, which is the server that is set with a higher priority on the AR System Server Group Operation Ranking form. It is also recommended that the FTS collection directory (location where the search index files are stored) and FTS configuration directory (location where the search configuration files are stored) be located on a separate computer in a location where both AR servers have network access.
Figure 1-4: Simple server group example
Complex Example
A more complex server group implementation contains three or more AR System servers. In this example we are using four AR System servers. It is still
recommended to install all Remedy products on each server, but it is not required. However, each application or component should be installed on at least two servers so that failover can be provided.
Again, the first server installed should be configured to be the primary administration server. The administration server handles all of the server
configuration tasks. Then, using the AR System Server Group Operation Ranking form, the applications are distributed evenly across all four servers, so that each server handles about one quarter of the load, and each server has at least one other server configured for failover on each of its applications and components.
In this case, even the email server and the flashboards have a failover server assigned. Configuring failover for those operations is somewhat complex because it means that each server has to be specifically configured to handle those items, but the secondary server for each needs to be disabled until the failover is activated.
Figure 1-5: Complex server group example
NOTE
The applications and components listed for the servers above are just the primary roles for each server. It is recommended that all applications and components be installed on each server. This is important because users could be accessing any components from any server in the group, and there are dependencies such as plug-ins and other binary files that could be called when a user opens certain forms, creates a new record, or modifies a record.
In this example, FTS is setup on AR Server 2, so that is the only server with read-write access to the FTS collection directory. The FTS feature should be installed on each server. It is also recommended that the FTS collection directory (location where the search index files are stored) and FTS configuration directory (location where the search configuration files are stored) be located on a separate computer in a location where all AR servers have network access.
BMC Software Installation Planning - Products and Components
Make sure you know exactly what products you plan to install, and determine which servers will be running each product. BMC recommends that you install all products on all servers, and then to use the AR System Server Group Operation Ranking form to distribute the load; however, there are may different ways to do this and each decision is based on the specific implementation.List out the name for each server and its IP address, as well as all accepted fully qualified names. Do this for the database server and the load balancer if used. In the installation instructions, a two system server group is used and the systems have the following information.
Make sure to obtain similar information for your implementation, before you start the installation process:
AR System Server 1: Name: svr_grp_tst0.svgroup.com IP: 92.161.135.31 AR System Server 2: Name: svr_grp_tst3.svgroup.com IP: 92.163.169.2 Database Server: Name: svr_grp_tst10.svgroup.com IP: 192.168.112.40 Other Info Needed:
Server -Name (resolves to the load balancer name): RemedyServerGroup
Domain: svgroup.com
Subdomain: test.svcgroup.com
Installing a server group (new installation)
This section descries how to install a server group on a a set of ‘clean’ systems.
Before starting the installation
The following sections provide a summary to help you make sure you are ready for the installation.
Using clean systems for installation
A clean system is one that has not had any BMC Remedy software previously installed on it. If there has been BMC Remedy software installed on one or more of your servers, it is strongly recommended to re-image the system back to a state where is has never had any BMC Remedy products installed. Uninstalling previous products is a good idea, but it is recommended to re-image to make sue they are clean.
Follow your plan
At this point, you should already have a plan of how many AR System servers you will use, what BMC Remedy products and components you are installing, what hardware you are going to use, and how you are going to distribute those products across the servers. If you do not have this, then go back to the previous sections that describe how to plan this out.
Hardware and software requirements
Before beginning, verify that each system you plan to install within your server group meets the minimal server requirements for an AR System server. See the BMC Remedy Action Request System Installation Guide for details on server software requirements and minimal hardware requirements.
Using the Stack Installer
The BMC Remedy ITSM Suite Preconfigured Stack 7.6.04 Installer is an installation package that contains the AR Server as well as the products and components listed below. Use this installer to save time when you are installing all applications on a server. The BMC Remedy ITSM Suite Preconfigured Stack 7.6.04 Installer can be used with a server groups installation; however, it requires a slightly different procedure on the server configuration for all servers, except for the first server. This difference is noted in the installation procedures listed later in this document.
BMC Remedy AR System server version 7.6.04
AREA LDAP Directory Service Authentication
ARDBC LDAP Directory Service Data Access
Web Services Plugin
Simple Network Management Protocol (SNMP) Configuration
Full Text Search (FTS) Configuration
Approval Server
Assignment Engine
Email Engine
Flashboards
BMC Remedy AR System clients
BMC Remedy User
BMC Remedy Alert
BMC Remedy Developer Studio
BMC Remedy Data Import
BMC Remedy Migrator
Crystal Reports
ODBC
BMC Atrium Core
BMC Atrium CMDB version 7.6.04
Product Catalog version 7.6.04
Atrium Impact Simulator version 7.6.04
BMC Remedy ITSM Suite
BMC Remedy Asset Management version 7.6.04
BMC Remedy Change Management version 7.6.04
BMC Remedy Service Desk 7.6.04
BMC Service Level Management version 7.6.04
BMC Remedy Knowledge Management version 7.6.04
BMC Service Request Management version 7.6.04
Server group installation overview
The following is a brief outline of the server group installation procedures that are explained in detail throughout the next several sections:
Step 1 Install and configure the first AR System server as a stand-alone server with all products. This includes the database installation. See the Installation Guide for details.
NOTE
BMC recommends that you install the mid tier on the first server so that you can use it to confirm that the product installations are working correctly, and to access the server administration console to perform configuration options.
Step 2 Test the first server and verify that all installed products and components are working properly.
Step 3 Configure the first server for server groups.
Step 5 Perform the final manual configuration for server groups, including updating the server configuration file.
Step 6 Install all the additional products on the next server, also making sure to use the server group installation method. This server group method is automatically selected in most cases, as the products and component installations can see that the already installed AR Server is setup as a server group. Each of the product installations handle that a little differently and most of them display some type of message indicating that they recognize the server group and are installing as such. Verify that each product or component is working before you attempt to install another product or component.
Step 7 Repeat steps 4, 5 and 6 for each subsequent server in the group. Figure 1-6: Server Groups Installation Flow
Installing the first server and the database
This section describes how to install and configure the first server in the server group installation. These procedures assume that you have downloaded the software and are ready to go. If not, see the Installation Guide for details.
IMPORTANT
On Windows, you must be logged in as a domain user on the server rather than a local user when performing installations. The installer needs to copy files to the database server. On Unix, you must also be logged in with a user the has permissions to copy files to the database server.
Note that you will be installing the database during the installation of the first server, so make sure to have the database server name and IP Address available. Make sure to read through the instructions for each step before attempting to perform the procedures.
1 Install the Remedy AR Server v7.6.04 or the BMC Remedy ITSM Suite Preconfigured Stack 7.6.04 installer on the first system.
It’s strongly recommended that you install the mid tier on the first server. See the Installation Guide for installation procedures and details.
Do not select the option for server group installation. The first server needs to be setup and configured as a stand-alone server when installed. Following the installation, the server is manually configured to be a server group member.
2 Test that the that the installation is working properly by opening the mid tier, confirming that the forms render in the mid tier interface. For details on accessing and using the mid tier, see the BMC Remedy mid tier Guide.
3 Install the Remedy AR Server License on the first server. See the Configuration Guide for details on installing the license.
NOTE
If you used the BMC Remedy ITSM Suite Preconfigured Stack 7.6.04 installer, skip the next step.
4 Follow the same procedures as in steps 1 - 3 to install each of the rest of the BMC Remedy products on the first server (when repeating step 3, make sure to install the correct product license for the specific product being installed). See the Installation Guide for each individual product for details in individual product and component installation details.
After the Remedy AR Server is installed, it is recommended to install the products in the following order:
BMC Atrium Core
BMC Remedy ITSM Suite
BMC Service Request Management
BMC Service Level Management
Configure the first server to be a server group member
There are two steps to configuring the first server to be a member of a server group. The first step uses the mid tier to change a setting in the Administration Console and the second step is to modify the Remedy server configuration file, ar.cfg.
1 Using a Web browser, access the mid tier on first server. For details on accessing and using the mid tier, see the BMC Remedy mid tier Guide.
2 From the mid tier, choose Administration Console > General > System Information, then select the Configuration tab. Then, select the Server Group Member check box and click Apply.
3 Locate the ar.cfg file. On a Windows server, it is located in the \[Remedy Install Folder]\ARSystem\Conf folder, and open it in a text editor.
a Confirm that the Server-Group-Member: setting is set to T ( Server-Group-Member: T). If you don’t see this setting, create it and set it to T, or it’s set to F, change it to T.
b Also confirm that the Multiple-ARSystem-Servers: setting is set to T (Multiple-ARSystem-Servers: T). If you don’t see this setting, create it and set it to T, or it’s set to F, change it to T.
c Check for an entry called Server-Name. If it exists, then verify that is set to a name that resolves to your load balancer, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our
installation test example the name is RemedyServerGroup.
d Check for an entry called Server-Connect-Name. If it exists, then verify that is set to the name if the first server, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our installation test example the name of the first Remedy AR Server is svr_grp_tst0.
e Check for an entry called Domain-Name. If it exists, then verify that is set to the lowest subdomain name used in your implementation, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our installation test example the lowest subdomain name is
test.svgroup.com.
f Create an IP-Name: entry for every possible way a client could connect to the current server. For example, in our test scenario we added the following four entries for the first server:
IP-Name: svr_grp_tst0
IP-Name: svr_grp_tst0.svgroup.com IP-Name: svr_grp_tst0.test.svgroup.com IP-Name: 192.161.135.31
g Now, looking ahead to the installation of the rest of the servers, create the same number of entries for each of your AR Servers. Since our test scenario only uses two servers, we added:
IP-Name: svr_grp_tst3
IP-Name: svr_grp_tst3.svgroup.com IP-Name: svr_grp_tst3.test.svgroup.com IP-Name: 192.163.122.40
NOTE
Once you create this for all of your servers, copy the entire set of IP-Name entries and save it into a separate text file. This will used later on to save you time when configuring the other servers. Each server must have the exact same set of entries containing all resolvable names for each server.
4 Save the ar.cfg file and restart the server.
Test and confirm that the first server is working properly
After the server has rebooted, perform the procedures listed below to confirm that it is properly configured in server group mode and that everything is working properly.
1 Using a Web browser, access the mid tier for Server 1, and log in.
2 Type the following into the URL to get to the AR System Server Group Operation Ranking form:
http://[midTierServer:portNumber]/arsys/forms/[arSystemServer]/ GroupAR+System+Server+Group+Operation+Ranking/
3 From that page, click Search, in the top left corner of the screen to get the display of all current rankings. Verify that all of the Remedy applications and components are listed in the form, and that they are all set to Rank 1.
Installing the next server in the server group
This section describes how to install and configure the next server in the server group installation (servers B-Z). These procedures assume that you have downloaded the software. If not, see the Installation Guide for details.
NOTE
This set of procedures is repeated for each additional server, following the initial server installation.
1 Open the mid tier on the first server and configure the Remedy AR Server License for the next server. See the Configuration Guide for details on installing the license.
2 Install the Remedy AR Server v7.6.04 or the BMC Remedy ITSM Suite
Preconfigured Stack 7.6.04 installer on the next server. See the Installation Guide for installation procedures and details.
For all the second server and all servers afterward, be sure to select the Server Group option. For many of the product installs, when you enter the database name, it will automatically go into server group installation mode because it recognizes that the database is already populated by an AR Server which is configured for server groups.
Configuring the next server for the server group
There are two steps to configuring the next server to be a member of a server group. The first step uses the mid tier to change (or confirm) a setting in the Administration Console and the second step is to modify the Remedy server configuration file, ar.cfg.
1 Now, locate the ar.cfg file. On a Windows server, it is located in the \[Remedy Install Folder]\ARSystem\Conf folder, and open it in a text editor.
2 Confirm that the Server-Group-Member: setting is set to T ( Server-Group-Member: T). If you don’t see this setting, create it and set it to T, or it’s set to F, change it to T.
3 Also confirm that the Multiple-ARSystem-Servers: setting is set to T
(Multiple-ARSystem-Servers: T). If you don’t see this setting, create it and set it to T, or it’s set to F, change it to T.
4 Check for an entry called Server-Name. If it exists, then verify that is set to a name that resolves to your load balancer, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our installation test example the name is RemedyServerGroup.
5 Check for an entry called Server-Connect-Name. If it exists, then verify that is set to the name if the current server, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our installation test example the name of the first Remedy AR Server is svr_grp_tst3.
6 Check for an entry called Domain-Name. If it exists, then verify that is set to the lowest subdomain name used in your implementation, or if not modify the setting’s value so it does. If the entry does not exist, create it and set accordingly. For our installation test example the lowest subdomain name is
test.svgroup.com.
7 Obtain the text file that you created with the IP-Name: entries for all servers in the server group, when you were configuring the first server. Remove any existing IP-Name: entries from the file and paste the text with the IP-Name: entries for all
8 Save the ar.cfg file and restart the server.
Test and confirm that the current server is working properly
After the server has rebooted, perform the procedures listed below to confirm that it is properly configured in server group mode and that everything is working properly.
1 Type the following into the URL to get to the AR System Server Group Operation Ranking form:
http://[midTierServer:portNumber]/arsys/forms/[arSystemServer]/ GroupAR+System+Server+Group+Operation+Ranking/
2 From that page, click Search, in the top left corner of the screen to get the display of all current rankings.
3 Verify that all of the Remedy applications and components are listed in the AR System Server Group Operation Ranking form for all installed servers are there. Also verify that the rankings for the current server are all set to the rank number of the current server (i.e. 2 for the second server, 3 for the third, etc.?
NOTE
If you used the BMC Remedy ITSM Suite Preconfigured Stack 7.6.04 installer, you can skip the next step in this section because all of the applications have already been installed on the current server
4 Install each of the rest of the BMC Remedy products on the current server.
5 If this is the last server you are installing, you are now finished. If it is not the last server, go back to the “Installing the next server in the server group” section, and go through the same process for the next server. Repeat this until all servers are installed and configured.
NOTE
If the product you are installing was already installed and licensed on the first server, then you don’t need to install a license for it. With server groups each license, other than the Remedy AR System server license, only needs to be installed once.
After the Remedy AR Server is installed, it is recommended to install the other products in the following order:
BMC Atrium Core
BMC Remedy ITSM Suite
BMC Service Request Management
BMC Service Level Management
BMC Remedy Knowledge Management
Configure the server group check interval
The Check Interval setting determines how often each server in the group
identifies itself to other servers in the group, as well as how often it checks to see whether other servers are still alive. The Check Interval works together with the Delinquent Threshold to determine the total time from a server failure to when the next ranked server takes over an operation. See “Delinquent threshold” on page 35.
Checking server group status and reporting status generates a certain amount of database traffic, so you should tune this setting to balance the time to failover against the frequency of posting the check and query to the database.
To set the Check Interval
1 Open the AR System Administration: Server Information form.
2 Click the Advanced tab.
Figure 1-7: Setting the check interval on the Advanced tab
3 In the Check Interval field, enter how often you want the server to identify itself and check the status of other servers in the group.
Values are
Default—60 seconds
Minimum—30 seconds
Maximum—None
If you change this value after a server group is running, you must restart all the AR System servers.
The information shared between servers in the group includes the following:
The current server’s own status
Whether any server is delinquent
The parameters needed for sending signals
Information about operational responsibilities
For an explanation of delinquency and rankings, see the Configuration Guide.
4 Click OK and then restart each server in the server group.
Configure the server group signaling option
When object definition changes are made on the administrative server in a server group, other members of the group can be notified either by a database posting or by a signal:
Notification type Description By default, handles these changes Database posting Reduces server activity when object
definition changes are communicated between servers and reduces the number of cache reloads when a series of changes is made.
A delay occurs before other members of the server group pick up the changes.
Server definition changes—such as changes to forms, active links, filters, and
escalations—and user group changes.
Signaling Other servers are notified of changes immediately.
No delay occurs in resynchronization or updating definitions.
All other changes, such as Alert registration, DSO activity, and so on.
Note:You can configure the server to use signaling for all changes, including server object changes. See “Using signals for all changes in server groups” on page 32.
About server group signaling
Beginning with release 7.5.00, servers in a server group use the arsignald daemon to notify other members of the group about signaled changes. This daemon (arsignald.exe on Windows and arsignald on UNIX) is a persistent process started by the AR System server at server start up. It maintains a pipe to the associated AR System server and handles all signals to the other members of the server group. Therefore, signaling between members of a server group requires only one additional server process per server.
NOTE
In previous releases, server group signaling was performed by the arsignal
program, which caused a separate process to be spawned and then closed down for every change. This could significantly impact resources on the host computer. The arsignal program is still available for use by AR System workflow, but is no longer used for server group signaling.
Using signals for all changes in server groups
The Server-Group-Signal-Option server configuration file option tells the server whether to use arsignald for all signals instead of using a combination of signaling and database posting. If this option is set to true (T), all server object changes are communicated by arsignald. Use this option to avoid any delay in communicating server object changes to other servers in the server group. To use signals for all changes in server groups, add the following line to the configuration file (ar.conf or ar.cfg) of each server in the server group:
Server-Group-Signal-Option: T
If this option is set to false (F) or is not included, server group signals are accomplished by the default method described in this section.
NOTE
Form, workflow, and escalation time changes can significantly increase the workload on a production server. In a server group environment, that effect is magnified when other servers are notified of the changes and they recache definitions from the database. Consider this when planning changes of this type.
Disabling automatic signaling in server groups
Signals triggered by certain object definition changes on the administrative server can cause recaches on target servers that significantly increase memory use temporarily. This increase can adversely affect the response time of the target servers. To change object definitions on the administrative server without
impairing the performance of target servers, you can disable automatic signaling from arsignald and the database for changes to the following data:
Form definitions
Group information from the database
Workflow definitions
Signals triggered by changes to user, licensing, and computed group information are not disabled.
Later, when memory use is low, you can manually send the signals to the target servers by using the arsignal program.
To disable automatic signaling for the specified changes, add the following line to the configuration file (ar.conf or ar.cfg) of each server in the server group:
Disable-ARSignals: T
If this option is set to false (F) or is not included (the default), automatic signaling remains in effect for all object definition changes.
Set failover rankings for servers and operations
The AR System Server Group Operation Ranking form contains entries that define the failover ranking for servers that handle certain operations in the server group, and the delinquent threshold that helps determine when another server takes over an operation. This form is automatically created when you specify the first server as a member of the server group and restart the server.
When the form is created, it is populated with default entries and the first server added to the server group is assigned the primary ranking for all operations. The remaining server group members have null (empty ranking) entries, serving as placeholders. Entries for operations that require a license (for example, DSO) are not pre-populated unless a valid license is detected. You can add these operations at any time.
NOTE
Remove the default entries for operations that do not run in your environment. Figure 1-8: AR System Server Group Operation Ranking form
Operation rankings
The fields named Operation, Server, and Rank work together to define which server is the primary server for the operation, which server takes over if the primary server fails, and so on.
To establish operation rankings in the server group
1 In BMC Remedy User or a browser, log in as a user with Administrator privileges to any server in the group.
2 Open the AR System Server Group Operation Ranking form in search mode.
3 Perform an unqualified search to see the entries in the form.
4 Modify entries as required to construct a fail-over hierarchy for ownership of operations.
Use the following guidelines to determine how to set operation rankings for the server group:
The servers for any one operation are ranked lowest to highest. A value of 1
indicates the server chosen first to perform the operation. The next highest value indicates the server that takes over the operation if the first server fails, and so on.
Ranking numbers do not need to be consecutive.
If a value is null, the server ignores the entry.
If an operation has no server designated with a valid rank, it is not run on any of the servers in the group.
Avoid assigning two servers the same ranking for the same operation. (For ease of configuration, the form enables you to do this temporarily.)
Operations can be spread freely across different servers, with the exception of operations involving BMC Remedy Approval Server, BMC Atrium CMDB, and the BMC Service Level Management engine (labeled Business Rules Engine in the form). These operations must reside on the same server as the administration operation; therefore, the operations must have the same ranking as the administration operation so that they move as a unit.
If you are implementing full text search (FTS), an additional restriction for multi-platform server groups exists. The Administration and Full Text Indexing operations must be restricted to server group nodes that can have a compatible directory structure for the Search Server configuration files. For more information, see the Installation Guide.
5 Save the AR System Server Group Operation Ranking form.
Delinquent threshold
The Delinquent Threshold field determines the number of times the specified server can miss reporting its status before the next server in the ranking takes responsibility for the operation. This setting works together with the Check Interval to determine the total time to failover for any operation.
Configure BMC Remedy Alert for the server group
If you do not use BMC Remedy Alert and the alert mechanism in your environment, you can skip this section.
Mapping IP addresses
Mapping load-balancer IP addresses to AR System server IP addressesWhen an alert client registers with an AR System server in a server group, it registers on only one of the servers. That registration is automatically broadcast to the other servers in the group, but some IP address configuration is required to properly register the client on the other servers. In a load-balanced environment, the registration information contains the load balancer IP address instead of an actual server IP address. Therefore, each server must have an IP address mapping from the load balancer IP address to its own IP address.
To map the load-balancer IP address to the AR System server IP address
For each AR System server in the group, add the following line to the configuration file of the server:
Map-IP-Address: loadBalancerIP ARSystemServerIP
The parameters are as follows:
loadBalancerIP—The virtual IP address of the load balancer that Alert clients use for connecting.
ARSystemServerIP—The IP address of the local AR System server.
NOTE
Because each AR System server is configured individually, you must repeat this procedure for all AR System servers in the group.
Mapping remote server IP addresses to local server IP addresses
In a non-load-balanced environment (or a load-balanced environment where users are allowed to connect directly to a server), the registration information contains the server IP address of the server to which the user connected. Therefore, each server must have an IP address mapping from all other servers to its own IP address.
To map all other server IP addresses to the AR System server IP address
For each AR System server in the group, add the following line for each remote and local server combination to the configuration file:
The parameters are as follows:
remoteServerIP—The IP address of a remote server to which an Alert client could connect.
localServerIP—The IP address of the local AR System Server.
For example, if you have three AR System servers (A, B, and C) in the group, add the following lines to the ar.conf (ar.cfg) file for server A:
Map-IP-Address: serverB_IPAddress serverA_IPAddress Map-IP-Address: serverC_IPAddress serverA_IPAddress
NOTE
Version 5.1 or higher of the BMC Remedy Alert client is required in a server group environment.
For more information about BMC Remedy Alert, see the Configuration Guide.
Configuring the server to ignore client IP addresses
When an alert client registers with the AR System server, it provides its own IP address as part of the registration information. If the server detects a difference between the registered IP address and the actual IP address from which the client call originates, the server tries to send alerts to the actual address by default. In a load-balanced environment, the actual IP address is the address of the load balancer, and alerts sent to that address fail.
To send alerts to registration IP addresses and ignore actual IP addresses
Add the following line to the configuration file:Alert-Ignore-Actual-Client-IP: T
Configure full text search (FTS) for a server group
If you use FTS in a server group, only one server in the group can index data at a time.
In a server group, the server that owns the full text indexing operation processes all pending indexing tasks regardless of their server of origin. If the server that owns the indexing operation fails, the ownership of the indexing fails over to the next highest ranking server that is running, as set on the AR System Server Group Operation Ranking form. The full text indexing operation can also fall back to a higher ranking server when that server comes back online. This fallback is delayed until in-progress indexing tasks are completed.
To enable FTS in a multiple-server environment
1 Make sure that all AR System servers in the group share the same FTS index collection directory. You can specify the location of this directory in the following ways:
Provide the directory name during installation when you are prompted.
Specify the directory name in the Full Text Search tab on the AR System Administration: Server Information form.
Add the following server options to the ar.cfg or ar.conf file:
Full-Text-Collection-Directory: <directory_name> Full-Text-Configuration-Directory: <directory_name>
Make sure that all AR System servers in the group can access the collection and configuration directories.
2 If you change the collection or configuration directory, restart the AR System server.
Windows configuration example:
If the shared full-text collection directory resided in a shared directory called
Collection on serverX, the following server option would be added to the
ar.cfg file:
Full-Text-Collection-Directory: \\serverX\Collection
If the shared full-text configuration directory resided in a shared directory called
Configuration, the following server option would be added:
Full-Text-Configuration-Directory: \\serverX\Configuration
Be sure that the login account for the AR System server service has access to the shared location. The default System Account does not have access, and therefore you cannot use it with this configuration.
UNIX configuration example:
If the shared full-text collection directory were named /home/root/collection, the following server option would be added to the ar.conf file:
Full-Text-Collection-Directory: /home/root/collection
If the shared full-text collection directory were named /home/root/ Configuration, the following server option would be added:
Full-Text-Configuration-Directory: /home/root/Configuration
Configure DSO for the server group
If Distributed Server Option (DSO) is not used in your environment, you can skip this section.
NOTE
To configure DSO in a server group environment, you must specify a server group name.
In an AR System environment with load balancing or multiple servers, a DSO server process runs on each server, but only one process actively distributes data. The configuration needed to support DSO fail-over is limited to modification of the distributed mappings to indicate that any server in the server group can act as the source server.
In BMC Remedy Developer Studio, double-click Distributed Mappings in the AR System Navigator object tree. On the Basic panel of each distributed mapping, select the Allow Any Server in Server Group check box. This setting indicates to the servers in the group that regardless of the source (From) server name specified in the mapping, any server in the group is qualified to be the source server. In the event of a fail-over where the distributed operation moves from one server to another, the data continues to be processed.
See the BMC Remedy Distributed Server Option Guide.
Configure Email Engine for the server group
If the port number (RMIPort) for email administration in BMC Remedy Email Engine is different from the default (1100), set the corresponding option in the AR System server configuration file (ar.cfg or ar.conf) to the same port number.
For a single email engine configuration, the syntax is as follows:
Server-Group-Email-Admin-Port: 2020
If multiple email engines are configured for the server, each engine must have a unique RMIPort. For a multiple email engine configuration, use semicolons to separate the RMIPort numbers as follows:
Server-Group-Email-Admin-Port: 2020;2030;2040
This enables the server to communicate email administration information to BMC Remedy Email Engine during server group processing. When multiple port numbers are specified, the server sends the same information to each port number. For more information about BMC Remedy Email Engine, see the BMC Remedy Email Engine Guide.
Configure Flashboards for server groups
If the port number (RMIRegistryPort) for flashboards administration in the Flashboards server is changed from the default (1099), set the AR System server configuration in ar.cfg or ar.conf to the same port number. The syntax is as follows:
Server-Group-Flashboards-Admin-Port: 2021
This enables the server to communicate flashboards administration information to the Flashboards server during server group processing.
Working with server groups
This section describes how to connect directly to a specific server in the server group, such as the administrative server, how to configure server group logging, and how to add or remove a server from a server group.
Bypass the load balancer to work on a specific server
To change object definitions and workflow on the administrative server in a load-balanced environment, you must bypass the load balancer to connect to the administrative server with BMC Remedy Developer Studio. The same
consideration applies when you must make configuration changes on a specific server in the group with BMC Remedy User or a browser.
To do this, you can connect to the server by using the unique server name rather than the load balancer name (which is the same as the common server name alias for the group).
To ensure that the server recognizes references to itself as the current server while designing workflow and applications, add the IP Name: setting to the
configuration file. This setting indicates to the server that any of a given set of server names is recognized as the current server. You might need to designate a short name and a long name for each server as shown in the following example:
IP-Name: serverA
IP-Name: serverA.remedy.com
If BMC Remedy Developer Studio used either of these server names to log in, that server is recognized as the current server in workflow.
Configure logging for server groups
Information tracked in the server group log file includes the starting and stopping of operations, the evaluation of other servers, and the timing of each event. The
arsignald log file tracks the startup and shutdown of the arsignald daemon and all signals sent between servers in the group.
To turn on the server group and arsignald log files
1 Open the AR System Administration: Server Information form, and then click the Log Files tab.
2 Select the logging to use:
For server group logging, select the Server Group Log check box.
For arsignald logging, select the ARSIGNALD Log check box. Figure 1-9: Server Group entry on the Logging tab
3 Enter a path for each log file if you do not want to use the default paths.
4 Click OK.
To log server group activity in the Server Events form
1 Open the AR System Administration: Server Information form, and then click the Server Events tab.
