Chapter 02
Diagnostics &
Troubleshooting
Chapter 02 Module 01
Using Diagnostic Tools
Module Introduction
This module provides information about Avaya provided diagnostic tools.
After completing this module you will be able to:
Summarize the use of the diagnostic tools provided by Avaya for IP Office
Identify where to acquire the Avaya provided diagnostic tools
Explain the capabilities of Avaya provided diagnostic tools
Capture traces using Avaya provided diagnostic tools Avaya provides three diagnostic tools for use specifically with IP Office: Debug View, System Status Application and System Monitor.
Avaya Diagnostic Tools
Depending on the nature of an issue, there are several tools that may be useful for
troubleshooting problems. For example, Ping, Trace Route and Netstat are useful for
troubleshooting network connectivity issues. Additionally, Avaya provides three
diagnostic tools for use specifically with IP Office:
Debug View – used with Voicemail Pro System Status Application - a diagnostic tool for system managers and administrators, in order to monitor and check the status of IP Office systems
System Monitor – used by system managers and administrators in order to capture system
Avaya Diagnostic Tools installation
Installation – The System Monitor and System Status Application are provided with the IP
IP Office Debug View
The voicemail console mode (Interact with Desktop) is no longer used and Avaya recommends using the Debug Viewer to view voicemail server activity. The DbgView
application can be found in the attachments of this course DVD or can be downloaded on:
http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/ general/debug/index.htm
It can be used to view server activity real time or logged to file
Reporting detail level can be configured
Used for tracing various functions including:
– Voicemail Connections – Outcalling
– Message Processing
Debug View Logging Options
By default events are shown in the DbgView window. The output can also be logged to a specified file.
– Select File | Log Preferencesfrom the DbgView menu
– Specify which Log Mode to use
– Enter the name of the file to store the information
Debug View Filtering
The level of filtering can be set from 0 – None to 9 – High Trace
to control the level of detail.
Specific items can be selected or deselected to target specific output message types (e.g. TTS Support or MAPI Support)
Starting System Status Application (SSA)
SSA can be launched independently or via Manager. For IP500v2 systems, it can also
be launched by browsing to the IP Office IP address without needing to install the
application.
Click the Windows Start icon and select Programs | IP Office | System Status
From within the IP Office Manager application, select File | Advanced | System Status Using a browser, enter the IP address of the IP500v2 system, for example 192.168.42.1 The Offline tab of the SSA Login provides the option to open previously saved files such
Services Base TCP Port
The port(s) on which the IP Office listens for requests to access services on the LAN1 IP address.
Each service uses a port offset from the base port value. If this value is changed from its
default, the Manager application must be set to the same Base TCP Port through its Services Base TCP Port setting (File | Preferences).
The port is set on the System Details screen within the IP Office Security Settings (File | Advanced | Security Settings).
Service Method Port Used Default IP Office
Configuration Unsecured Base TCP Port 50804 3.2+
Secured Base TCP Port plus 1 50805 4.1+
System Status Interface Unsecured Base TCP Port plus 4 50808 4.0+
Security Administration Unsecured Base TCP Port plus 8 50812 3.2+
Secured Base TCP Port plus 9 50813 4.1+
Assigning Security Rights
The default IP Office service users Administrator, Manager and Operator have SSA access rights.
There can be up to two SSA clients connected to an IP Office unit at one time. The CCR application uses the same interface to the IP Office control unit as SSA.
System Status Application (SSA)
SSA provides information on the following:
AlarmsSSA displays all alarms which are recorded within IP Office for each device in error. The number of occurrences and the date and time of the last occurrence are recorded.
Call Details
Information on incoming and outgoing calls; including call length, call reference and routing information.
Extensions
SSA details all extensions (including device type and port location) on the IP Office system. Information on the current status of a device is also displayed.
Trunks
IP Office trunks and connections (VoIP, analog and digital) and their current status are displayed.
System Status Application (Continued)
Resources
IP Office includes central resources that are utilized to perform various functions. Diagnosing these resources is often critical to the successful operation of the system. Those resources include:
– Voicemail Channels – Conference Channels – Data Channels
– VCM Channels – Modem Channels
Voicemail
SSA provides real time information on the number of licenses available and details of the voicemail mailboxes including the number of messages and the current status of basic voicemail settings for the associated mailbox user or hunt group.
IP Networking
SSA displays the details of the IP Office Routing Table, including static and RIP routes as well as information on configured IPSec or L2TP VPN tunnels.
– A Ping can be executed from the IP Routes screen choosing LAN1, LAN2 (WAN) or RemoteManager (DialUP) as the source interface to aid in troubleshooting connectivity issues.
System Status Application (Continued)
System
The System screen within SSA provides information on the type and state of hardware installed in the IP Office to include:
– Hardware Summary – Memory Cards – Control Unit Ports – Extension Ports – Expansion Modules – H.323 Extensions – VoIP Trunks
SSA Snapshot Feature
Include switch configuration: embeds a copy of the IP Office configuration file when
Snapshot only or Continuous log is selected.
Snapshot only captures the IP Office system status to a file with the .ssh extension which can be viewed offline at a later time.
Continuous log: records all IP Office activity to a file or series of files with the .slo
extension until logging is stopped. If logging is enabled for an extended period, new files are created based on the settings Daily at midnight or After the log reaches
Reviewing Files Offline
Using the Offline tab of the SSA Login screen provides the option to review previously saved files with the .ssh or .slo file extensions.
When using SSA to open a Continuous log file, the Replay options are available to control the log file playback.
The Manager application can be used to review the IP Office configuration embedded into the SSA log files with either the .ssh or .slo file exensions.
SSA Tracing Options
The System Status Application provides the ability to generate traces for calls,
extensions and lines / trunks:
Calls: From the Call Details screen the trace will show the changes of state for that call and events relating to both ends of the call.
Extensions: From the screen for a specific extension, the trace will show events relating to the extension including button presses. Trace information for a call from a specific
extension will provide the same detail as the Call Details screen for the same call.
Trunks: The Trunk screen will show events relating to the selected channels, such as protocol messages, plus traces of all calls associated with these channels, for as long as the calls are connected.
SSA Tracing Options (Continued)
Trace information is presented at the bottom of the screen. The Pause button and scroll bar enable viewing the information while the application continues to record new trace events.
The Resume button displays all the events recorded when a trace is paused, as well as further new events as they occur.
When a trace is displayed, the option to Print and/or Save As are available.
A trace can be saved to file either as a .txt or .csv file.
– If the trace is paused, only the information currently displayed will be saved and/or printed.
To diagnose problems with a call, it is generally best to trace the source of the call; e.g. trace the trunk for an incoming call or the extension for an outgoing call.
Using System Monitor
The Password used for System Monitor is set in the IP Office system Security Settings. If the IP Office doesn’t have a Monitor Password set, the System Password is used.
Starting System Monitor
System Monitor can be run on a PC on the same local subnet as the IP Office or from a
PC on a remote network with established bi-directional routing.
Click on Start | Programs | IP Office | System Monitor
Enter the IP address of the IP Office control unit
System Monitor Interface
The IP Office System Monitor application is used to assist in the diagnosis of problems. Through configuration of its settings it is able to display information on a specific area of an IP Office's operation. It can capture that information to log files for later analysis.
System Monitor Filters / Trace Options
The System Monitor can be set to display and log a wide range of traffic types on the IP Office. In order to avoid missing significant events due to the monitor operation it will be helpful to use the filter options.
The Filters | Trace Options menu allows for customization of the events to trace.
All IP Office installers and maintainers need to understand how to run Monitor when necessary as Avaya may request copies of Monitor traces to resolve support issues.
All Settings Options
On the bottom of the All Settings screen there are buttons to aid in the selection of options.
Additionally, if a combination of settings is configured to troubleshoot a specific issue, the settings can be exported using the Save File button for later use.
On occasion Avaya support staff may provide specific settings which can be imported using the Load File button.
System Monitor File Logging
Logging is enabled for the System Monitor application from the menu File | Log Preferences.
Log File Naming Convention
The naming convention is as follows:
NAME DATE TIME (N)
In the above example, the NAME of the trace is CallTrace1 The DATE format is: YYYYMMDD where Y represents the Year, M represents the Month and
D represents the Day of the Month
The TIME format is: HHMMSS where H represents the Hour, M represents the Minute and
S represents the Seconds
The (N) represents the Number of times the log file of the same name has ‘rolled over’ either
Event Notification
The IP Office supports a number of methods by which events occurring on the system
can be reported. These are in addition to the real-time and historical reports available
through the IP Office System Status Application (SSA).
SNMP Reporting: The IP Office SNMP client allows the IP Office system to respond to SNMP polling and to send alarm information to SNMP servers.
SMTP Email Reporting: IP Office can send alarms using an SMTP email server.
Syslog Reporting: IP Office can also send alarms and audit trail events to a Syslog server (RFC 3164) without needing to configure an SNMP server.
SNMP Reporting
Support for SNMP is enabled by entering the required information on the
System Events | Configuration tab.
– Community (Read-only) and SNMP Port
– Enabling SNMP is not required if using SMTP or Syslog.
The QOS Parameters are used if
Enable RTCP Monitor on Port 5005 is selected
(System | LAN1 | VoIP).
If the thresholds are exceeded an alarm is sent to SSA and to a destination configured on the Alarms tab.
SNMP Settings
Enter the IP address of the SNMP server (Trap Receiver)
Verify the Port setting (162 is the default transmit port)
SMTP Settings (Continued)
In order to send alarms to email, the IP Office Manager System | SMTP tab must be configured with the senders information.
Depending on the authentication requirements of the SMTP server, this may need to be a valid email address hosted by that server. Otherwise the SMTP email server may need to be configured to support SMTP relay.
Syslog Reporting
Enter the IP address of the Syslog server
02-01 Using Diagnostic Tools
Explore System Status Application and System Monitor Task 1:
1. Start the System Monitor Application
2. Place test calls and observe the real time output of each application
– Answer call
– Answer call and transfer to another extension – Allow call to go to voicemail
02-01 Using Diagnostic Tools (Continued)
Task 2:
1. Start the System Status Application
2. Place test calls and observe the real time output of each application
– Answer call
– Answer call and transfer to another extension – Allow call to go to voicemail
02-01 Using Diagnostic Tools (Continued)
Task 3:
1. Enable file logging in System Monitor and review the output of the log files 2. Place test calls and observe the real time output of each application
– Answer call
– Answer call and transfer to another extension – Allow call to go to voicemail
02-01 Using Diagnostic Tools (Continued)
Task 4:
1. Enable continuous logging in System Status Application and review the output of the log files 2. Place test calls and observe the real time output of each application
– Answer call
– Answer call and transfer to another extension – Allow call to go to voicemail
Summary
This module provided information about Avaya provided diagnostic tools.
After completing this module you should be able to:
Summarize the use of the diagnostic tools provided by Avaya for IP Office
Identify where to acquire the Avaya provided diagnostic tools
Explain the capabilities of Avaya provided diagnostic tools
Capture traces using Avaya provided diagnostic tools Avaya provides three diagnostic tools for use specifically with IP Office: Debug View, System Status Application and System Monitor.
Summary (Continued)
Links to the documents used in this Learning Module:
IP Office System Status Application IP Office System Monitor
Chapter 02
Diagnostics &
Troubleshooting
Chapter 02 Module 02
Best Practices
Module Introduction
This module provides information about using a structured approach for troubleshooting and collecting information for escalating IP Office issues.
After completing this module you will be able to:
Identify data collection requirements for escalating IP Office issues
Summarize the use of the OSI Model for troubleshooting
Describe the purpose of a network assessment
Using a structured approach when troubleshooting IP Office issues will help obtain the information required for resolution more quickly.
Best Practices Overview
Using a structured approach when troubleshooting IP Office issues will help obtain the
information required for resolution more quickly.
Identify and quantify symptoms of the issue:
– Who: individual, group, everyone
– What: hardware, application, feature, performance – When: time, day, frequency, repeatable, call volume – Where: local, remote
Additional Discovery
Determine if there is additional documentation to assist in resolving the issue:
Network assessment / baseline Network diagrams / schematics / IP address schemes
Wiring diagrams
IP Office Audit trail
System backup procedures
– IP Office backup configurations – VoiceMail Pro
IP Office Knowledgebase
For troubleshooting IP Office issues Avaya provides a number of online resources
that can be used for assistance.
The IP Office Knowledgebase is a central repository for information IP Office including:
- Technical Tips,
Bulletins and Caveats (known issues)
- Product
Documentation - Documentation
corrections
- Application Notes for integrating
Avaya Support Web Site
In addition to links to IP Office Documentation and software downloads, the Avaya Support web site hosts a searchable knowledgebase called the InSite Knowledgebase.
InSite contains articles, job aides and other information useful for troubleshooting IP office issues.
Registered users can access on-line services and additional content. (http://support.avaya.com)
Using OSI Model for Troubleshooting
The OSI Model (Open System Interconnection) provides a common framework for dividing communications system into logical segments.
Methods of using the OSI model:
Bottom up – troubleshooting by going from the physical layer (layer 1) up to the application
layer (layer 7)
Top down – troubleshooting by going from the application layer (layer 7) down to the physical
layer (layer 1)
Divide and Conquer – in this method, you start with whatever layer you feel is most likely the
cause of the problem, then move in whatever direction you feel is the more likely cause of the issue (either up or down the OSI model)
OSI Fundamentals
The segment division can be used to isolate problems. For example, testing each layer, starting from the bottom up makes it possible to isolate problems to a specific layer.
Each layer of the OSI Model relies on the layers below it in order to operate.
OSI Layer Layer Number Verify Represented by
Application 7 IP Softphone, WireShark
Applications
Presentation 6 CODECS (G.729 / G.711)
Session 5 H.323 SIP
Transport 4 Access Control TCP / UDP RTP / RTCP Ports
Network 3
ICMP Trace Route
IP Address, Subnet Mask, Default
Gateway
Data Link 2 NIC
Network Assessment
A network assessment should include a determination of the following:
A network audit to review existing equipment and evaluate its capabilities, including its ability to meet both current and planned voice and data needs.
A determination of network objectives, including the dominant traffic type, choice of technologies and setting voice quality objectives.
The assessment should leave you confident that the network will have the capacity for the foreseen data and voice traffic, and can support SIP, H323, DHCP, TFTP and jitter buffers in VoIP applications.
Network Assessment Targets
Latency: Less than 180ms for good quality. Less than 80ms for toll quality. Note that the different audio codecs used each impose a fixed delay caused by the codec conversion as follows: G711: 20ms, G723a: 80ms, G729: 40ms
Packet Loss: Less than 3% for good quality. Less than 1% for toll quality.
Jitter: Less than 20ms
Round Trip Delay: Less than 160ms is high quality. Less than 350ms is good quality.
Quality of Service
When transporting voice over low speed links it is possible for normal data packets (1500 byte packets) to prevent or delay voice packets (typically 67 or 31 bytes) from getting across the link. It is important that all routers and switches in a network to have some form of Quality of Service mechanism.
IP Office supports the DiffServ (RFC2474) QoS mechanism.
– This uses a Type of Service (ToS) field in the IP packet header. The IP Office uses this field to prioritize voice and voice signaling packets on its WAN interfaces.
Note
The system does not perform QoS for its Ethernet ports including the WAN Ethernet port on the Small Office Edition.
Escalating IP Office Issues
It is essential that all relevant information is documented and collected for escalation. This will save valuable time as the problem is escalated to the appropriate level of support.
Items required for escalation of application issues vary based on the application and the specific issue. However, some items are required for ALL escalations, including:
A detailed description of the customer issues including what services are being affected, which user actions, user numbers and user types.
A copy of the IP Office(s) configuration file(s)
The version / release of IP Office software
log files generated by the IP Office System Monitor application
Escalating IP Office Application Issues
Items required for
ALL application escalations, include:
Computer information:– CPU Type / speed
– Amount of installed memory
– Operating System & Service Pack level – Hard Drive capacity and space available – Other applications on the computer
• IP Office Applications (one-X Portal, ContactStore, etc.)
Escalating Voicemail Issues
In addition to the IP Office and server PC information, items required for escalating
voicemail issues include:
Voicemail type and version:
– Embedded – VoiceMail Pro
Copy of the VoiceMail Pro database (.mdb file)
Debug View trace(s)
Note
While not generally required for escalation purposes, System Status Application Snapshots and Continuous Logs can be very useful in isolating IP Office issues.
Escalating VoIP / IPO Network Issues
Items required for escalating network related issues include:
Network assessment / network diagram(s)– Detailed network connectivity information including connection types and bandwidth – IP address scheme of main switches and routers in the network
– QoS Settings – VLAN information
Make / model and software / firmware versions of IP phones
Number of IP phones registered to the IP Office
Verify the VCM hardware / licensing
Summary
This module provided information about using a structured approach for troubleshooting and collecting information for escalating IP Office issues.
After completing this module you should be able to:
Identify data collection requirements for escalating IP Office issues
Summarize the use of the OSI Model for troubleshooting
Describe the purpose of a network assessment
Using a structured approach when troubleshooting IP Office issues will help obtain the information required for resolution more quickly.
Summary (Continued)
Links to the documents used in this Learning Module:
Global Technical Tip 195
IP Office Caveats (internet connection is required) System H.323 Gatekeeper
(http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/mergedProjects/manager/index.ht m?gatekeeper.htm)
Chapter 02
Diagnostics &
Troubleshooting
Chapter 02 Module 03
Global Assistance Calls
Module Introduction
This module provides information about common causes of degradation in the quality of VoIP calls.
After completing this module you will be able to:
Summarize the supported voice compression standards supported by IP Office
Identify when voice compression channels are required
Describe IP Office settings that should be configured to improve IP voice quality
Overview
It is likely that any fault on a network, regardless of the cause, will be reported as a
degradation in the quality of VoIP calls. This module provides some items to consider:
End-to-End Matching Standards: VoIP depends upon the support and selection of the samevoice compression, header compression and QoS standards throughout all stages of the calls routing. The start and end points must be using the same compression methods. All
intermediate points must support DiffServ QoS.
Avoid Hubs: Hubs introduce echo and congestion points. If the customer network requires LAN connections beyond the capacity of the IP Office Unit itself, Ethernet switches should be used. Ethernet switches are recommended as they allow traffic prioritization to be implemented for VoIP devices and for other device such as the Voicemail Server PC.
Voice Compression
Calls to and from IP devices often require conversion to the audio codec format
being used by the IP device.
For IP Office systems this conversion is done by voice compression channels on a Voice Compression Module (VCM).
IP Office VCMs support the common IP audio codecs:
– G711 – G723
– G729a
For IP400 control units channels can be added by fitting IP400 Voice Compression Modules (VCMs). For the IP500 control units, channels can be added using IP500 VCM cards and licenses or IP400 Voice Compression Modules.
When VCM Channels are Used
In order to minimize quality issues with IP audio due to VCM utilization it is important to
understand when VCM channels are used.
IP device to non-IP device
– Also includes MOH, voicemail access, conference resources, etc.
IP device to IP device
– Short code confirmation, ARS camp on and account code entry tones require a voice compression channel
– If the IP devices use differing audio codecs
Note
T3 IP devices must be configured to 20ms packet size for the above conditions to apply. If left configured for 10ms packet size, a voice compression channel is needed for all tones and for non-direct media calls.
When VCM Channels are Used (Continued)
– SIP Line Call to/from Non-IP Devices – Incoming SIP Line Call to IP Device
(Voice compression channel reserved until call connected.)
T38 Fax Calls
IP Office 5.0+ supports T38 fax on SIP trunks and SIP extensions. Each T38 fax call uses a VCM channel.
– Within a Small Community Network, an T38 fax call can be converted to a call across an H323 SCN lines using the IP Office Fax Transport Supportprotocol. This conversion uses 2 VCM channels.
– In order use T38 Fax connection, the Equipment Classificationof an analog extension connected to a
VCM Configuration Options
Echo is typically generated by impedance mismatches when a signal is converted from one circuit type to another, most notably from analog to IP.
To resolve this issue, an estimated echo signal can be created from one output and then subtracted from the input to hopefully remove any echo of the output.
The System | VCM form in the Manager application allows adjustment of the operation of VCMs installed in the IP Office control unit.
Automatic Codec Preference
The setting in Manager on the System | Telephony tab is used by all IP lines and extensions where the Compression Mode has been set to Automatic Select.
The codec selected here will be the first codec the line or extension uses for codec negotiation.
Note that the G.711 codecs are treated as a pair, so if one is selected as the first preference, the other will be second in the list.
Setting Preference Order
G.729 G729(a) 8k CS-ACELP / G711 U-Law 64K / G711 A-Law 64K / G723.1 6K3 MP-MLQ G723 G723.1 6K3 MP-MLQ / G729(a) 8K CS-ACELP / G711 U-Law 64K / G711 A-Law 64K G.711 U-Law G711 U- Law 64K / G711 A-Law 64K / G729(a) 8K CS-ACELP / G723.1 6K3 MP-MLQ G711 A-Law G711 U- Law 64K / G711 U-Law 64K / G729(a) 8K CS-ACELP / G723.1 6K3 MP-MLQ
VoIP Settings for IP Endpoints / IP Lines
There are common VoIP settings on all IP extensions (H.323 / SIP) and IP Lines (H.323 / SIP / SES) for:
Compression Mode
VoIP Silence Suppression
Compression Mode
This field defines the compression mode (codec) or modes offered during call setup.
Automatic Select
– For pre-IP Office 5.0 systems, Automatic Selectuses the codecs in the order of preference G729a, G711 ALAW, G711 ULAW and G.723.1.
– If required, a specific codec can be selected.
– If negotiation for a specific codec fails during call setup, the IP Office will fallback to using
VoIP Silence Suppression
The VoIP Silence Suppression option will detect periods of silence on a call and will not send any data during those silent periods.
This feature is not used on IP lines using G.711 between IP400 and IP500 systems.
A typical voice conversation consists of 40 to 50 percent silence. Since there is not any voice going through the network for 40 percent of a voice call, some bandwidth can be saved by deploying Silence Suppression.
There is a trade-off. There is a small amount of time before the codecs detect speech activity following a period of silence which may result in the clipping of received audio.
Allow Direct Media Path
This settings controls whether H323 calls must be routed via the H323 gatekeeper (the IP Office) or can be routed alternately if possible within the network structure.
If enabled, H323 calls can take routes other than through the IP Office and removes the need for a voice compression channel.
– Both ends of the calls must support Direct Media.
If Disabled (not selected)
– On pre-4.0 calls between IP devices using the same audio codec require a voice compression channel. – On IP Office 4.0 and higher with RTP relay support, calls between devices using the same audio codec
Line | Analog Options
IP Office has configuration options to address echo generated by impedance mismatches on the Line | Analogue Options form.
The Impedance field is only available for certain system locales; Brazil, Korea and the United States.
The Quiet Line setting may be required to compensate for signal loss on long lines. (North American locales only)
The Automatic Balance Impedance Match option can be used to test the impedance of a line and display the best match resulting from the test. (North American locales only)
Summary
This module provided information about common causes of degradation in the quality of VoIP calls.
After completing this module you should be able to:
Summarize the supported voice compression standards supported by IP Office
Identify when voice compression channels are required
Describe IP Office settings that should be configured to improve IP voice quality
Summary (Continued)
Links to the documents used in this Learning Module:
Voice Compression Module settings VoIP Settings
Analog Options
