Troubleshooting Citrix MetaFrame
Procedures
Document name
Troubleshooting a Citrix MetaFrame environment v1.0.doc
Author
Marcel van As
Last Revision Date
28 February 2006
Index
Index ... 2
Introduction ... 3
Intended Audience ... 3
Basic flow for troubleshooting... 4
Troubleshoot Secure Gateway ... 5
Troubleshoot Server and Services ... 7
Troubleshoot Citrix Services... 9
Troubleshoot Terminal Services... 12
Troubleshoot printing problems ... 14
Troubleshoot ICA Session Performance ... 16
Troubleshoot Server Performance ... 18
Troubleshoot Session Sharing Problems ... 20
Introduction
Trying to track down and resolve technical problems on a Citrix MetaFrame Presentation Server environment could be one of the most frustrating experiences. To help you through the process, this document explains the necessary steps you need to do for effective troubleshooting on a Citrix environment. In addition, this document addresses some common problems.
This document is version 1.0, so if you have questions and/or comments then please send them to: [email protected]
Intended Audience
This procedure is intended for system engineers or administrators responsible for maintaining and supporting a Citrix MetaFrame Presentation Server environment.
The described procedures are based on the assumption that you have: • An understanding of server based computing concepts
• Knowledge of Citrix MetaFrame Presentation Server • Demonstrated knowledge of at least Windows 2000 Server
Basic flow for troubleshooting
The basic flowchart for troubleshooting a Citrix MetaFrame environment and Citrix Secure Gateway is shown in Figure 1 below. Each of the circled numbers is described on the pages after the flowchart. All actions highlighted in blue are described in other paragraphs.
Troubleshoot Secure Gateway
The flowchart for troubleshooting the Citrix Secure Gateway is shown in Figure 2 below. Each of the circled numbers is described on the pages after the flowchart.
1. CSG service running - The secure gateway runs as a service “Secure Gateway Service”. Launch the services MMC on the corresponding server and check if the “Secure Gateway Service” is running.
2. Start CSG service - Through the services MMC the “Secure Gateway Service” must be start.
3. Start CSG diagnostics - Launch the Citrix Secure Gateway diagnostics tool. Watch the output to find the problem area.
4. Reinstall or renew certificates - When the CSG diagnostics identifies the SSL Certificates are invalid reinstall the certificates or renew the certificated when they are expired. In case private certificates are used verify the root certificate is installed and not expired.
5. Check if the IIS service is running - When the IIS service is not running the Web Interface is not accessible and the logon agent is not working. Launch the IIS manager MMC and check to see if the IIS service is running and if all the websites, particularly the Web Interface, are online.
6. Start IIS service - Start the IIS services if not running. Enable the websites of not running within IIS.
7. Check if the XML services are running - Check all the MetaFrame servers to see if the Citrix XML Service is running
8. Start XML Service - Start the Citrix XML Service if not running on the Citrix MetaFrame server. Start the IIS service and enable the website (default website) containing the Web Interface.
9. Launch the WIAdmin - By launching the Citrix Web Interface Console we can view and or change the secure gateway settings and Web Interface settings. 10. Consult IIS specialist - If you reach this state you have IIS running but the
website is not functional. This could be caused by port conflicts or other faulty configured IIS setting. This document does not provide information for troubleshooting IIS. Consult Microsoft or IIS Specialist.
11. Reconfigure the Web Interface - Reconfigure the Web Interface farm settings and provide the correct Citrix servers for the farm. Reconfigure the Web Interface server side firewall settings and provide the correct secure gateway server and the STA servers.
12. Check if IIS is running on the STA servers - Launch the IIS manager MMC and check to see if the IIS service is running and if all the websites are online.
13. Start IIS Service - Start the IIS services if not running on the STA servers. Enable the websites if not running within IIS
14. Check if IIS Script path is accessible - Check if the default website containing the scripts path and if its accessible and has scripts only execute rights.
15. Set ACL’s - Set ACL’s on the scripts path and set scripts only execute permissions.
16. Consult Citrix specialist - Consult Citrix or a Citrix specialist for troubleshooting the secure gateway configuration.
Troubleshoot Server and Services
The flowchart for troubleshooting the server and standard services is shown in Figure 3 below. Each of the circled numbers is described on the pages after the flowchart. All actions highlighted in blue are described in other paragraphs.
Yes Start Server Online Check server & services Power on server No 1 Server accesible Check Network No All services running Yes
Start Services Success
Logon using RDP
Yes
Success Check Citrix Services
2 3 Logon on console Failed Check Terminal Services Success Administrative logon Failed Success Join Domain Failed Citrix related service Failed Yes Finished Member of Domain Rebuild server Failed No Check Eventlog Known Errors 4 5 6 7 9 10 11 12 13 Fix errors 8 Debug logon proces Unknown Errors Success No Failed Success
1. Server online – Check if server is powered on.
2. Server accessible - Check if server is online in network.
3. Check Network - Check if network cables are plugged in. Check TCP/IP settings and change if necessary.
4. All services running - Use services MMC to check if all services are running. 5. Logon using RDP - Logon as a user to check if it is a Citrix problem or not. 6. Logon on Console - Logon as a normal user on the console.
7. Administrative logon - Logon as a administrator on the console
8. Debug logon process - Set the following registry key on the server to debug the logon process:
Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon Value: UserEnvDebugLevel
Type: REG_DWORD Data: 10002 (Hex)
The data value of 10002 will enable verbose logging to a file on the server. Once you set this value, reboot your server and check for a “userenv.log” file in the %System Root%\Debug\UserMode\ folder. Remember to turn this off when you’re done troubleshooting it, since each user logon can easily add 100KB to the size of this log file. 9. Member of domain - Check if the server is still a member of the domain and if the
server can successfully contact the domain controllers. 10. Start Services - Start the services which are not started
11. Citrix Related Service - Is the failing service a Citrix Related Service such as the Citrix XTE server or the Independent Management Architecture
12. Check Event Log - Check the Event Log for warning or error messages. Lookup the Event ID’s to check if it’s a known error.
13. Fix error - If the error message is a known error fix it according the given resolution.
Troubleshoot Citrix Services
The flowchart for troubleshooting the Citrix Services is shown in Figure 4 below. Each of the circled numbers is described on the pages after the flowchart.
1. Application or server problem - Find out if it’s a server or application problem. If it’s a server problem no Citrix application will start. If it’s an application problem some applications can start with Citrix.
2. IMA Service running - Check with the services MMC if the service Independent Management Architecture is running and no errors for the IMA service exists in the server Event Log.
3. Restart IMA service - If the service Independent Management Architecture is not running start the service
4. IMA directory in path - Check the PATH environment to see if the directories <ProgramFiles>\Citrix\System32
–and-
<ProgramFiles>\Citrix\System32\IMA\Subsystems are in the path Set PATH
Change the path environment and add the directories <ProgramFiles>\Citrix\System32
-and-
<ProgramFiles>\Citrix\System32\IMA\Subsystems to the path environment 5. Datastore online - Check if the data store is online. Start the Citrix
Management Console and on the Resource Manager Watcher Tab check if Data store Connection Failure is present and in error state. Or check this from the MS SQL Enterprise Manager and see if the CITRIXIMA database is online. 6. Bring data store online - Bring the data store online by restoring the
database or rebuild the entire SQL server. As a temporary option fail over to the replicated database.
To use the replicated database edit <ProgramFiles>\Citrix\Independent Management Architecture\MF20.dsn and change the server entry to the server hosting the replicated database.
Execute the following command:
dsmaint config /user:sa /pwd:Citrix
/dsn:”<ProgramFiles>\Citrix\Independent Management Architecture\MF20.dsn”
7. Datastore online - Clean the localhost cache by executing the command: dsmaint /recreatelhc
Start the IMA service
8. Config IMA DSN - From within the ODBC administrator create a new file DSN with the connection to the CITRIXIMA database.
Execute the following command:
dsmaint config /user:sa /pwd:Citrix /dsn:”<path_to new_dsn_file>\<new_dsn_file>”
9. ICA Port enabled - Telnet to port 1494 on the server and check if the ICA port is enabled and has a listener on it.
10. Enable ICA Port - Start the Citrix connection configuration and enable the ICA-TPC protocol
11. DSCheck - Run dscheck.exe from the command prompt and watch for errors. 12. DSCheck /Clean - Run dscheck.exe /clean from the command prompt to
clean the data store.
13. Restore data store - Restore the data store from backup
14. Check Published Application - Check if the published application is enabled and the application location and startup options are correct.
15. Install / Repair Application - From the add/remove software control panel repair the application. Or (re)install the application from within the Citrix Installation Manager
16. Check application client options - Check the application client options from within the Citrix management console. Check if the client options like screen, colors and encryption are correct and the same as the other applications. 17. Set application client options - Set the application client options according
the standard 640x480@16bit with basic encryption and sound enabled.
Troubleshoot Terminal Services
The flowchart for troubleshooting the terminal services is shown in Figure 5 below. Each of the circled numbers is described on the pages after the flowchart.
1. Logon Enabled - Check if logons are enabled by executing the command “change logon /query” or from the MetaFrame settings of the server properties within the Citrix management console.
2. Enable Logons - Enable logons by executing the command “change logon /enabled” or by checking the option “enable logons to this server” from the MetaFrame settings of the server properties within the Citrix management console.
3. User TS Enabled - Check if the user has the properties “Allow logon to terminal server” enabled in active directory.
4. Enable Allow TS logon - Enable the options “Allow logon to terminal server” for the user in Active Directory.
5. Application Mode Enabled - Launch the Terminal Services Configuration MMC and check if Terminal Server Mode is set to Application Server.
6. Enable Application Mode - Launch the add/remove software control panel. Click the Add/Remove Windows Components button and click next. Change the Terminal Services option to Application server mode.
7. TS Licenses Available - Check if a Terminal Server License Server is available, activated and licenses are installed.
8. Install Licenses - Install a Terminal Server License Server and install the Terminal Server Licenses.
9. Administrative Logon - Try to logon as a administrator using a RDP session 10. Debug usrlogon process Set the following registry key on the server to debug
the logon process:
Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon Value: UserEnvDebugLevel
Type: REG_DWORD Data: 10002 (Hex)
The data value of 10002 will enable verbose logging to a file on the server. Once you set this value, reboot your server and check for a “userenv.log” file in the %System-Root%\Debug\UserMode\ folder.
Remember to turn this off when you’re done troubleshooting it, since each user logon can easily add 100KB to the size of this log file.
11. Check Event Log - Check the Event Log for warning or error messages. Lookup the Event ID’s to check if it’s a known error.
12. Fix errors - If the error message is a known error fix it according the given resolution.
Troubleshoot printing problems
The flowchart for troubleshooting printing problems is shown in Figure 6 below. Each of the circled numbers is described on the pages after the flowchart.
1. Printers available on Client - Are the desired printers and drivers installed on the client.
2. Install Printers on Client - Install the printers and drivers on the client machine. 3. Client Printer Mapping Enabled - Check in the “Citrix Connection Configuration”
Client Settings if the windows client printer mapping is enabled.
4. Enable Client Printer Mapping - Enable the Windows Client printer mapping in the “Citrix Connection Configuration” tool.
5. Events 1106 and 1107 in server Event Log - Check the Application Event Log on the Citrix servers for 1106 and 1107 errors from source MetaFrameEvents and category Printer Management. The event description will tell which printer failed for auto-creation and what driver should be used.
6. Printer Driver installed on Server - In the Citrix Management Console - Printer Management - Drivers, check whether the driver is available in the farm and installed on all servers.
7. Install Printer Driver on Server - If the Printer Driver is available in the Citrix Farm on a server select the server with the driver and replicate the driver to the servers with the missing driver. Depending on the load and number of servers this can take up a few hours before replication is completed.
If the driver is not in the Farm install the Windows 2000 version, NOT a version-2 (NT) driver, of the printer driver on a server and replicate the driver from that server to the other servers.
8. Create Printer Mapping - When driver names are not exactly the same on the client and the server, printer creation will fail. In the Citrix Management Console create a printer mapping with the client driver name with the corresponding server driver.
Open the 1106 and 1107 error in the application Event Log and copy the client driver name. Paste the client driver name in the Citrix management console and select the corresponding or compatible server driver.
Examples:
Client: HP Laserjet 4000 PCL Server: HP Laserjet 4000 Series PCL Client: HP Laserjet 4/4M Server: HP Laserjet 4
Client: HP Laserjet 1100 Server: HP Laserjet Series II Client: Lexmark 2205 Server: HP Laserjet 4
Note: If you are receiving a lot of 1106 and 1107 errors then you might be interested in
DABCC Advanced Print Manager (www.dabcc.com/apm)
Troubleshoot ICA Session Performance
1. Session Active - Start the Citrix Connection Center by double clicking on the ICA symbol in the notification area (clock). Check if ICA session is still active.
2. Reconnect Session - Reconnect the session by launching the application again or from the Citrix Web Interface with the reconnect button (if available).
3. Noticeable Traffic - Check the properties of the ICA session. Check if the session generates data to and from the Citrix server.
4. Network Active - Check if the network is still active. Refresh application on the Web Interface. Log off from the Web Interface and logon in the Web Interface again.
5. Normal Resource Usage - Application looks unresponsive. Check if application consumes too many resources like memory and CPU or causing the CPU queue to fill up. If abnormal resource usage is detected logoff or kill user process on Citrix server.
6. Force logoff after 15min- Disconnect the users session and try reconnecting. If the application is still unresponsive, let the application run for 15 minutes. Watch resource usage of the application during this period. If the application is still unresponsive, logoff the user session or kill the user process.
7. Fix Network - Take appropriate actions to repair the network.
8. Kill Application - Kill the application if it is using too many resources which can affect other users performance. Try logging of the user session first, otherwise kill the application or reset the session.
9. Launch new application - Launch a new application and watch if it is launched within the same session or a new session is created.
10. Original app unresponsive - If the original application is unresponsive a new session is created by launching a new application. Profile corruption, networking issues, database timeouts and macro’s (like VBA script) are often the cause of hanging or unresponsive applications.
11. Force logoff after 45 minutes - If normal resource usage is normal, let the unresponsive application run for least 45 minutes. Watch resource usage of the application during this period. If the application is still unresponsive, logoff the user session or kill the user process.
12. New App normal Speed - Is the new application running at expected or reasonable speed.
13. Original App problem - The original application is having a problem. The problem is often caused by networking issues, database timeouts or internal application faults. Make note of the last actions made in the application like selections and queries.
14. Contact application support - Contact application support to troubleshoot the application.
15. Other users affected - Do other users have the same problem ? If so, make note of the other users and contact Citrix administrators / Citrix support to troubleshoot server performance.
16. Shadow Session - If possible try to shadow the users session.
17. Expected behavior restored - Is the expected behavior of the application restored by shadowing the users session?
18. Reduce color quality - Does reducing the color quality (256 or 16 colors) on the users desktop increase the session performance?
19. Increase Session Memory / Update ICA Client - Increase session memory for the Citrix farm in the Citrix management console. If possible update the ICA client to at least a version 8 Client. If the problem happens often with the same application, the application is suffering on a GDI memory leak.
Troubleshoot Server Performance
1. Monitor Server Performance - Monitor the following performance counters
Object Counter Instance Description
Terminal Services Active Sessions Total user session connected Process Working Set _ Total Real physical memory used
Memory Pages Input/Sec Memory read from pagefile (1 page = 4kB) Memory Pages Output/Sec Memroy written to pagefile (1 page = 4kB) Memory Available MBytes Estimate memory free
Memory Free System Page Table Entries Free System PTE’s Paging File %Usage _Total Pagefile usage Processor % Processor Time All
Instances CPU usage
System Processor Queue Length Number of processes waiting
Cache Copy Read Hits % Memory efficiency of system file cache
2. Run Away User Processes - Check with the task manager for run away processes. 3. Kill Process - Kill the run away process by logging of the user holding the process or
by terminating the process.
4. Memory Usage / User process - Check with the task manager the memory usage and VM size per process.
5. Kill Process - Kill the process if it is consuming too many memory and causing excessive paging.
6. Memory Efficiency - Check the performance counter Cache | Copy Read Hits %. Constant values below 90% indicates a problem with kernel memory.
7. CPU Queue - With performance monitor measure System | Processor Queue Length. This counter represent the number of processes waiting to be processed. A rule of thumb is a maximum of 2 processes per processor.
8. Reduce Processes - Reduce the number of processes by stopping some services or by logging of users from the system.
9. Network Queue - Check the Output Queue Length of the network interface with the performance monitor. If the output queue length is any value above 0 it indicates a problem with the network or network interface.
10. Troubleshoot Network - Troubleshoot the network connections, network load, network interfaces or other network components.
11. Disk Queue - Check the performance counter LogicalDisk | Current Disk Queue Length.
12. Troubleshoot Disk Subsystem - Check the server for failing hard drives. Put pagefile on separate hard drive with it’s own scsi or raid controller.
13. 3rd Level Support - Escalate the problem to a 3rd level support group. 14. Restart Spooler - Restart the spooler service to free up some memory.
15. Stop Kernel Processes -Stop some kernel processes by stopping some services like NetIQ, SNMP, HP/Compaq monitoring agents. This will free up some kernel memory and creates a shorter process list.
16. Reduce Users - Logoff some of the users or add more servers. This will increase the performance.
Troubleshoot Session Sharing Problems
Yes Start Session active and responsive Session Sharing Set Application Appearance No 1 Publish on Server 2 Troubleshoot ICA Session Performance Application on Same Server No 3 Yes Same Application Appearance 4 No 5 Yes Same Client Options 6 Set Client Options No 7 Yes Update ICA Client 8 < 8 ICA Client Version 3rd Level Support => 8 9 101. Session active and responsive - Is the first session still active and responsive?
2. Application on Same Server - Is the new application installed and published on the same server?
3. Publish on Server - If possible Install and Publish the application on the other servers. 4. Same Application Appearance - In the Citrix Management Console check if the
applications have the same application appearance.
5. Set Application Appearance - Set the application appearance the same for all applications.
6. Same Client Options - Check the client options in the Citrix Management Console. Sound and Encryption options should be set the same for all applications.
7. Set Client Options - On the client options on each published application. Set the sound and encryption options the same for all applications.
8. ICA Client Version - Check the ICA Client Version on the users desktop. There should be at least version 8 installed.
9. Update ICA Client - Update the ICA Client to at least version 8 to improve the ICA session sharing feature.
10. 3rd Level Support - Escalate the problem to a 3rd level support group.
