Worldwide Consulting Solutions WHITE PAPER Support and Maintenance. Operations Guide: Support and Maintenance

(1)

Operations Guide:

(2)

Introduction

When implementing Citrix environments, support and maintenance aspects for new farms often get overlooked. Effectively maintaining a Citrix environment necessitates reliable systems be in place to ensure smooth day to day operations. This document covers main duties involved in maintaining of Citrix infrastructures.

This white paper covers the following 3 sections:

 Support – When problems arise, technical support often is the first point of contact for issue resolution. This section addresses the proper staffing, organization,

training, and tools utilized in effective support organizations.

 Testing and Change Control – Regular upgrades are required to ensure a farm environment is up to date. Change management processes are critical to ensure improvements are properly approved, tested, and validated by appropriate parties. This section covers the proper processes that ensure changes in production environments are deliberate, proven, and accountable.

 Ongoing Operations – Maintenance, issue prevention and resolution are core responsibilities in running a Citrix infrastructure. When the responsibilities and assignments are structured properly, friction is kept to a minimum, reducing issues and their resolution times. This section discusses routine operations that Citrix environments require for optimal performance.

(4)

Support

Well trained support members with a clear organizational structure are critical for an effective support team. Since technical support is the first place many users go, thought needs to be given to its structure, training, and tools utilized by support members.

Structure

Multi-tier support levels have been found to be the most effective means in addressing user support issues. Low criticality, low complexity or frequently occurring issues should be managed and resolved at the lower tiers. High criticality and complex issues are escalated to more experienced architects or infrastructure owners. The diagram below outlines a

common multi-tier support structure:

The first level should resolve 75% of all issues encountered. These issues include routine problems requiring limited knowledge of the Citrix environment. Issues are quickly resolved and may also be automated, like password resets. When non-routine problems exceeding the abilities of level one‟s are encountered, they are then escalated to level two. Information on

(5)

the end user‟s problem and attempted troubleshooting steps are documented at the first level allowing level two technicians to immediately begin addressing the problem. Level two technicians should handle only 20% of the support tickets and are highly knowledgeable on the Citrix environments. Should a complex issue arise that exceeds level two‟s abilities it is escalated to the third, and final active support tier… level three. Level three issues are complicated and often mission critical requiring expert knowledge of the Citrix environment. Tier three support tickets should be uncommon, amount to no more than 5% of all support issues, and be resolved as soon as possible.

The final tier, level four, is not involved in active support of a production environment. It is focus solely on the strategic improvements for enterprise environment, testing new

technologies, planning migrations, and other high level changes.

Should support discover an issue is related to an application or underlying infrastructure the ticket is handed to the appropriate team for troubleshooting. If a program bug is discovered the issue is then re-escalated and a ticket is established with the program‟s vendor.

In parallel with the active support structure, self-service portals are available for end users to assist themselves with basic troubleshooting tasks. This assists in resolving frequently encountered issues and helps prevent the creation of new support tickets.

(6)

Support Level Description Responsibilities Skill Set

Level 1 (Call Entry)

Provide first-line support of reported issues. Initially, servicing support messages and phone calls. This level needs to perform initial issue analysis, problem definition, ticket routing, and simple issue resolution. Additionally can handle requests for application access or support with configuring plugins.

Perform issue definition, initial analysis and basic issue resolution

Perform initial troubleshooting to determine the nature of the issue

Collect basic user and session information

Create ticket and log all

troubleshooting steps performed

Resolve basic Citrix related issues, connectivity problems and application related issues using existing Wiki articles

 Escalate issue to Tier-2 if it requires advanced skills or elevated permissions

If the issue is related to particular applications or other technologies and not the Citrix infrastructure, escalate the issue to the appropriate application or technology support units

If the issue is deemed important enough, escalate directly to Tier-3 Generate requests for additional issue resolution guides as necessary

Tier-1 support personnel should be provided with basic training on Citrix XenApp, Citrix XenDesktop and supporting technologies. This can include internal training from subject matter experts or formal training. The training provided should focus on the following topics:

High-Level overview of the XenApp and XenDesktop implementation

 Using Citrix Desktop Director to manage user sessions

Troubleshooting Citrix XenApp and XenDesktop sessions

Managing AD group policies user profiles

Troubleshooting methodology In addition, regular training should be provided to the Tier-1 team members on the latest troubleshooting

recommendations from the Tier-2 and Tier-3 teams as well as details on any relevant changes to the environment. This will help ensure a base knowledge level amongst the team and consistent customer service.

(7)

Level 2 (Production Support Engineer)

Primarily supporting day-to-day operations of virtual desktops environment, may include proactive monitoring and management. In addition, this role would also perform advanced troubleshooting and utilize available monitoring / advanced troubleshooting tools. Assist with resolving issues escalated by Level One Support.

Perform intermediate issue analysis and resolution

Identify root cause of issues

Respond to server alerts and system outages

 Review vendor knowledge base articles

 Respond to out-of-hours helpdesk calls

 Respond to critical monitoring alerts

 Generate internal Wiki articles and issue resolution scripts and maintain Tier-1 troubleshooting workflows

 Perform basic server maintenance and operational procedures

Manage user profiles and data

Escalate ticket to Tier-3 or appropriate technology owner if advanced skills or elevated permissions are required Generate requests for additional issue resolution scripts and Wiki articles as necessary

All members of the Tier-2 team should achieve the Citrix Certified

Administrator (CCA) certification for both XenApp and XenDesktop. In addition, basic and advanced training on Windows concepts will be essential for team members who do not have desktop or server support experience. Finally, on-the-job training along with close integration with Tier-3

administrators is essential as the Tier-2 roles are formalized and responsibilities are handed over from Tier-3 to Tier-2. In addition, regular training should be delivered to the Tier-2 team to maintain competencies on technology and architectural updates. This will ensure a baseline knowledge level across the team.

(8)

Level 3

(Build Engineer)

Central point for implementing, administering and maintaining Citrix desktop and application virtualization infrastructure. This role focuses on deploying new use cases and leading lifecycle management initiatives. Generally, one Build Engineer could focus on one use-case at a time. For example, three new concurrent use cases would require three Build Engineers. Escalates issues to software vendor specific Technical Support and notifies Level 4 about this issue.

Perform advanced issue analysis and resolution

Perform maintenance and environment upgrades

Addressing high severity issues and service outages

Manage the Citrix environment

Oversee and lead administrative tasks at the Tier-2 level

Manage network and storage

infrastructure as it relates to the Citrix environment

 Review periodic reports of server health, resource usage, user experience, and overall environment performance

Review vendor knowledge base articles and newly released updates

Perform policy-level changes and make Active Directory updates

Review change control requests that impact the Citrix environment

Perform advanced server and infrastructure maintenance

Review Wiki articles and issue resolution scripts for accuracy, compliance, and feasibility

Create Wiki articles and issue resolution scripts to address Tier-2 requests

Tier-3 support team members with a minimum of three years experience implementing and supporting XenApp, XenDesktop, Provisioning Services and Windows operating systems.

The Tier-3 team should have enterprise experience with industry recognized certifications. The Tier-3 administrators should complete the Citrix Certified Enterprise Engineer certification track as this will prepare them to proactively manage the user community and the Citrix solution. Citrix Certified Integration Architect (CCIA) level certifications would be optimal at this level.

(9)

Level 4 (Architect)

The Level 4 team has minimal exposure to administrative tasks but focuses on translating business requirements into technical architectures, designing the

infrastructure or planning migrations. Not involved in day-to-day support.

Provide technical leadership for upcoming projects

Lead design updates and architecture revisions

Address high severity issues and service outages

Oversee technology integration workflows

Review periodic reports of server health, resource usage, user experience, and overall environment performance to determine next steps and upgrade paths

Review change control requests which impact the Citrix environment

 Review frequently recurring helpdesk issues

Ensure technical specifications continue to meet business needs Update design documentation

A qualified Tier-4 resource should have a minimum of five to six years of experience implementing, supporting, and serving in a technology architect role for a XenApp and/or XenDesktop environment as well as additional administrative experience with integrated technologies such as application and profile management solutions. The ideal candidate will have served in such a capacity at two or more environments for purposes of product exposure and in at least one

environment of over 1,200 concurrent users.

The Tier-4 resource should have enterprise experience with industry recognized certifications. A Citrix Certified Integration Architect (CCIA) certification or comparable training and experience should be a prerequisite of the role. Training or practical

knowledge on XenCenter, AD Group Policies and the utilized hypervisor should be considered essential.

Vendor Support Vendor assistance may be necessary should defects in a program be discovered. At this stage level 3 engineers need to establish

a support tickets with the appropriate vendor to assist with finding a solution.

Self Service A self-service portal is utilized for noncritical tasks such as application access, permissions, etc. The portal can range from a

simple FAQ page to a fully automated process requiring no human interaction. The purpose of the self-service portal is to add an additional touch point for end users to address basic issues, preventing the creation of new support tickets.

(10)

Support Tools

Providing the appropriate tools for the aforementioned support levels is a key aspect in order to achieve a high First-Time-Fix-Rate and a low Time-to-Resolution. The table below highlights several tools available for supporting Citrix environments.

Tools\Processes

Tools Details

Ticket Management

System Used to document customer information and issues. Help Desk should also document date and time of occurrence, the application

and module, steps to repeat error, obtains screenshots as needed.

Call Scripts The first contact help desk personnel should have documented scripts

or triage guides to ensure that all relevant data is captured while the user is on the phone. This practice also assists in proper triage and allows the next support tier to perform research off-line and review logs prior to customer contact.

Shadowing Tool Shadowing users is a useful way for troubleshooting new issues. This

allows for support technicians and administrators to remotely observe a user‟s actions.

Knowledge Base Documentation should be created and maintained in a knowledge

base or library of known issues. Articles should be searchable for quick recovery. Knowledge bases help quickly resolve issues upon reoccurrences and reduce the need to perform future research. Citrix Administrative

Consoles The incident management team will have access to the Citrix Administrative Consoles for focused troubleshooting. Citrix

recommends that the Help Desk also have same level of access. The initial support tier will primarily focus on resetting connections and obtaining data for escalation.

The diagram below outlines the recommended tools for every support level:

Tool Description Product Support Level XD XA PVS XS L1 L2 L3

Desktop

Director Desktop Director provides an overview of XenDesktop hosted

desktops and XenApp sessions. It enables support teams to perform basic maintenance tasks and to monitor and troubleshoot system issues.

X X X X

Desktop

Studio Desktop Studio enables administrators to perform

configuration as well as

(11)

Tool Description Product Support Level XD XA PVS XS L1 L2 L3

maintenance tasks for a

XenDesktop site and associated virtual desktops.

AppCenter AppCenter provides support teams

the ability to monitor and

troubleshoot user sessions as well as server systems within a XenApp farm. In addition AppCenter enables administrators to perform configuration and maintenance tasks for a XenApp farm.

X X X

HDX Monitor HDX Monitor is a tool to validate the operation of the Citrix

ICA/HDX stack of a user session. Hereby HDX Monitor provides information about client

capabilities, network performance / activity, session settings and many more items.

X X X X

Provisioning Services Console

The Provisioning Services Console enables administrators to perform configuration and maintenance tasks for a Provisioning Services farm.

X X X

XenCenter XenCenter enables administrators

to perform configuration and maintenance tasks for a XenServer Resource Pool.

X X

License Administratio n Console

The License Administration Console is the interface required to manage license files and a license server or appliance.

X X X X X

Please refer to the following appendices for details on how delegated rights could be configured for the support staff

 Appendix A (XenApp)

 Appendix B (XenDesktop)

 Appendix C (Provisioning Services)

In addition to the aforementioned Citrix support tools, every support organization should leverage the following instruments:

(12)

 Ticket Management System: It is vital for support organizations to implement a Ticket Management System for tracking and escalating support calls (in between the levels) as well as to document incident and customer related information.

 Knowledge Base / Wiki: Ever support organization should document and maintain a knowledge base or library of known issues. The articles should be searchable for quick recovery. Knowledge bases help quickly resolve issues upon reoccurrences and reduce the need to perform future research.

 Call Script: The first contact help desk personnel should have documented scripts or triage guides to ensure that all relevant data is captured while the user is on the phone. This practice also assists in proper triage and allows the next support tier to perform research off-line and review logs prior to customer contact. Please refer to Appendix E – Sample Call Script / Questionnaire for a sample call script.

Staffing Requirements

This section provides a guideline for the number of support staff required for several sizes of Citrix XenApp or XenDesktop enterprises. This information is primarily based on Citrix experience in deploying enterprise environments, taking into account the most typical design requirements. Additional refinement may be necessary to optimize the support of the environment based on specific business requirements.

XenApp centric organization

Farm or Load Managed Groups

#Servers #Users Help Desk Admins Architects

1 <100 <3000 1 CCA 1-2 CCEE 1 CCIA

2-5 <200 <4000 3 CCA 3-4 CCEE 1 CCIA

>6 >200 >5000 3 CCEE 5+ CCEE 2 CCIA

XenDesktop centric organization

XenDesktop Sites #Desktops #vDisks Help Desk Admins Architects

1 <500 1-2 1 CCA 1-2 CCEE 1 CCIA

2-3 1000-5000 3-5 3 CCA 3-4 CCEE 1 CCIA

(13)

Testing and Change Control

Regular updates and maintenance is an everyday part of Citrix environments. Standard processes must be closely followed to ensure modifications do not inadvertently damage a production

environment. This includes maintaining dedicated infrastructure where modifications are tested prior to production and processes that ensure changes are deliberate and accountable.

Testing Phases

Since changes to Citrix infrastructure can impact thousands of XenApp and XenDesktop users, multi-phase testing is critical for the reliability of the environment. As such, Citrix recommends implementing the following testing strategy:

 Development. The development infrastructure exists outside of the production network. Typically it consists of short-lived virtual machines, whose configuration should match the configuration in production as close as possible. The purpose of the development phase is to provide change requestors a non-production

environment to perform proof of concepts, determine integration requirements and perform iterative testing as part of a discovery phase to making changes. Proposed changes should be documented to they can be applied in the test phase.

 Test. The test environment is a standalone 1:1 copy of the production infrastructure and is used to confirm that the proposed changes can be easily repeated prior to the pre-production staging environment. The changes made should follow exactly from the documentation provided from discoveries made in the development stages. If testing fails within the testing stage, the architect must determine the severity of failure and determine whether minor updates to

documentation is sufficient or a full development cycle is needed.

 Pre-Production. The staging environment can be co-located with the production environment as the changes should be tightly managed. The goal of staging is to implement the proposed changes with little risk or uncertainty. It is expected that any changes made to the staging infrastructure have been tested and documented for repeatability. There should not be any iterations or adjustments required within this phase. For updates, the starting point for all systems in the staging environment should be the current production status. However for new deployments, the architect will determine the appropriate start point; whether an existing system or newly created. Suring this phase and within this environment User Acceptance Testing (UAT) should be performed.

(14)

 Production. The production environment is the fully redundant and scalable solution for normal usage of the XenDesktop and XenApp environments. There should be minimal changes to the environment.

Infrastructure

There is a need for four dedicated levels of testing that ensures changes do not harm a production farm. An isolated development environment should be used initially to try any new changes, application installations, and updates. Conflicts resulting from these changes have no impact in this environment and can be resolved before they affect any

infrastructure. Once proven, safe changes are then applied to a larger test environment. The test environment should mimic production as closely as reasonably possible. This includes all patches, applications, and policies be similar so that testing is as realistic as possible. Once changes prove stable in the testing environment it is recommended the changes not be initially rolled out to production all at once. Updates should be first applied to a targeted “pre-production” subset of users to again validate and approve of changes made. This way if any problems go undetected in the test environment the problems affect only a small subset of users. When validated in pre-production changes can be applied to the production farm. Change Management Processes

Standardized processes that manage application lifecycles through installation, updating, and end of life are necessary to ensure consistent and accountable performance of a farm. The change control process should be closely followed starting with a change request. For any changes to be made to a production environment a change request form must first be filled out detailing changes requested, reasons for the change, and intended timeframes for the action. This is then reviewed and edited if required by a change manager and then advisory board. When the change request has gone through the entire change approval process it is then given to a change implementer who then stages the change for testing, and finally conducts the implementation in production. The table and steps below further detail the process:

(15)

(16)

Procedure

1. The Request for Change (RfC) form is completed by any person requesting a change 2. After appropriate manager approvals have been acquired, the RfC is forwarded to the

appropriate Change Manager(s).

3. The Change Manager validates the RfC for completeness and logs the RfC information into the Change Control Log for tracking. Incomplete CR‟s are returned to the requestor for update and re-submission.

4. The Change Manager assesses the impact of the change in conjunction with subject matter experts and/or managers of the teams associated/affected by this change. 5. The Change Manager works with the associated/affected teams as well as the change

requestor in order to confirm the priority, category and type of the change as well as the proposed back-out plan.

6. If the process succeeds the RfC is forwarded to the Change Advisory Board for approval. If the process does not succeed the change control log is updated with the current status as well as the reason of the rejection and the RfC is send back to the requestor.

7. The Change Advisory Board reviews and validates the change in detail, and discusses and evaluates purpose, reasons, impact, cost and benefits. Each Board member

represents their department and provides guidance on the change requests. The Change Advisory Board also reviews multiple requests to coordinate implementations and “package” requests into a single release schedule.

8. Upon approval the change is send back to the Change Manager to schedule the change for implementation into the staging environment.

9. The change is implemented and test are conducted. The results are sent back to the Change Manager.

10. If the staging implementation and testing was successful the change is scheduled for production implementation. In case the staging phase was not successful another staging iteration will be conducted.

11. The change is rolled out to production.

12. The Change Manager reviews the implementation and finally updates the Change Control Log.

13. On a periodic basis, the Change Manager reviews the Change Control Log to analyze for trends on type, frequency and size of changes and forwards the results to the Change Advisory Board for review.

In the instance of an emergency the processes may be expedited. Should an issue be declared an emergency a change request form is still filled out and delivered to the appropriate change management representative. When approved the requested change is immediately

(17)

implemented and the advisory board notified. Due to the criticality of time in an emergency timeframes are not required.

Change Implementation Process

The process to implement changes in Citrix environments need to be highly structured. Standardization helps ensure reliable results while reducing risks of unexpected results to production environments. Update requests undergo a centralized approval and then must go through an engineering and tracking process as they are introduced into environment. The process begins at a development environment and progresses through testing,

pre-production, and finally full production.

Potential updates must first undergo an assessment to determine compatibility and

functionality in a development environment. This environment provides a rudimentary setup where changes are initially attempted to observed potential conflicts or performance issues. These results provide feedback on areas to be further tested while not risking more

significant environments.

Once proven and approved in the development environment further tests are conducted in a testing environment. This is larger environment that models what is used in production. Having an arrangement that closely matches production ensures testing results are accurate. All changes to this environment are document for repeatability in the production

environment, future reference, and for change control.

After rigorous testing in the test environment changes must be approved and then re-duplicated to a pre-production subset of users in the production environment. This selection of users provide additional feedback on their experiences and ensure the changes perform as expected. Should no issues be encountered by the pre-production users updates are then written off by appropriate parties and applied to the rest of the farm. The below diagram highlights the key sections in the change implementation process.

(18)

Ongoing Operations

It is best practice to keep operating systems, applications, and Citrix components all up to date with applicable hotfixes and updates to ensure stability, performance, and security of a farm. In general Citrix recommends that Hotfix Rollup Packs and Service Packs always be installed, general hotfixes be installed on an “as needed” basis only, and that all changes follow standardized change

management processes for any changes (i.e. development, testing, pre-production, and production). For more information regarding general hotfix maintenance refer to (CTX132799).

Monitoring of a new environment enables administrators to address issues proactively. By having an in depth understanding of current and expected behavior of the various components, administrators are better equipped to discover an issue before it impacts the user community.

Make sure to define the monitoring activities that administrators will perform on a periodic basis, as well as the tools that will be used. In addition, determine which reports will be made available to management.

A variety of built-in and customizable reports exist within the Citrix products so that administrators can track trends, effectively plan capacity, and address auditing requirements. In addition,

troubleshooting is typically minimized when administrators fully utilize the tools integrated into the Platinum edition, as well as other third party tools.

It is recommended that support staff and administrators involved in implementation review the product documents, articles, and guides to support, manage and administer the environment. All of these documents are available from the Citrix Product Documentation Library

(http://support.citrix.com/proddocs/index.jsp) or on the Citrix Knowledge Base (http://support.citrix.com).

The remaining sections in this section identify the maintenance tasks (daily, weekly, yearly and preventative) required to support each component of the XenDesktop Platinum edition.

XenDesktop Operations

This section covers the regular maintenance and upkeep involved in a XenDesktop environment.

Overview

Several tasks need to be performed to ensure a stable, scalable Citrix XenDesktop platform. The team responsible for supporting and managing the Citrix XenDesktop infrastructure should perform regular tasks that should be incorporated into the existing standard operating procedures. These are usually broken down into daily, weekly or yearly tasks.

(19)

Periodic Tasks Daily Periodic Tasks

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a daily basis:

Daily Tasks – Citrix XenDesktop

Event log checking Checking of the event logs on all Citrix XenDesktop (Desktop Delivery _{Controller) servers including Windows System, Application and Error logs.}

Citrix XenDesktop Alerts Checking of the Citrix XenDesktop alerts in Desktop Studio needs to be performed at least once per day to ensure no issues have occurred with the Citrix XenDesktop environment.

Check Citrix XenDesktop logs

Checking of Citrix XenDesktop specific application logs including the Pool Management Logs. To assist with troubleshooting efforts, refer to

Comprehensive List of Event Log Messages for XenDesktop on http://support.citrix.com/article/CTX119978

Test Virtual Desktop

Access Simulate a connection to ensure access to Desktop Groups (Pooled and Assigned) is available to users.

Weekly Periodic Tasks

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a weekly basis:

Weekly Tasks – Citrix XenDesktop

Review Access

Management Console Check the status of the Citrix XenDesktop Desktop Groups, Idle Pool count and connectivity to backend hosting infrastructure. Hotfixes and Patches Review the latest Citrix hotfixes and ascertain whether the XenDesktop _{Controller and virtual desktops require them.} Printer Drivers Review the local installation of printer drivers on the virtual desktops, if _{required for auto-creation.} Review Citrix Knowledge

Base for latest incidents Review incidents logged for any common issues related to Citrix XenDesktop on http://support.citrix.com/ Backup Citrix XenDesktop

Database Perform full-data backup of the Citrix XenDesktop Datastore (could be a daily task due to size of the transaction log) Backup User Data Backup user-related file data.

Yearly Periodic Tasks

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenDesktop administrators on a yearly basis:

Yearly Tasks – Citrix XenDesktop

Perform restore test Conduct backup tests to validate the actual restore process is functioning _correctly. Citrix Policy Assessment Review Citrix Policies and determine whether new policies are required and _{existing policies need to be updated.}

(20)

Yearly Tasks – Citrix XenDesktop

Software Upgrade Review and asses the requirement for new Citrix XenDesktop software _{releases/versions.} Capacity Assessment Perform capacity assessment of the Citrix XenDesktop environment.

Preventative Maintenance

As a part of your preventative maintenance routine for your Citrix XenDesktop environment, Citrix Xnapshot for XenDesktop should be run on regular three month intervals. This tool collects a wealth of information about the underlying system and its configuration, such as BIOS information, registry information, device drivers, Windows services, installed hotfixes, Citrix binaries, and Citrix XenDesktop farm information. Citrix recommends that the following key Microsoft Windows services be monitored on all of the Citrix XenDesktop (Desktop Delivery Controller) servers to ensure reliability.

Citrix XenDesktop Services to be Monitored

Citrix Desktop Delivery Controller Service Citrix Diagnostic Facility COM Server Citrix Management Server

Citrix MFCOM Service

Citrix Pool Management Service Citrix Services Manager

Citrix XTE Server IPSEC Services

Remote Procedure Call (RPC) Server

Windows Management Instrumentation Driver Extensions Workstation

Updating MCS Images

MCS images require regular maintenance throughout the image‟s lifecycle. It is standard procedure to first take a snapshot of a master image prior to any updates. New changes are made to this snapshot and it will become the new master image. The new image, following standard change management policies, is fully tested in an isolated test environment. When the new image successfully passes testing and is proven stable the image is rolled to production subset in of production users. When a new image is shown to be reliable in pre-production it can then be rolled out to the rest of the farm. Below is a diagram and steps further detailing this process.

(21)

1. When a change request is approved a snapshot is made of the master image. 2. The desired changes are made to the new snapshot.

3. Functionality and stability is of the new image is tested within an isolated test environment.

4. When validated and written off as stable in the testing environment the image is migrated to a “pre-production” catalog where IT friendly users use the new image and ensure image stability.

(22)

Hotfixes

For hotfixes, backups of affected infrastructure (i.e. database, registry, etc…) should be made. Then half of the controllers should be updated and observed for several days. Should no conflicts arise, the rest of environment may be updated.

XenApp Operations

This section covers the regular maintenance and upkeep involved in a XenApp environment Overview

Several tasks need to be performed to ensure a stable, scalable Citrix XenApp platform. The team responsible for supporting and managing the Citrix XenApp infrastructure should perform regular tasks that should be incorporated into the existing standard operating procedures. These are usually broken down into daily, weekly or yearly tasks.

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenApp administrators on a daily basis:

Delete Contents

Event log checking Checking of the event logs on all Citrix XenApp servers including Windows _{System, Application and Error logs.}

Reboot schedule checking All Citrix XenApp servers are rebooted on a regular basis as part of an automated process. Ensuring the reboot of the servers has been successful is essential to maintain server health and performance.

Citrix XenApp Alerts Checking of the Citrix XenApp server alerts in the Access Management Console needs to be performed at least once per day to ensure no issues have occurred with the XenApp environment.

Suggested daily Citrix XenApp commands to be run

 Qfarm /load - to check that the load of the Citrix XenApp servers are reporting correctly

 Qfarm /zone - to confirm that the correct Zone Data collector is assigned Qfarm /online - to confirm all Citrix XenApp servers are online

(23)

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenApp administrators on a weekly basis:

Weekly Tasks – Citrix XenApp

Administration

Configuration and Logging Reports

Confirm that Citrix XenApp farm changes taken place during the previous week have been approved.

Citrix XenApp Commands

to be run DSCheck.exe - verify that the contents of the datastore are still valid. Hotfixes and Patches Review the latest Citrix hotfixes and ascertain whether the Citrix XenApp _{farm requires them.} Review Citrix Knowledge

Base for latest incidents Review incidents logged for any common issues related to Citrix XenApp on http://support.citrix.com/ Printer Drivers Review the local installation of printer drivers on each Citrix XenApp server _{to ensure no rogue drivers have been installed}

Backup Citrix XenApp related databases

Perform full-data backups of the following Citrix XenApp databases:  Citrix XenApp Datastore

 Citrix SmartAuditor Datastore

Citrix XenApp Configuration Logging Database Backup User Data Backup user-related file data.

Backup Application

Profiles Perform full-data backup of the Citrix XenApp application profiles used for streaming.

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix XenApp administrators on a yearly basis:

Delete Contents

Perform restore test Conduct backup tests to validate the actual restore process is functioning _correctly. Citrix Policy Assessment Review Citrix Policies and determine whether new policies are required or _{existing policies need to be updated.} Perform BCP/DR test Conduct functional Business Continuity Plan (BCP)/Disaster Recovery (DR) _{test to confirm DR readiness.} Software Upgrade Assess the validity of applications. Review and asses the requirement for new _{Citrix Web Interface software releases/versions.} Capacity Assessment Perform capacity assessment of the Citrix XenApp environment to determine _{environment utilization and any scalability requirements.}

(24)

Citrix recommends that the following key Microsoft Windows services be monitored on all of the Citrix XenApp servers to ensure reliability.

Citrix XenApp Services to be Monitored

Citrix Client Network

Citrix Diagnostic Facility COM Server Citrix Encryption Service

Citrix Health Monitoring and Recovery Citrix Independent Management Architecture Citrix MFCOM Service

Citrix Print Manager Service Citrix Services Manager Citrix SMA Service Citrix Streaming Service Citrix WMI Service Citrix XTE Server Print Spooler Server

Remote Desktop (Terminal) Services Citrix Systems Monitoring Agent

Application Updates\Installs

1. When an application change request is approved the application is tested in a development environment to observe its performance and resource consumption 2. Once written off that the application is safe to install and use it is included in the test

(25)

3. Upon satisfying testing requirements the changes are then duplicated in the

production farm but assigned only to a specific “pre-production” subset of users for further validation testing.

4. When proven in pre-production changes then may be signed off on and rolled out to the rest of the users.

For additional information on installing and publishing applications in XenApp please refer to Citrix‟s eDocs - Publish a resource using the Publish Application wizard.

Hotfixes

The general hotfix recommendations still standing XenApp hotfix updates should be first applied to the Zone Data Collector (ZDC), backups of the ZDC, the XML broker, and then the rest of member servers in this order. This ensures against accidental elections or

undesired farm reactions from occurring. For more info reference CTX120842 Printing

It is recommended that environments leverage Citrix‟s Universal Print driver. In the instance additional printer drivers are still required the driver should first be installed in an isolated test environment. Be sure to ensure to test the printer when it is out of paper, offline, as well as to stress test the printer leveraging the Citrix StressPrinters 1.3 application (CTX109374)

PVS Operations

This section covers basic maintenance involved in a Provisioning Server environment. Overview

Several tasks need to be performed to ensure a stable, scalable Citrix Provisioning Services platform.

The team responsible for supporting and managing the Citrix Provisioning Services infrastructure should perform regular tasks that should be incorporated into the existing standard operating procedures framework. These are usually broken down into daily, weekly or yearly tasks.

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix Provisioning Services administrators on a daily basis:

(26)

Daily Tasks – Citrix Provisioning Services

Event log checking Checking of the event logs on all Citrix Provisioning Servers including _{Windows System, Application and Error logs.}

Check Citrix Provisioning Server logs

Checking of Citrix Provisioning Server specific application logs including the following:

 Citrix Provisioning Server logs - %APPDATA%\Citrix\Provisioning Services\logs

Reference of PXE Error Codes -

http://support.citrix.com/article/CTX120956 Check Citrix Provisioning

Server utilization Check the number of target devices connected to the Citrix Provisioning Servers and balance the load across the servers, if required.

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix Provisioning Services administrators on a weekly basis:

Weekly Tasks – Citrix Provisioning Services

Backup vDisk Image Files Backup of Citrix Provisioning Server vDisk files (.VHD) on local and/or _{shared storage.} Backup Citrix Provisioning

Server Datastore Backup of Citrix Provisioning Server database hosted on SQL Server infrastructure.

Hotfixes and Patches Review the latest Citrix hotfixes and ascertain whether the Citrix Provisioning _{Server farm requires them.} Review Citrix Knowledge

Base for latest incidents Review incidents logged for any common issues related to Citrix Provisioning Server on http://support.citrix.com/

Perform vDisk Updates

Update the „golden‟ vDisk image files and apply the following:  Windows software updates and patches

 Operating system and application changes  Anti-virus pattern and definitions updates

Deploy updated vDisk version to target devices post update process Check Storage Capacity Review the storage utilization – used and free storage space for vDisks and _{write cache files.} Check Auditing Reports Review the Citrix Provisioning Server Auditing Logs.

Check Network and

Streaming Services Confirm DHCP, TFTP (or PXE, if applicable) and Streaming services are functional.

The following table outlines the necessary tasks that should be undertaken by the appropriate Citrix Provisioning Services administrators on a yearly basis:

Yearly Tasks – Citrix Provisioning Services

(27)

Yearly Tasks – Citrix Provisioning Services

Update vDisk Workloads Update the vDisk desktop and/or server workloads (.VHD files) to reflect the _{current standard operating environment, if required.} Perform restore test Conduct backup tests to validate the actual restore process is functioning _correctly. Perform BCP/DR test Conduct functional Business Continuity Plan (BCP)/Disaster Recovery (DR) _{test to confirm DR readiness.} Software Upgrade Review and asses the requirement for new Citrix Provisioning Server software _{releases/versions.}

Capacity Assessment Perform capacity assessment of the Citrix Provisioning Server environment to determine environment utilization and scalability requirements including server, storage and network components.

Citrix recommends that the following key Microsoft Windows services be monitored on all of the Citrix Provisioning Services servers to ensure reliability.

Citrix Provisioning Services to be Monitored

Citrix PVS Soap Server Citrix PVS Stream Service Citrix PVS TFTP Service

Citrix PVS Two-Stage Boot Service

HTTP SSL (This is a service dependency for Citrix PVS Soap Server service) DHCP Service (Located on Microsoft DHCP Server/s)

(28)

vDisk Updates

Due to infrastructure requirements for Provisioning Services farms utilize a slightly modified testing\updating environment arrangement. Instead of isolated environments testing and production share several key infrastructure components. Each environment has its own dedicated XenDesktop farm and collections but share networks and Active Directory (utilizing differing organizational unit structures and group policies) with one another. Testing, pre-production, and production devices are assigned separate collections but all leverage the same vDisk. In this setup a master device gets maintained that vDisks are created from. These vDisks are assigned to the various device collections as the testing flow progresses. These differences aside general updating best practices still apply in the updating of vDisks.

For high level updates the change process utilizes Provisioning Services‟ versioning capability. For incremental changes such as hotfixes and patches a new vDisk version is created which the changes are applied to. These updates can also be automated utilizing PVS‟s automated vDisk update process (Citrix eDocs). It is recommended that any changes be applied to new versions are replicated on the master device for consistency.

More significant changes such as updating XenTools, NIC drivers, etc. require that a new master vDisk be captured and tested. The steps below walk through the process:

1. Following standard procedures approved changes are initially attempted within an isolated development environment.

2. Once approved these updates are replicated into a new vDisk within the test environment for further validation.

(29)

3. After successful testing and approval, the vDisk is then assigned to a “Pre-Production” subset of users for final validation testing prior to implementation. 4. When shown to be stable, this new vDisk is assigned to the rest of the production

(30)

Appendix

Appendix A –XenApp Delegated Rights

Console Permission Tier-1 Tier-2 Tier-3 Tier-4

Farm Node

Farm Management √

Edit All Other Farm Settings √

Edit Configuration Logging

Settings √

Edit Zone Settings √

View Farm Management √ √

Administrators Node

Administrators √

Edit centrally configured

Web Interface sites √

Log on to Management

Console √ √ √ √

View Administrators √ √

Applications Node

Published Applications √ √

Publish Applications and

Edit Properties √ √

View Published Applications

and Content √ √ √ Servers √ √ √ Terminate Processes √ √ √ Sessions √ √ √ Connect Sessions √ √ √ Disconnect Users √ √ √

Log Off Users √ √ √

Reset Sessions √ √ √

Send Messages √ √ √

View Session Management √ √ √ √

Load Evaluators Node

Load Manager √

Assign Load Evaluators √

Edit Load Evaluators √

View Load Evaluators √ √

Load Balancing Policies Node

Load Balancing Policies √

Edit Load Balancing Policies √

(31)

Console Permission Tier-1 Tier-2 Tier-3 Tier-4

Policies Node

Policies √

Edit Policies √

View Policies √ √

Printer Management Node

Printers √

Edit Printer Drivers √

Replicate Printer Drivers √

View Printers and Printer

Drivers √ √ Servers Node Published Applications √ Assign Applications to Servers √ √ Servers √

Edit Other Server Settings √

Move and Remove Servers √

Terminate Processes √

View Server Information √ √

Sessions √

Connect Sessions √

Disconnect Users √

Log Off Users √

Reset Sessions √

Send Messages √

View Session Management √ √

Worker Groups Assign Applications to

Worker Groups √

View Worker Groups √ √

For further information about delegated rights within a XenApp Farm, please refer to eDocs - Delegating Tasks to Custom Administrators.

(32)

Appendix B – XenDesktop Delegated Rights

Admin Role Support Tier

Help Desk Administrator Tier-1 Assignment Administrator Tier-2

Full Administrator Tier-3

Read-Only Administrator Tier-4

For further information about delegated rights within a XenDesktop Site, please refer to eDocs – XenDesktop Delegated Administration.

Appendix C – Provisioning Services Delegated Rights

- Tier-1

Device Operators Tier-2

Farm Administrator Tier-3

Farm Administrator Tier-4

For further information about delegated rights within a Provisioning Services Farm, please refer to eDocs – Provisioning Services Managing Administrative Roles.

Appendix D – XenServer Delegated Rights

- Tier-1

Virtual Machine Operator Tier-2

Pool Administrator Tier-3

Read-Only Tier-4

For further information about delegated rights within a Provisioning Services Farm, please refer to CTX130420 – XenServer 6.0 Administrators Guide (see chapter Role Based Access Control).

(33)

Appendix E – Sample Call Script/Questionnaire

The following tables provide sample call scripts / questionnaires which should be used by call entry employees in order to capture all information required to solve an incident. Items marked grey should be captured for all incoming calls. All other information should be captured in case the incident needs to be escalated to a higher support level.

Scripts

Steps Details

1 Name of Customer, particular site or location if possible. 2 Name of User, department, Logon ID and callback number.

3 Do any other users at the site experience the same issue? Can they have a colleague logon from same and/or different workstation? Helps to determine whether this is a workstation issue vs. user account issue.

4 Identify workstation name or Inventory ID. 5 Name of Application attempting to access.

6 Review high level steps taken to authenticate. This validates that steps are performed correctly.

7 Can user see the Web Interface or authentication page? Can users see other web pages, such as Yahoo.com? This helps to identify and troubleshoot network issues. 8 Does user see the appropriate icon? Helps to troubleshoot user access and group

memberships.

9 Does application launch when icon is selected? Does the application logon screen appear? Helps to determine if a connection is made into Citrix architecture. 10 Can user authenticate into the application? Does the issue occur after application

authentication? Helps determine if this is a Citrix infrastructure vs. Application Issue.

11 What is the application error(s)? Provide a screenshot.

Question Answer P erso nal Inf or m ati on Name Organization

Contact Number /Email

Contact Preference (Phone/Mail) Resident Office

Current Location Current Time Zone

Incid

ent

Severity of the incident _{# of affected users} Name of affected application

(34)

Question Answer Envir onm ent Client name Client IP

Client hardware type Client OS

Version Citrix Receiver

Name of XenApp Server / virtual desktop

(35)

Product Versions

Product Version XenDesktop 5.0 / 5.5 / 5.6 XenApp 5.0 / 6.0 / 6.5 Provisioning Services 6.x

Revision History

Revision Change Description Updated By Date

1.0 Finalized Document Thomas Berger, Charles Wright May 30, 2012

About Citrix

Citrix Systems, Inc. (NASDAQ:CTXS) is a leading provider of virtual computing solutions that help companies deliver IT as an on-demand service. Founded in 1989, Citrix combines virtualization, networking, and cloud computing technologies into a full portfolio of products that enable virtual work styles for users and virtual datacenters for IT. More than 230,000 organizations worldwide rely on Citrix to help them build simpler and more cost-effective IT environments. Citrix partners with over 10,000 companies in more than 100 countries. Annual revenue in 2011 was $2.20 billion. ©2012 Citrix Systems, Inc. All rights reserved. Citrix®, Access Gateway™, Branch Repeater™, Citrix Repeater™, HDX™, XenServer™, XenApp™, XenDesktop™ and Citrix Delivery Center™ are trademarks of Citrix Systems, Inc. and/or one or more of its subsidiaries, and may be registered in the United States Patent and Trademark Office and in other countries. All other trademarks and registered trademarks are property of their respective owners.