Migration of File Servers to NAS and Multi-Tiered Storage
EMC Proven™ Professional Knowledge Sharing
May, 2007
Bryan Horton
Senior Systems Engineer
A Healthcare Services Provider
Table of Contents Table of Contents...2 Executive Summary...3 Introduction...3 Business Rationale...3 Systems Analysis...4 Beginning Environment...4 Goals...5 Ending Environment...5 Migration...5 Provisioning...6 Security...6
Backups and Restores...6
Communication...7 DFS...7 Home Drives...7 Quotas...7 CIFS Migrations...8 NFS Migrations...8 OS Data Sharing...8 Archiving...9 Data Shaping...9 Future Plans...9 Conclusion...10
What Went Well...10
Problems, Limitations, and Cures...11
Final Thoughts...11
Appendix 1: Final NAS Configuration Diagram...12
Disclaimer: The views, processes or methodologies published in this article are those of the author. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
Executive Summary
Businesses require a robust file server environment that is dynamically expandable,
dependable, easily maintained, and flexible. When businesses outgrow their current file servers and are concerned about regulatory compliance, it is time to upgrade the infrastructure. This article describes how we migrated Windows and UNIX file servers from many servers using a single tier of storage to a Network Attached Storage (NAS) based solution utilizing a multi-tiered environment.
Introduction
Many companies use open system servers running various flavors of UNIX and Windows to host file shares and NFS exports. With today’s data storage requirements, this type of file server environment can very quickly grow to be large, complex, and unwieldy. Our company needed a better solution. After evaluating alternatives, we chose to grow our NAS environment and introduce tiered storage.
Business Rationale
Our company is one of the United States’ largest healthcare services providers, employing more than 37,000 people. The nationwide IT infrastructure consists of thousands of Windows and UNIX desktops and servers. The total external storage environment is hundreds of terabytes, including SAN, NAS, and CAS storage.
We analyzed our file server infrastructure environment, and exposed several potential problems:
• Many servers had single connections to the SAN
• None of the servers had automated failover designs
• All external storage for the file servers was on the most expensive tier of disks
• Some servers had slower connections to the LAN
• Backups were not completing within the backup window
• Several servers had storage over-allocated
• Certain servers seemed always to be out of space
• The environment was difficult to expand or grow
• Several third party solutions were in use
• Inconsistent workarounds were in place to keep pace with changing demands
• Documentation was inconsistent and suspect
We discovered numerous ways to save money, increase service levels, and enhance compliance with industry regulations by reviewing our file server environment and storage assets.
Our planned solution had to provide:
• Transparent storage across different storage subsystems
• Decreased storage cost by using less expensive disks where feasible
• High availability in the form of dual paths and failover
• Faster throughput and file serving capability
• Faster backups and more backup options
• Interoperability of CIFS and NFS clients
• Centralized file server management
• Increased control of file server content
• The ability to set mandatory retention times
• Better storage utilization
• The ability to manage future growth
• Additional disaster recovery options
• Elimination of third-party software
• Reduced file server infrastructure complexity
• Future technology options
With storage growth and increased file server demands projected in the short and long terms, we decided to upgrade. We already had many of the building blocks in place for this
infrastructure, but needed to acquire software and increase hardware capacities. The next step was to identify what we had and what we needed.
Systems Analysis
Our SAN environment consists of EMC® Symmetrix® DMX-3000 and CLARiiON® CX700
storage. A combination of mirrored, RAID5, fiber channel, and ATA disks are within these
arrays. The SAN fabrics consist of McData ED-140 Directors in a Connectrix® cabinet and
several McData DS-24 switches. EMC Celerra® is our NAS system, and EMC Centera® is our
CAS system. We have two EMC CLARiiON Disk Libraries and a Sun StorageTek 8500 Tape Library Unit connected to the SAN.
We used several tiers of storage in our environment. Tier 1 storage is DMX mirrored disks, and is reserved for the applications requiring the best performance. Tier 2 is DMX RAID5 disks. Tier 3 and Tier 4 are CLARiiON fiber channel and ATA RAID5 disks. Tier 5 is CAS storage in the EMC Centera. The CLARiiON Disk Library and tape library storage are usually together in Tier 6. We engage BCVs and clones as needed for data protection. The classification and movement of data within the Tiers of storage is an ongoing process.
Beginning Environment
The original file server environment contained Windows 2000 and 2003 servers hosting all user-accessed data and Home drives. Microsoft’s Distributed File System (DFS) consolidated all the scattered shares into one virtual folder. Solaris UNIX servers hosted the NFS exports for some database backups and file archival. We used FTP and
Samba to ferry data across operating systems. We utilized Northern Quota Server to control file usage limits. A large amount of old data resided on many shares, and users could add any kind of data to any accessible share.
Goals
We needed a solution to address all of our concerns, and provide room to grow. The migration had to be minimally disruptive, and automated maintenance of the new environment was a key requirement. We needed to reduce complexity by eliminating FTP, Samba, and Quota Server. After weighing our needs, wants, and goals, we
decided to upgrade our Celerra, add more storage, and purchase DiskXtender® for NAS.
Ending Environment
The ending file server environment consists of EMC Celerra NS704G hosting all file server duties. The Celerra connects on the back end to the SAN, and can access storage on the DMX and CLARiiON. LAN connectivity is two 1 GB connections per Celerra Data Mover, expandable to eight.
DXNAS used automated policies and schedules to archive files from the Celerra to the EMC Centera or CLARiiON ATA disks. The Celerra handled both user and tree quotas, Home drives, and all CIFS shares and NFS exports. The Celerra provides access for UNIX and Windows clients to the same folders as needed. We integrated security with Active Directory for Windows clients, and permission bits for UNIX clients.
We performed backups through the SAN using NDMP, and they were very fast. We used Celerra Snapsure to create file system checkpoints that were automatically accessible using Microsoft’s Previous Version client. File system provisioning and expansion wais online with no end user disruption.
All Data Movers were dual connected to the SAN and LAN, and are configured to
failover to a standby Data Mover as needed. Both the Celerra and EMC ControlCenter®
can receive alerts and notify administrators. Migration
The basic migration procedure required the creation of new space on the Celerra. Then we used EMCopy, robocopy, or rsynch to perform incremental copies of the data. We considered using the Celerra Data Migration Service, but wanted to reorganize data in different shares than the source servers. We also wanted to make use of the Tree Quota feature for better
management of departmental file usage. The steps involved for a typical migration:
• Size the data – data is NOT compressed on the Celerra
• Create a target file system or Tree Quota on the Celerra
• Use EMCopy to perform an initial copy of data
• Use EMCopy for an incremental data copy, recording the time
• Schedule an outage for the length of incremental copy time
• Stop the source CIFS share
• Perform the incremental copy
• Create the new share
• Update DFS links
• Confirm that all is well • Delete the original data
Rollback or recovery using the above steps was as easy as recreating the old share and updating DFS. Most migrations followed the typical model, but we engineered workarounds for the many caveats we experienced. The sections below details how we handled migration, and lists the issues we encountered.
Provisioning
We sized Celerra file systems at between 200 and 600 GB, although some by necessity are larger. We used Automatic Volume Management to create all file systems, and turned on Automatic File System Extension for critical file systems. All file systems for CIFS shares use CLARiiON fiber channel disks, which provide a large savings when compared to DMX FC disks.
By default, Snapsure checkpoints use the same disks as the production file system but we set our checkpoints to use CLARiiON ATA disks to reduce costs. Checkpoints by default will extend their disk usage as needed to manage the Snapsure copies. Using ATA for Snapsure savvols is cost efficient, but has a marked effect on Snapsure performance.
We mounted all file systems to custom mount points, and modified the root level security by removing the default Everyone Group and adding Administrators. This had to be done to initialize Windows ACLs on the file system; without accessing the file system first through a Windows share, the permissions would default to Everyone having full control. We created an admin share at the file system mount to provide top-level management for permissions and quotas.
Security
Before migrating a CIFS file system, we used Somarsoft DumpACL to back up file security information. We set EMCopy, robocopy, and rsynch to duplicate security descriptors as they copied data. This led to a few unidentified SIDs in the Windows world, but worked well for the most part.
Backups and Restores
Netbackup host clients handled backups on the source file servers, sending data over the network to Media servers, which then wrote the data to a TLU. When data outgrew backup windows, we utilized TimeFinder BCVs and spun the data to tape using
dedicated hosts. Large restores were slow.
After migrating, the Celerra uses NDMP to back up data over the SAN straight to the TLUs at six times the previous rate, dramatically reducing backup times. NDMP sends only file metadata to the Netbackup Media Master server database, the bulk of the traffic is through the fiber links in the SAN. This also resulted in much faster restores,
particularly when coupled with the CLARiiON Disk Library.
We used Celerra Snapsure to create several checkpoints of each file system for backups and easy restores. These checkpoints were automatically mounted, and any Windows client that has Microsoft’s Previous Version Client installed can view earlier versions of their files. Restore is simply dragging and dropping. This has reduced restore requests by distributing the power to restore recent files. The only drawback was user readiness, as we needed to deploy training to educate users about this technology.
Communication
Communication between affected departments occurred using our change control process and email. We addressed any problems or difficulties through our standard trouble ticket process.
Distributed File System (DFS)
Distributed File System enabled a smooth migration for the CIFS shares. The DFS root server was the first file server migrated. We exported DFS using the built-in command line tools, renamed the DFS root server, and then created a CIFS server on the Celerra with the DFS root server name. A quick update to WINS and DNS effectively transferred identity. We created a new DFS root and imported the DFS information; it took about 10 minutes for the move.
After each share migration, a simple update to DFS kept users pointed to the right shares without having to change drive mappings or scripts. DFS allowed us to select which share to migrate at a given time; the caveat was that the old file servers had to stay online until all shares were migrated.
Users who did not log into the domain, primarily contractors, encountered problems. DFS roots showed them blank folders. The workaround solution was to modify the direct share mappings for each of the contractors.
Home Drives
There are over 900 users with Home drives. The old way of handling Home drives involved creating a folder, a hidden file server share, and updating the Active Directory (AD) to map the drive at logon. Each AD entry was unique.
The Celerra’s built-in Home drive feature greatly simplified this process. Using Home drive with the Windows plug-in, we were able to automatically create Home drives. All Home drive shares now have the same share name; the Celerra connects each user to their own assigned area. On the cutover night, we ran a script in AD to set the drive mappings, then all worked well.
Quotas
The Celerra manages User/Group Quotas and Tree Quotas. The User Quotas use file owners to collect quota information. Tree Quotas use file usage, block usage, or number of files to enforce quotas. Using these built-in features enabled us to remove a third party quota solution from our environment, saving money.
User Quotas must be enabled for a file system or within a Tree Quota. We also ran scripts to set correct owners for files before the quotas worked correctly. A little
freeware program called Chown enabled us to change file owners in bulk. We managed User Quotas through the Celerra tools or by using Windows Quota Manager.
We set Tree Quotas for most major shares migrated to the Celerra. This enabled us to set boundaries for departments so we can better manage space allocation. Setting a Tree Quota creates the folder. Crossing soft and hard limits generates alerts. We can grow or shrink Tree Quotas, allowing quick provisioning and space recovery.
We did experience some frustration with Celerra quotas. Limit alerts contain quota IDs, (i.e., UIDs, GIDs or treeids), not names. This requires us to locate quota names using a separate Celerra command. An existing folder cannot convert into a Tree Quota, and Tree Quotas cannot be subfolders of an existing Tree Quota.
CIFS Migrations
We used EMCopy and robocopy for almost all CIFS migrations to the Celerra. This gave us the ability to do the initial copy at our leisure, then an incremental copy at cutover. Large shares still experienced several hours of downtime, but the incremental copies required less than one-tenth the initial copy times.
One problem involved alternate data streams (ADS). This is extra data sometimes added to a file as an additional bit of metadata. Executable files downloaded from the internet will sometimes have these alternate data streams, which trigger Windows to caution the user about running the files. The files cannot copy from one computer to another, including the Celerra.
The Celerra natively supports ADS, and we should be able to copy files using ADS to and from the Celerra, unless a specific parameter change has been made to prevent these copies. One can remove ADS by manually unblocking the file through the file properties, based on Windows documentation. This would be difficult for thousands of files in various folders. The Sysinternals program streams.exe. is a particularly useful program to remove ADS in bulk.
NFS Migrations
We provisioned NSF exports and copied data using rsync, a free tool. We also used tar and cp as needed.
Using the built-in Celerra tools for NFS exports, we presented the exports only to the servers that needed them. We allowed root level users to access the exports on the mount hosts, another of Celerra’s practical options. We also hosted UNIX Home drives on the Celerra.
OS Data Sharing
Sometimes data generated in Windows must be imported into a UNIX system or vice versa. The Celerra replaced the old method of using Samba or FTP with native access to the same data locations. The same file system folder or directory can have a CIFS share secured with Windows ACLs and an NFS export secured with UNIX permission bits. With both types of servers mounting the same shared space, we simply copied files to the location to share among the operating systems.
Sharing files between dissimilar operating systems allowed us to reduce complexity and eliminate FTP and Samba’s security risks. Automated jobs using dedicated, bridged space and users were posted to an area set aside for miscellaneous use. This
simplification eliminated several perpetual trouble spots, reduced convolutions, improved performance, and provided better security.
Archiving
DiskXtender for NAS is a software program from EMC/Legato that works with the native Celerra FileMover API to move files to different locations based on user-defined policies. We employed DX for NAS to archive files to the EMC Centera or to less expensive CLARiiON ATA disks. User-defined policies were very flexible, containing almost any option needed for file classification.
Archiving files reduced the number of files to back up, saved money by utilizing less expensive storage, and kept files quickly accessible. The Celerra holds a small stub file on the production disk that points to the real file, and can be set to pass the file to the user from the less expensive storage or retrieve the file as necessary. The EMC Centera retains files for a specified length of time to comply with industry regulations. DX for NAS reduced our production file system usage by over 60%, decreasing backup time and saving money. The stub files can be backed up or moved and will still point to the correct archived storage.
One issue with DX for NAS is the way in which it stores target files in an unfamiliar directory structure and file name. It is difficult to browse the archived files and determine anything about them.
Another issue is there was no way to “unarchive” the files. There are ways to retrieve them using backups or copying, but there is no native support within the software. NAS and DX for NAS integration could be simpler, but it worked well once deployed. Be aware that only accounts with appropriate permissions will work properly for file
migrations using DX for NAS. In addition, the migration destination must have the proper permissions set for users to access their files after archiving.
Data Shaping
We do not allow certain types of data on some user shares. By using Celerra’s File Extension Filtering feature, we can block certain files at the share level. This enabled us to prevent unwanted files (ie. .pst or .mp3) on certain file systems. We also used this feature to permit only certain groups to access particular files, or to allow only files with a defined extension.
Data shaping supported corporate policy compliance and saved disk space. Although users may find it bothersome, it ensured compliance with corporate policies.
Future Plans
Our new file server environment using NAS and multi-tiered storage is expandable and flexible. The infrastructure will manage normal growth far into the future with minimal hardware
additions. The implementation of additional features and technologies are among the more exciting plans for the future.
We plan to continue our efforts to identify and classify data tiers. As data ages, it will likely migrate tiers. This overall cost savings effort has a snowball effect. It begins with less expensive disks and reduced array software licensing fees, continuing on to less production data to backup and manage. We will continue to research and implement as many automated procedures as possible.
The EMC Celerra is a feature-rich device, offering many advantages in our environment. We are currently evaluating Access-Based Enumeration (ABE) that would present users only with folders they are authorized to access. This will support our “need to know” policies.
We are also evaluating the Celerra’s ability to support iSCSI. Using this feature, servers would connect to disks hosted by the Celerra through a VLAN. We want to use iSCSI to reduce SAN port and HBA costs. So far, iSCSI on the Celerra looks very promising and performance seems adequate for a large subset of our servers.
Our storage infrastructure and NAS are now better equipped to support a disaster recovery hot site. Future plans involve file system replication using Celerra Replicator, and array-to-array data replication.
We have yet to set up Celerra AntiVirus Agent (CAVA). Our Enterprise anti-virus solution is one of the few that charges extra money to support CAVA. Our workaround is to mount shares and scan them from a Windows server, although we plan to implement CAVA in the future.
Conclusion
Overall, the file migration process from open systems to NAS went well. We realized an immediate cost savings by changing the hardware tier the file server data lived on, especially when factoring in the storage licensing costs within the DMX array for TimeFinder. Backups are quicker, and we have more backup and restore options. The Celerra Data Movers are
intrinsically set up for high availability, removing single points of failure. Performance has improved. The new environment is expandable, flexible, and centrally managed.
What Went Well
Things that went very well:
• Setting up data tiers in the DMX and CLARiiON arrays
• Spanning of data tiers using NAS and DX for NAS
• Ease of manageability on the NAS
• Sharing data between UNIX and Windows servers
• Reduced learning curve for Windows administrators using Celerra Windows
integrated tools
• Sizing of data, compressed and uncompressed using an in-house developed
tool, FolderSize
• Security worked flawlessly on the Celerra and across tiers
• Using DFS as the central director while migrating Windows shares
• Improved backup speeds
• NAS checkpoint integration into the backup routine, including Previous Version
Client support
• File archiving and forced retention options
• The final file serving I/O and performance on the NAS
Problems, Limitations, and Cures Problems and limitations encountered:
• Unlike Windows file servers, the Celerra does not compress data – this caused
data to use 2 – 4 times the space on the NAS. This was acceptable because we migrated to 9x cheaper disks, generating a net savings.
• We used Virtual Data Movers at first, but VDMs are set up to support CIFS, not
NFS. We needed areas to have CIFS shares and NFS exports. *There is a workaround to set up exports on VDMs, but not by using regular means. We moved data to regular Data Movers and removed the VDMs.
• We needed to set file ownership to use User Quotas on the Celerra. Windows
only provides for taking, not setting, file ownership. We had to find a freeware tool named Chown, and use it to set the file owners before migration.
• The Celerra by default supports alternate data streams (ADS). For some reason,
the ADS support bit was off for our Data Movers. We set the bit, but a DMs reboot is required for it to take effect. The ADS streams causing issues were Internet Zone tags on downloaded files. Windows only provides a way to do this one file at a time. We researched and found a Sysinternals tool named
streams.exe that worked to clear the ADS in bulk.
• We had to mount newly created file systems on the NAS intended for Windows
once we had permissions set at the root level before any file copying permissions would apply correctly.
• Celerra alert is effective but difficult to set up and maintain.
• DiskXtender for NAS works very well once set up, but can be difficult to integrate
with the Celerra.
• This is not a problem, but deduplication support on the Celerra and CDL would
benefit end users. Final Thoughts
At the end of the migration, tens of terabytes of data changed where it lived and how users accessed it. Users were patient (most of the time) and disruptions minimized, making this endeavor a success. There were some expected and unexpected issues, but we were able to find viable workarounds. EMC could improve some things with the Celerra and EMC storage environments, but this author would definitely recommend that other companies implement multiple storage tiers and migrate their file server data to NAS.
*EDITORIAL NOTE: VDMs should be used for all resented CIFS servers. Exporting filesystems from a VDM-mounted CIFS server for use by NFS clients is an EMC supported process and is done using regularly Celerra export capabilities. The minimal additional configuration work is far outweighed by the many advantages of VDM usage (enhanced security, local portability, remote server replication).
