"Backend cleanup process for factory re-installation of VNX OE for File (NAS software for VNX File)"
ID: emc257758 Usage: 14 Date Created: 12/10/2010 Last Modified: 02/03/2012 STATUS: Approved Audience: SupportQuestion: Backend cleanup process for factory re-installation of VNX OE for File (NAS software for VNX File) Environment: EMC SW: VNX Operating Environment (OE) for File
Environment: Product: VNX File/Unified
Environment: Backend cleanup using nas_raid -s cleanup
Environment: Factory re-installation using Express Installation DVD image on Control Station
Problem: Requirements to perform a factory re-installation of the Operating Environment for File (that is, NAS code) Problem: nas_raid script fails if system is part of a multi-domain system:
Cannot cleanup domain master. Please move master to another array.
Fix:
Backend cleanup for factory re-installation of the File O/S Cautions:
During the cleanup process, the Control LUNs are zeroed out so as to make a fresh re-installation possible. It should be noted that the Control LUNs and default Storage Group (~filestorage) are now part of the FLARE private LUN space, and no longer directly accessible from the GUI or NaviCLI.
After the cleanup process, verify that all Control LUNs are owned by SP A on Chain 0, or the installation process will fail.
Cleanup script may not remove other Storage Groups, Storage Pools, and the like. Backend cleanup script does not remove the default ~filestorage HBAUID records and
must be done manually.
VNX FILE/UNIFIED BACKEND CLEANUP PROCEDURE
1. Deconfigure Proxy ARP—the main task here is to get the SPs back on the 128.221.252 & 253 networks:
# /nasmcd/sbin/clariion_mgmt –stop
Note: If you cannot stop Proxy ARP services or cleanup the backend, see emc287103 for possible workarounds. LUNs 0 & 1 must be zeroed out in order to perform a fresh reinstall of File OE.
2. Verify that the storage processors (SPs) are up and running with the default internal network IP addresses:
# ping 128.221.252.200
PING 128.221.252.200 (128.221.252.200) 56(84) bytes of data. 64 bytes from 128.221.252.200: icmp_seq=1 ttl=128 time=0.535 ms # ping 128.221.253.201
PING 128.221.253.201 (128.221.253.201) 56(84) bytes of data. 64 bytes from 128.221.253.201: icmp_seq=1 ttl=128 time=0.353 ms
3. Make sure /tftpboot directory is available at root of system--untar from /nas/tools if required:
# cd /
# tar zxvf /nas/tools/tftpboot.tar.gz
4. Unset the NAS_DB environment and stop NAS Services:
Note: If running dual Control Stations, shutdown CS1. If onsite, unplug the power cable from CS1 and leave it offline.
# unset NAS_DB # /sbin/service nas stop
5. Run the Cleanup script (which may take 15-20 minutes to complete): # cd /tftpboot/setup_backend
# ./nas_raid -n ../bin/navicli -a 128.221.252.200 -b 128.221.253.201 –s cleanup Do you want to clean up the system [yes or no]?: yes
Cleaning Storage Group "~filestorage" Removing LUN
PXE boot slot 2...
Starting NBS on all control LUN Zero LUN 1 with dd.
Finished with LUN 1. Zero LUN 0 with dd. Finished with LUN 0. Removing diskgroup
The following storage groups still exist: ~filestorage
Removing spares Security domain removed Done
Note: If the nas_raid script fails with 'Cannot cleanup domain master', you will need to remove any other systems from the domain before the script will complete.
# /tftpboot/bin/navicli -h 128.221.252.200 domain -messner -remove 10.241.216.233 [SP IP of other domain to remove from the current domain]
6. Verify that Control LUNs have been properly zeroed out: # /sbin/fdisk –l | grep partition
Disk /dev/nda doesn't contain a valid partition table Disk /dev/ndb doesn't contain a valid partition table Disk /dev/ndc doesn't contain a valid partition table Disk /dev/ndd doesn't contain a valid partition table Disk /dev/ndf doesn't contain a valid partition table
Note: It is possible that you may not have NBS access to the backend LUNs from your Blades. If this is so, you must first PXEBoot a blade in order to restore backend LUN access. The /dev/nde partition is not zeroed out.
Optional Method for zeroing LUNs 0 & 1:
success, try to zero out LUNs 0 & 1
# dd if=/dev/zero of=/dev/nda bs=1MB count=134 # dd if=/dev/zero of=/dev/nde bs=1MB count=134
7. Manually remove other Storage Groups, if necessary:
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagegroup -list
or
# /tftpboot/bin/navicli -h 128.221.252.200 storagegroup -list Storage Group Name: SG_Celerra_c125
Storage Group UID: E2:12:0B:D6:F5:FC:DF:11:8F:CA:00:60:16:41:67:7D HLU/ALU Pairs:
HLU Number ALU Number --- ---
0 3
1 1
2 0
3 2
#/tftpboot/bin/navicli -h 128.221.252.200 storagegroup -destroy -gname SG_Celerra_c125
Destroy Storage Group SG_Celerra_c125 (y/n)? y
8. Manually remove Pool luns from the ~filestorage Storage Group if required: # /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 lun -destroy -l 13
Are you sure you want to perform this operation?(y/n): y Cannot unbind LUN because its contained in a Storage Group Get List of HLU numbers for ~filestorage SG:
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagegroup -list -gname ~filestorage
Remove HLU Luns from ~filestorage:
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagegroup -removehlu -gname ~filestorage -hlu 18
Remove HLU 18 from ~filestorage
The specified operation will potentially affect a File System Storage configuration. Do you want to continue (y/n)? y
9. Manually destroy Pool LUNs first, if necessary:
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagegroup -removehlu -gname ~filestorage -hlu 25 -->Remove Pool lun from SG first
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 lun -list
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 lun -destroy -l 0 -->Syntax for removing Pool luns once the StorageGroup has been destroyed
Are you sure you want to perform this operation?(y/n): y 10. Destroy the Storage Pool once the Pool LUNs are removed:
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagepool -list
Pool Name: Pool 0 Pool ID: 0 Raid Type: r_10
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 storagepool -destroy -id 0
Are you sure you want to perform this operation?(y/n): y
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 storagepool -list
11. Manually destroy RAID Group LUNs and RAID Groups, if necessary:
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 getrg -lunlist -->In this example, there are RAID Group LUNs and a RAID Group to destroy
RaidGroup ID: 1 List of luns: 7 8
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 unbind 7
Unbinding a LUN will cause all data stored on that LUN to be lost. Unbind LUN 7 (y/n)? y
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 unbind 8
Unbinding a LUN will cause all data stored on that LUN to be lost. Unbind LUN 8 (y/n)? y
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 removerg 1
MetaLuns:
# /tftpboot/bin/navicli -h 128.221.252.200 metalun -list -->There may be metaluns [e.g., 8184, 8185, etc] if layered apps were in use
# /tftpboot/bin/navicli -h 128.221.252.200 metalun -destroy -metalun 12 -->Select metalun from list, within the 8184 lun
12. Verify whether any Control LUNs are trespassed from SP A to SP B: # /nasmcd/sbin/t2tty -c 2 "camshowconfig"
CAM Devices on scsi-0:
TID 00: 0:d0+ 1:d1+ 2:d2+ 3:d3+ 4:d4- 5:d5- -->d4 & d5 are trespassed to SP B CAM Devices on scsi-16:
TID 00: 0:d6- 1:d7- 2:d8- 3:d9- 4:d10+ 5:d11+
1291584475: ADMIN: 6: Command succeeded: camshowconfig
Note: Through the use of - and +, the above output shows that Control Luns d4 and d5 are NOT on Chain 0 (SPA). These LUNs must be trespassed back to Chain 0 on all
servers before a new install can succeed.
13. Trespass back all Control LUNs to SPA Chain 0 as required using the following commands:
# /tftpboot/bin/t2tty –c 2 ―camtrespass c0t0l4‖ # /tftpboot/bin/t2tty –c 2 ―camtrespass c0t0l5‖ # /tftpboot/bin/t2tty –c 3 ―camtrespass c0t0l4‖ # /tftpboot/bin/t2tty –c 3 ―camtrespass c0t0l5‖
Note: In the above example, LUNs 4 & 5 were trespassed back to Chain 0 SPA on each of the two blades present on the system.
14. Verify existing Data Mover WWN HBAUID Records, remove HBAUID records, and verify:
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 storagegroup -list -gname ~filestorage |head -15
Storage Group Name: ~filestorage
Storage Group UID: 60:06:01:60:00:00:00:00:00:00:00:00:00:00:00:04 HBA/SP Pairs:
HBA UID SP Name SPPort --- --- ---
50:06:01:60:C6:E0:14:97:50:06:01:69:46:E0:14:97 SP B 2 50:06:01:60:C6:E0:14:97:50:06:01:61:46:E0:14:97 SP B 3 50:06:01:60:C6:E0:14:97:50:06:01:68:46:E0:14:97 SP A 2 50:06:01:60:C6:E0:14:97:50:06:01:60:46:E0:14:97 SP A 3
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 port -removehba -o -hbauid 50:06:01:60:C6:E0:14:97:50:06:01:69:46:E0:14:97 # /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 port -removehba -o -hbauid 50:06:01:60:C6:E0:14:97:50:06:01:61:46:E0:14:97 # /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 port -removehba -o -hbauid 50:06:01:60:C6:E0:14:97:50:06:01:68:46:E0:14:97 # /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 port -removehba -o -hbauid 50:06:01:60:C6:E0:14:97:50:06:01:60:46:E0:14:97 # /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 storagegroup -list -gname ~filestorage
Storage Group Name: ~filestorage
Storage Group UID: 60:06:01:60:00:00:00:00:00:00:00:00:00:00:00:04
15. Verify whether array security was destroyed. If not a shared system, manually destroy security.
# /tftpboot/bin/naviseccli h 128.221.252.200 user sysadmin password sysadmin -scope 0 domain -list
Security is not initialized. Security must be initialized before any domain operations can be performed in this system. Create a global administrator to initialize security. Note: The above return indicates that no security domain remains and has been destroyed--no further action required
# /tftpboot/bin/navicli -h 128.221.252.200 domain -list Node: c250
Name: spb Port: 80 Secure Port: 443 IP Address: 128.221.252.200 (Master) Name: spa Port: 80 Secure Port: 443 IP Address: 10.241.216.235 Name: c250 Port: 80 Secure Port: 443
Note: The above return indicates that a security domain does exist and must be destroyed
# /tftpboot/bin/navicli -h 128.221.252.200 -user sysadmin -password sysadmin -scope 0 domain -messner -destroy
WARNING: You are about to destroy the local directories on the following systems:128.221.252.200
Please note that this operation will not update the master directory database.Proceed? (y/n) y
# /tftpboot/bin/navicli -h 128.221.253.201 -user sysadmin -password sysadmin -scope 0 domain -messner -destroy
WARNING: You are about to destroy the local directories on the following systems:128.221.253.201
Please note that this operation will not update the master directory database.Proceed? (y/n) y
16. Using the proper bootable Express Install media, reboot the Linux system and perform a "boot:install". For Dual Control Station environments, make sure that CS1 is powered off during the factory install of CS0. See the "Note" section below for a representative example of the questions and answers given for a typical Express Installation. Make sure to toggle the option from Yes to No when the screen for setting up the Control Station LAN IP address appears, since you DO NOT want to set the External IP address yet (you will use the VNX Installation Assistant after the installation is completed to set the Control Station name and IP address and to initialize the File/Unified system). Reboot CS0 after the File OE installation completes so as to generate the "Waiting for VIA..." initialization message.
17. Once CS0 has completed software installation and reboot, perform the factory installation of CS1 using either the DVD media or CD2 media, and keep CS0 powered up during the CS1 installation. At the end of the successful FIle OE installation on CS1, reboot it, and via the serial console, ensure that it displays the "Waiting for VIA..." initialization message. At this point, the dual CS environment is ready to be initialized using the VNX Installation Assistant.
18. Before running the VIA, however, perform the following actions, depending on whether the system is a File-only or Unified configuration:
For File-only VNX Systems:
a) A File-only installation should not have the -UnisphereBlock enabler installed--use navicli ndu -list to check.
b) A File-only installation should have the -UnisphereFile enabler installed--use navicli ndu -list to check.
c) Run the VNX Installation Assistant to complete the system initialization. For Unified VNX Systems:
a) A Unified installation should have both the -UnisphereBlock and -UnisphereFile enablers installed--use navicli ndu -list to check.
b) Set the Unified flag on the Control Station:
# /nas/sbin/nas_hw_upgrade -option -enable -clariionfc
c) Run the VNX Installation Assistant to complete the system initialization.
Notes:
Typical Express Installation questions, inputs, and/or answers: 1. Express Installation using DVD or 2-disc CD set:
boot:install ---
Is this a Secondary Control Station (y/n/a)? n
---
Is this a Control Station Fresh Install? yes
---
Is this a Secondary Control Station? [yes or no]: no ---
Accept the defaults for the "Primary Internal Network Setup", "IPMI Network Setup", and "Backup Internal Network Setup" screens
---
DO NOT SETUP THE EXTERNAL LAN NETWORKING AT THIS TIME (we will setup the Control Station external LAN using the VIA initialization wizard after the File OE reinstallation is completed)
For the Network Configuration screen, "Do you want to configure LAN (not dialup) networking for your installed system?"
Tab to "No" and enter. ---
Detecting movers in cabinet: 2
Is this the expected number of movers in the cabinet? [yes or no]: yes ---
Pick a NAS Administrator username Username [default: nasadmin] : nasadmin New UNIX password: nasadmin
Retype new UNIX password: nasadmin ---
Do you wish to enable UNICODE? [yes or no]: yes
2. At the end of the File OE installation, log into the Control Station as nasadmin, su to root, then reboot the Control Station. When the following message is displayed at the Control Station serial console, initialize the Unified system using the VIA:
# reboot
---
