Data Administration

Data management is a primary consideration when developing enterprise GIS architectures. Enterprise GIS normally benefits from efforts to consolidate agency GIS data resources. There are several reasons for

supporting data consolidation. These reasons include improving user access to data resources, providing better data protection, and enhancing the quality of the data. Consolidation of IT support resources also reduces hardware cost and the overall cost of system administration.

The simplest and most cost-effective way to manage data resources is to keep one copy of the data in a central data repository and provide required user access to this data to support data maintenance and operational GIS query and analysis needs. This is not always practical, and many system solutions require that organizations maintain distributed copies of the data. Significant compromises may have to be made to support distributed data architectures.

This section provides an overview of data management technology. Several basic data management tasks will be identified along with the current state of technology to support these tasks. These data management tasks include the following:

Ways to store spatial data Ways to protect spatial data Ways to back up spatial data Ways to move spatial data

New ways to manage and access spatial data

6.1 Ways to Store Spatial Data

Storage technology has evolved over the past 20 years to improve data access and provide better management of available storage resources. Understanding the advantages of each technical solution will help you select the storage architecture that best supports your needs. Figure 6.1 provides an overview of the technology evolution from internal workstation disk to the storage area network architecture.

Figure 6-1

Advent of the Storage Area Network

Internal Disk Storage. The most elementary storage architecture puts the storage disk on the local machine.

System Design Strategies 6 Data Administration C11144-12

Most computer hardware today includes internal disk for use as the storage medium. Workstations and servers can both be configured with internal disk storage. The fact that access to it is through the local workstation or server can be a significant limitation in a shared server environment: if the server operating system goes down, there is no way for other systems to access the internal data resources.

File server storage provides a network share that can be accessed by many client applications within the local network. Disk mounting protocols (NFS and CIFS) provide local application access over the network to the data on the file server platform. Query processing is provided by the application client, which can involve a high amount of chatty communications between the client and server network connection.

Database server storage provides query processing on the server platform, and significantly reduces the required network communication traffic. Database software improves data management and provides better

administration control of the integrity of the data.

Internal storage can include RAID mirror disk volumes that will preserve the data store in the event of a single disk failure. Many servers include bays that support multiple disk drives for configuring RAID 5 configurations and support high capacity storage needs. The internal storage access is limited to the host server, so a many data center environments grew larger in the 1990s customers would have many servers in their data center with too much disk (disk not being used), and other servers with too little disk making disk volume management a challenge (data volumes could not be shared between server internal storage volumes). External storage architecture (Direct Attached, Storage Area Networks, and Network Attached Storage) gives a way for organizations to “break out” from these “silo based” storage solutions and build a more manageable and adaptive storage architecture.

Direct Attached Storage. A direct attached storage (DAS) architecture provides the storage disk on an external storage array platform. Host bus adaptors (HBA) connect the server operating system to the external storage controller using the same block level protocols that were used for Internal Disk Storage, so from an application and server perspective the direct attached storage appears and functions the same as internal storage.

The external storage arrays can be designed with fully redundant components (system would continue operations with any single component failure), so a single storage array can support high available storage requirements.

Direct attached storage technology can provide several fiber channel connections between the storage controller and the server HBAs. For high availability purposes, it is standard practice to configure two HBA fiber channel connections for each server environment. Direct Attached Storage solutions provide from 4 to 8 fiber channel connections, so you can easily proved up to 4 servers with two redundant fiber channel connections from a single direct connect storage array controller. The disk storage volumes are allocated to specific host servers, and the host servers control access to the assigned storage volumes. In a server failover scenario, the primary server disk volumes can be reassigned to the failover server.

Storage Area Networks. The difference between direct attached storage and a storage area network is the introduction of a Fiber Channel Switch to provide network connectivity between the Servers and the external Storage Arrays. The storage area network (SAN) improves administrative flexibility for assigning and managing storage resources when you have a growing number of server environments. The Server HBAs and the External Storage Array controllers are connected to the Fiber Channel Switch, so any Server can be assigned storage from any Storage Array located in the storage farm (connected through the same storage network).

Storage protocols are still the same as with Direct Attached or Internal Storage – so from a software

perspective, these storage architecture solutions appear the same and are transparent to the application and data interface.

Network Attached Storage. By the late 1990s, many data centers were using servers to provide client

application access to shared file data sources. High available environments require clustered file server storage, so if one of the servers fail users would still have access to the file share. Hardware vendors now provide a highbred appliance configuration to support network file shares (called Network Attached Storage or NAS) –

System Design Strategies 6 Data Administration C11144-12

the local area network, and client applications can connect to the storage over standard disk mount protocols.

The network attached storage provided a very simple way to deploy a shared storage for access by a large number of UNIX and Windows network clients. Figure 6-2 shows the evolution of the Network Attached Storage architecture.

Figure 6-2

Advent of the Network Attached Storage

Network attached storage provides a very effective architecture alternative for supporting network file shares, and has become very popular among many GIS customers. As GIS data moves from early file based data stores (coverages, LIBRARIAN, ArcStorm, Shapefiles) to a more database centric data management environment (Geodatabase servers), the network attached storage vendors suggest you can use a network file share to support a database server storage. There are some limitations: It is important to assign dedicated data storage volumes controlled by the host database server to avoid data corruption. Other limitations include slower database query performance over the chatty IP disk mount protocols than with the traditional fiber channel SCSI protocols, and the bandwidth over the IP network is lower than the Fiber Channel switch environments (1 Gbps IP networks vs 2 Gbps Fiber Channel networks) – implementation of Network Attached Storage as an alternative to Storage Area Networks is not an optimum storage architecture for geodatabase server environments. At the same time, it is an optimum architecture for file based data sources and use of the NAS technology alternative continues to grow.

Because of the simple nature of network attached storage solutions, you can use a standard local area network (LAN) Switch to provide a network to connect your servers and storage solutions; this is a big selling point for the NAS proponents. There is quite a bit of competition between Storage Area Networks and Network Attached Storage technology, particularly when supporting the more common database environments. The SAN community will claim their architecture is supported by higher bandwidth connections and the use of standard storage block protocols. The NAS community will claim they can support your storage network using standard LAN communication protocols and provide support for both database server and network file access clients from the same storage solution.

The network attached storage community is providing a more efficient iSCSI communication protocol for their

System Design Strategies 6 Data Administration C11144-12

storage networks (basically SCSI storage protocols over IP networks). GIS architectures today include a growing number of file data sources (examples include ArcGIS Image Server imagery, ArcGIS Server pre-processed 2-D and 3-D file caches, and the file geodatabase). For many GIS operations, a mix of these storage technologies provides the optimum storage solution.

6.2 Ways to Protect Spatial Data

Enterprise GIS environments depend heavily on GIS data to support a variety of critical business processes.

Data is one of the most valuable resources of a GIS, and protecting data is fundamental to supporting critical business operations.

The primary data protection line of defense is provided by the storage solutions. Most storage vendors have standardized on redundant array of independent disks (RAID) storage solutions for data protection. A brief overview of basic storage protection alternatives includes the following:

Just a Bunch of Disks (JBOD): A disk volume with no RAID protection is referred to as just a bunch of disks configuration, or (JBOD). This represents a configuration of disks with no protection and no performance optimization.

RAID 0: A disk volume in a RAID 0 configuration provides striping of data across several disks in the