Panzura Storage Controller Administration Guide

(1)

Panzura Storage Controller

Administration Guide

Release 6.1

December 2015

(2)

Copyright

© 2015 Panzura, Inc. All rights reserved. The information in this manual is subject to change without notice. Originated in the USA. Any other trademarks appearing in this guide are the property of their respective companies. 12/11/2015 695 Campbell Technology Parkway Suite 225 Campbell, CA 95008 [email protected] 1‐855‐PANZURA (1‐855‐726‐9872) +1 (408) 578‐8888 www.panzura.com

(3)

Preface . . . 6

1 Overview . . . 7

Panzura Storage Architecture . . . 7

Panzura Storage Controller . . . 8

Controller Features . . . 9

Description of Key Features . . . 10

Panzura Unified Namespace . . . 10

File Locking . . . 10

Data Mobility and Snapshots . . . 10

User‐Managed Snapshots . . . 10

Data Locality . . . 11

Extended File System ACLs . . . 11

Scale‐out Global Deduplication. . . 11

Intelligent Read/Write Cache . . . 11

Enhanced Cloud Diagnostics . . . 12

Configurable Mesh . . . 12

Intelligent Symantec NetBackup Integration . . . 12

High Availability . . . 12

DR Cloud Recovery . . . 12

10GbE and 1GbE NIC Support . . . 13

Unified Storage. . . 13

Enterprise AntiVirus Plugin . . . 13

Configuring the Controller . . . 13

Accessing the Web UI . . . 13

Navigating the Web UI. . . 14

Dashboard. . . 15

2 Deployment — The First 72 Hours . . . 24

Panzura Controllers. . . 24

Master and Subordinate Controllers . . . 24

Local and Remote Controllers . . . 24

Understanding CloudFS . . . 25

Deploying Panzura CloudFS . . . 26

3 CloudFS Map View . . . 33

Video Overview . . . 33

Using the CloudFS UI. . . 33

CloudFS UI. . . 34

Google map . . . 34

(4)

Determining the location of a controller on the map. . . 35

History bar . . . 36

Information windows . . . 37

Heat maps. . . 37

Site to site connectivity . . . 39

Monitoring details . . . 41

4 Configuration . . . 43

Basic Settings . . . 44

System (Basic) . . . 44

Network . . . 46

Time . . . 51

Encryption and Certificates (Basic). . . 52

CloudFS . . . 56

Active Directory (Basic) . . . 58

Dynamic Sites . . . 60

NFS . . . 61

SMB (Basic). . . 64

Snapshot . . . 65

License Manager . . . 67

Advanced Settings . . . 77

System (Advanced) . . . 77

KMIP . . . 79

High Availability . . . 81

Encryption and Certificates (Advanced). . . 83

Mesh . . . 85

Active Directory (Advanced) . . . 88

SMB (Advanced). . . 90

Snapshot Manager. . . 92

Cache. . . 93

Data Locality Settings . . . 94

Bandwidth Limit . . . 99

SNMP . . . 101

Logging . . . 104

(5)

5 Maintenance . . . 106

Diagnostics . . . 106

High Availability . . . 114

SMB Dashboard . . . 116

SMB Users . . . 119

Secure Erase . . . 119

CIoudFS . . . 120

CloudFS File Browser . . . 121

Cloud Metrics . . . 124

NFS . . . 127

Master Snapshot. . . 127

Cloud Delete . . . 128

Image Upgrade . . . 129

Reboot . . . 130

Support . . . 131

Password . . . 132

Advanced. . . 134

A Windows Users and Files that Are Slow to Open . . . 136

B Creating A Microsoft Azure Storage Container . . . 137

(6)

Welcome to the Panzura Storage Controller (controller). The preface contains the following information: “About this Guide” on page 6 “Document Conventions” on page 6 “Related Documents and Resources” on page 6

About this Guide

This guide describes how to deploy and manage the controller. It is designed for experienced storage administrators who are responsible for configuration and maintenance. It assumes that you are familiar with NFS and SMB/CIFS network storage operations. For installation information, see the Panzura Storage Controller Installation Guide.

Document Conventions

This document uses the following conventions.

Panzura Storage Architecture

The Panzura OS (PZOS) provides a highly scalable, high performance global file system called CloudFS that is natively integrated with object‐based cloud storage systems. CloudFS allows clients from multi‐site locations to interact with and lock files within a single global file system address space, known as the unified namespace. Within the Panzura architecture, global shared file read operations are serviced locally at LAN speeds. File write permissions are managed by the Panzura's Global Read Write file locking technology.

Within the Panzura network architecture (Figure 1), all controllers share access to a common storage cloud. The cloud is the authoritative source of data for all controllers. Because cloud storage is effectively infinite in capacity and elastic in scale, cloud storage is always larger than the local physical storage that is available in any particular controller (at any particular site). The Panzura architecture includes technologies to leverage this difference in capacity and manage the aspects of deploying a NAS storage system that operates under this paradigm. Each Panzura controller provides LAN speed access to CloudFS. Locally, each controller caches data while presenting a complete metadata view of the entire CloudFS that spans all connected sites.

(8)

Figure 1 Panzura Network Topology

Panzura Storage Controller

The Panzura Storage Controller (controller) is the core of the Panzura storage architecture. The controller is a reliable, high performance, optimized cloud storage appliance that can manage massive data densities within its scalable file system. The resilient storage subsystem protects data using military grade encryption, multiple RAID parity protection schemes, efficient user managed snapshots, and cloud storage. The controller provides local and cloud storage for widely‐used file storage protocols, file management technologies, and directory services integration. Network File System (NFS), used by Unix/Linux clients and servers Server Message Block (SMB) / Common Internet File System (CIFS), used by Microsoft Windows clients and servers

(9)

The controller can virtualize multiple disk media types within the same file system. Supported media include spinning hard disk drives (HDDs), solid state drives (SSD), networked WAN‐addressable cloud storage, and LAN‐addressable NAS filer volumes. PZOS serves data to clients with SMB/CIFS and NFS protocols.

Controller Features

The Panzura Storage Controller provides the following features: Highly scalable 128‐bit transactional object file system Intelligent read & write caching In‐band file system Policy Engine User managed snapshots Extended file system ACLs RAID data protection Globally distributed file sharing and locking for SMB/CIFS High availability Microsoft Active Directory integration Kerberos authentication Key Management Interoperability Protocol (KMIP) support Global namespace Scalable live global deduplication Policy defined variable length deduplication Multi‐protocol SMBv1, SMBv2, and NFSv3 file services SMB/CIFS load balancing Military grade encryption Unified SSD, HDD, and cloud storage Intelligent LAN and WAN bandwidth management Intelligent NetBackup support for multi‐site backup, recovery, and archive Tivoli TSM support for multi‐site backup, recovery and archive SNMPv3 monitoring, traps, and alerting Online remote monitoring and support High speed parallelized WAN‐optimized cloud I/O Multiple cloud topologies (public, hybrid, private) Real‐time cloud storage diagnostics Full system recovery from the cloud 1GbE and 10GbE NIC support (optical and copper) Bandwidth shaping and connection tuning

(10)

Description of Key Features

The next sections describe key features in detail.

Panzura Unified Namespace

The Panzura unified namespace is an in‐band file system fabric that consists of multiple physical file system instances converged into a single file system metaspace and mounted locally on each node with the root label “CloudFS.” The Panzura unified namespace does not rely on underlying distributed databases and thereby avoids common global namespace limitations that can affect speed, transactional data coherence, write order fidelity, open files, atomic precision, in‐band operation, and global snapshots. By contrast, other global namespace architectures require a database process on each storage system and changes to file metadata require complex out‐of‐band operations.

File Locking

PZOS and CloudFS support several file locking mechanisms. A traditional file lock is a lock issued against a file by a file system, a server or an application. The lock can consist of extensive application specific meta‐ information and be written into parts of the file payload and/or its file system metadata. File coherency locks are file system locks that are issued by applications to arbitrate guaranteed consistency between applications writing/reading to a single file, for example, MSFT Office Application locks. Opportunistic caching locks are delegated rights that are issued by a file server protocol engine for a remote client to cache a file locally to increase client‐side performance. This is not necessarily a guaranteed write lock, because the delegation can be revoked by a file server at anytime. Example: Microsoft SMB/CIFS OPLOCKS.

Data Mobility and Snapshots

When the controller is operating, it actively takes the configured file system snapshots. Snapshots can occur at the granularity of one per second and are virtually unlimited with respect to data address space and number of entities. The snapshot process captures and manages the user data state, individual file system state, and total system data. All snapshots are independent and able to be moved between systems. Snapshots capture the state of the global file system address‐space in a single operation with guaranteed data coherence at byte‐level granularity. Snapshots never overwrite data blocks. If a new data block is written, then the next snapshot contains a pointer to the new block. If an existing block is modified, a new block is written, while the old block is retained within the active file system for the previous snapshot.

User‐Managed Snapshots

User managed snapshots provide point‐in‐time copies to protect data with no performance impact and minimal consumption of storage space.

(11)

You can take user‐managed snapshots while applications are running to create copies in less than one second, regardless of size of your file system, the data set, or the level of activity on the system. You can schedule automatic snapshot creation or take snapshots on demand. Snapshots are visible on the desktop explorer environment for SMB/CIFS and NFS protocols. You can access and retrieve older snapshots from SMB/CIFS or NFS mounts without involving IT administration. Each controller has been tested for up to 10,000 user managed snapshots with its global file system. The total number of addressable snapshots is a function of the number of controllers in the entire unified namespace. For example, for a unified namespace with 16 nodes, the maximum tested capacity is 160,000 snapshots. See “Master Snapshot” on page 127.

Data Locality

Data locality (caching and pinning) governs what data is cached locally on the controller. PZOS provides fine‐grained configuration control over caching through the use of policies, rules and actions that result in improved performance and enhanced cloud storage availability for users. When configuring data locality, Panzura recommends using the auto cache action with prepopulate enabled. This ensures that files are available in disk cache for end users. Prepopulating makes the data available without forcing a reduction in cache. Pinning allows an administrator to forcefully localize (pin) data in the cache within a controller to provide guaranteed LAN speed performance. Because pinning consumes cache space, it should be considered only if needed for performance, with the trade‐off between performance and cache space kept in mind. Note that data is always protected in the cloud, irrespective of polices rules and pinning. See “Data Locality Settings” on page 94.

Extended File System ACLs

The Panzura file system supports extended file system access control lists (ACLs) with full POSIX semantic compliance. For NFSv3, the system supports read and write ACL operations through an SSH shell or application layer. Clients can dynamically use any supported ACL mode for setting and getting ACLs. For SMB/CIFS, clients use a native Microsoft method with the Server Message Block (SMB) protocol for reading and writing extended ACLs. PZOS provides the ability to turn SMB signing on or off. See “SMB (Basic)” on page 64.

Scale‐out Global Deduplication

The controller supports enhanced data deduplication within the global file system with high performance, scalability, and system‐wide efficiency. The deduplication data architecture and physical layout on disk is optimized for local and global write performance and data addressability. See “System (Advanced)” on page 77.

Intelligent Read/Write Cache

PZOS supports an intelligent read/write disk‐based cache (IRC) that reserves a percentage of local storage to intelligently track hot, warm, and cold file block structures as they are accessed. The cache dramatically increases data availability and I/O performance, because file data‐blocks reads have a higher probability of being serviced from local disk than directly from external cloud storage. The cache also

(12)

increases overall data availability by masking variations in cloud availability. This allows the file system to continue serving I/O and cache‐resident data‐block reads even when the WAN link to the cloud storage slows or is unavailable, or if the cloud itself is down. See the Panzura technical white paper TWP‐3002 for an in‐depth description of IRC.

Enhanced Cloud Diagnostics

The controller provides a robust set of diagnostic and measurement tools for monitoring and understanding the health, status, and performance of the Panzura cloud storage infrastructure and interactions. The cloud diagnostics features include tools to analyze the cloud, while the cloud metrics features present a detailed set of trended graphs to visually display metrics about cloud reads and writes. The system also generates alerts if the cloud becomes unavailable. See “Diagnostics” on page 106.

Configurable Mesh

Configurable Mesh allows an administrator to reduce the metadata that is synchronized with any particular controller, thus reducing the synchronization overhead, bandwidth requirements for synchronization events, and visibility to excluded controller. Each controller can be configured to synchronize only select metadata from other configured controllers within the same CloudFS. Sync relationships can be defined as unidirectional or bidirectional. With configurable mesh, the scope of deduplicated information is restricted to the metadata that is visible to a given controller.

Intelligent Symantec NetBackup Integration

PZOS supports intelligent integration with Symantec NetBackup, including awareness of the NetBackup data format stream. The system efficiently deduplicates the data stream inline with high optimization ratios. Tivoli TSM, Microsoft Robocopy and Symantec Backup Exec are also supported as cloud backup and cloud archive applications.

High Availability

The Panzura High Availability (HA) solution consists of two configuration options, HA‐Local and HA‐ Global. HA‐Local is similar to the methods used by legacy enterprise storage products. In this configuration, an active controller is protected by a dedicated, passive standby. When the active controller fails, the passive standby assumes its identity and takes over operation. The innovative HA‐Global configuration provides a more cost effective and flexible solution. This implementation makes it possible to protect many controllers with a single passive standby that can significantly reduce the cost of HA. When a failure occurs, the standby assumes the identity of the failed controller and takes over operation. If a greater level of protection is required, multiple standby controllers can be deployed to protect against multiple controller failures. And the standby controllers can be geographically distributed from the controllers they protect to provide greater flexibility. See “High Availability” on page 81.

DR Cloud Recovery

Full cloud disaster recovery (DR) allows rebuilding and recovery of an entire controller from the cloud

(13)

online and established as active from its cloud metadata instances as soon as possible following the disaster. A minimal set of data blocks are recovered from the cloud to bring the system to a state where clients can start using the controller, with remaining data downloaded in priority order.

10GbE and 1GbE NIC Support

PZOS provides support for high‐performance multi‐media network interface cards (NICs). 10GbE NIC options are available along with continued support for 1GbE NIC copper options. Optical and copper NIC SFP+SR media support are also available on 2U systems with the 10GbE NICs options.

Unified Storage

Each controller provides RAM, SSD, and HDD media that are homogeneously converged and managed as a single physical local storage pool. The physical configuration and number varies depending on the controller model. For larger systems, the number of SSD and HDD media can be expanded by adding disk expansion shelves (up to a total of 324TB RAW) which becomes available to the system's storage services, including IRC.

Enterprise AntiVirus Plugin

Protecting file servers from viruses is an important part of an overall security strategy, and products such as McAfee VirusScan Enterprise 8.8 with VirusScan Enterprise for Storage 1.1 address this need well. With the Panzura Enterprise Virus Scanning (VSE) license, you can use McAfee VSE to automatically scan files with the latest virus definitions based on Internet Content Adaptation Protocol (ICAP). McAfee VSE controls the actions taken in the event a virus is detected. These actions include the ability to block the file from being accessed, logging the detection event, and quarantining the file. If a file was previously scanned, it is not rescanned unless the virus signature information has changed, the cloud controller has rebooted, or the filename, contents or path have changed.

Configuring the Controller

The controller installation process includes initial configuration steps. See the Panzura Storage Controller Installation Guide for instructions. After the controller is installed, use the web UI for all additional configuration and management.

Accessing the Web UI

To access the controller Web UI, enter the controller’s IP address into your web browser and log in using the following default credentials: Account = admin Password = admin

(14)

Navigating the Web UI

The web UI contains the following areas (see Figure 2): Top information area — Displays the hostname of the controller, software version, user name, and link to log out. Tabs — Allows you to access the Dashboard, Configuration, or Maintenance pages. The Dashboard is selected when you first open the web UI. Left side menu — Allows you to navigate to individual configuration and maintenance pages. Visible only when you click the Configuration or Maintenance tab. The contents of the side menu depends on what license are active and can also depend on configuration settings. For example, the SMB Settings page is visible on the side menu only if a SMB/ CIFS license is active and SMB is selected as an option on the CloudFS page. Alert area—Presents current alert information. Shown only if there is are current alerts. Figure 2 Web UI Tabs Top information area Left side menu Alert area

(15)

Dashboard

The Dashboard page opens when you log in to the controller and is accessible at any time by clicking the Dashboard tab. Beginning with Release 6.1, a new Dashboard is available with a new look and feel and information. To see the new Dashboard, click Try the new Dashboard. To change back from the new to the legacy Dashboard, click the Dashboard tab again.

See “New Dashboard” on page 15 and “Legacy Dashboard” on page 20.

Figure 3 Selecting the New Dashboard

New Dashboard

The new Dashboard gives an at‐a‐glance visual picture of controller status and statistics, as described in this section.

(16)

Events

The Events box shows the most severe outstanding event (severity and count). Hover over the box to display a list all the events with that severity. The count in the box matches the number of event displayed. This example shows that there is one minor event, which is currently the most severe. Hover over the box to display the event name.

Systems

The System box shows the status of system resources. CPU load. Hover over the box to show the current CPU load value. The green box turns red if the CPU load reaches 90% or greater. Memory. Hover over the box to show the current percent of memory used. If swap space is being used, the swap space value is also presented. The green box turns red if memory usage reaches 90% or greater or if swap space used is greater than 1 GB. Disk. If the box is green, all disks are online. If any disks are offline, the box is red. Hover over to list the offline disks.

(17)

Network

Bandwidth. Shows the egress and ingress value in a bar chart, with the actual value (in Mbps) displayed at the bottom. If a bandwidth limit is configured, it is shown as a blue line. The bar color turns from green to red if the actual value reaches 80% or more of the bandwidth limit. If a bandwidth limit is not configured, the bar turns red if the value reaches the interface limit (1GB for GE and 10GB for XGE). In one‐arm mode, title of the box is Bandwidth and the statistics are incoming and outgoing octets, which are converted into Mbps at 5 minute intervals.

Cloud

The Cloud box shows upload and download status (failures vs success). The color turns yellow if there are both failures and successes, and red if there are only failures. Hover over to see the success and failure counts.

(18)

Latency

The Latency box show shows latency values between the local controller and other configured controllers. The green color changes to yellow if latency is between 175 ms and 250 ms and red when latency exceeds 250 ms.

File System

The File System box shows the following information: Local Synchronization. When a controller’s file system is synchronized, its box is green and labeled “Synced.” When the file system is in the process of syncing, its box is green and the estimated time is displayed. If the time to synchronize cannot be estimated because the snapshots behind are increasing, the box displays “Sync Error" and becomes red. The box is also red if the controller is down. Cache. Shows how the cache is being used by data: — Pinned — Cached — Dirty — Free The number inside the circle represents the percent of cache with used data (pinned + cached + dirty). The number inside the circle is green unless the value reaches 80% of the total cache, in which case it turns red.

(19)

Hover over the chart area to display the labels for specific parts of the pie and to show the actual values. Metadata. Shows how much metadata space is being used. The number inside the circle is green unless the value reaches 80% of the total cache, in which case it turns red. Hover over the chart area to display the labels for specific parts of the pie and to show the actual values.

Storage

the Storage box shows the amount of cloud and disk storage used. The used value is shown in blue and the available value is shown in grey. The number inside the circle is green unless the value reaches 80% of the available storage, in which case it turns red. Hover over the chart to see the data label and over a segment to see the actual value.

(20)

Legacy Dashboard

The legacy dashboard displays the following information:

Network Throughput – Shows ingress and egress bytes (bytes per second) for the selected time period. (See “Data Interval and Refresh Rate” on page 22.)

Bandwidth Summary (Last Week) – Shows a summary of bandwidth use over the last week, in 3‐ hour increments. Bandwidth is shown for traffic from the controller to clients. Disk and RAID – Shows the status of the local storage disks on this controller. Data storage availability counters are provided for all local disks plus cloud storage.

Note

In Unified Management deployments, this counter does not include storage for disks on other controllers in the group. On a new controller, the storage statistics are for the local device only. When you install a license for cloud storage, the storage statistics are immediately updated to include the storage available in the cloud. Multi‐Site Topology (CloudFS) – Shows status information for each controller (see the following section). Figure 5 Dashboard

(21)

Multi‐Site Topology

The Multi‐site Topology (CloudFS) section of the Dashboard now includes detailed information about the synchronization state of all controllers. This information can help a Panzura administrator identify when the controllers within a CloudFS are not in sync. In some extreme cases, controllers not being in sync can cause end users to report not being able to see recently created files or folders. Panzura controllers use system level snapshots behind the scenes to communicate local filesystem changes to all other controllers. These are unrelated to the user snapshots that can be scheduled within the Web UI. When a controller takes a system level snapshot, it is sent to the cloud. All other controllers download and apply this snapshot. This makes it possible for the users accessing the other controllers to have a view of the non‐local portion of the filesystem. In some extreme cases, a controller might fall behind in uploading its latest system level snapshots, or a controller might fall behind in downloading its peers' latest system level snapshots from the cloud. If either of these occurs, the CloudFS becomes out of sync. This can lead to end users reporting that they cannot see newly created files or folders. The Snapshot Generated column contains the reference number for the most current system level snapshot that the controller has created, while the Snapshot Uploaded column contains the reference number for the latest system level snapshot that the controller has successfully uploaded to the cloud. For example in the following figure, the most current system level snapshot generated by cc1‐blr has a reference number of 2026. The latest snapshot cc1‐blr has uploaded to the cloud is also 2026. Because these match, it is clear that the controller is not falling behind with uploading system level snapshots. Had the Snapshot Generated reference number been 2036 or higher and the Snapshot Uploaded reference number been 2026, that would indicate that the 10 most recent system level snapshots had not yet been uploaded to the cloud. In this scenario, the network connection from cc1‐blr to the cloud should be checked. The last column to the right is titled Snapshot Received Locally. It contains the reference number of the latest system level snapshot from the controller that has been downloaded and applied to the controller from which the admin is currently viewing the Dashboard. In the following figure, the data indicates that the cc1‐NVA system level snapshot 1743 has been received from the cloud and applied to controller cc1‐ca. So controller cc1‐ca is in sync with cc1‐NVA.

(22)

Had the Snapshot Received Locally reference number been 1733 and the Snapshot Uploaded number been 1743, that would indicate a problem. Users connected to cc1‐ca would have an old view of cc1‐ NVA. In this scenario, the network connection between cc1‐ca and the cloud should be checked. In general, all controllers should be within several snapshots of each other. When controllers are out of sync by five or more or more snapshots, end users will begin to notice problems. To see the exact date and time of the latest received snapshot, hover over the Snapshot Received Locally reference number.

Data Interval and Refresh Rate

By default, statistics are shown for the most recent 1‐hour period, and are refreshed once a minute. You can change the data interval and refresh rate. Data interval – The data interval can be one of the following: — 5 minutes — 30 minutes — 1 hour — 24 hours — 7 days — 30 days Refresh interval – The refresh interval can be one of the following: — Off (no automatic refresh) — 30 seconds — 1 minute

(23)

You also can refresh the data at any time by clicking Update Now.

Redisplaying the Dashboard

If you navigate away from the Dashboard, you can easily display it again by clicking the Dashboard tab.

Configuration Pages

The Configuration pages are divided into Basic Settings and Advanced Settings. The Basic Settings pages are listed when you click the Configuration tab. To list the Advanced Settings pages, click Advanced Settings. For information on configuration tasks, see Chapter 2, “Configuration.” Click the Maintenance tab to display the Maintenance pages. For information on maintenance tasks, see Chapter 3, “Maintenance.”

(24)

This chapter explains how to navigate the first few days of your Panzura controller deployment, including how to scope the rollout of a Panzura global cluster and anticipate end‐user expectations so that disruption is minimized during the rollout. “Panzura Controllers” on page 24 “Understanding CloudFS” on page 25 “Deploying Panzura CloudFS” on page 26

Panzura Controllers

Panzura controllers, physical or virtual, create the foundation for the CloudFS file system. Users mount the Panzura controller using SMB/CIFS or NFS protocols and experience the performance of a high‐speed LAN connected file server. In addition to providing local file services, the controller also shares file data and metadata with a cluster of controllers that can span the globe. Each controller has a complete global view of the CloudFS and provides LAN speed performance by locally caching the files that are being accessed at that site.

Master and Subordinate Controllers

The terms master/subordinate and local/remote refer to the roles of individual controllers in a CloudFS deployment. The master/subordinate configuration applies to the management relationship between controllers. The configuration details of a master controller are automatically replicated and distributed to all subordinate controllers. This simplifies the management of the CloudFS deployment. The replicated information includes license information, SMB/CIFS shares, NFS exports, and the user snapshot schedule. See “System (Basic)” on page 44 for configuration instructions.

Local and Remote Controllers

The local controller is the controller nearest the user on a local LAN. A remote controller is one that is physically located in another office, somewhere around the globe. These terms are used when describing the flow of files and metadata within CloudFS, from one controller to another controller. The following figure a Panzura deployment that includes three sites—Paris, New York, and London. A Panzura controller is physically deployed at each site. Users connect to their local controller, have a complete view of the shared file system, and experience LAN access speeds to the data in the global file system.

(25)

Figure 6 Panzura Network Topology

Understanding CloudFS

CloudFS is a global file system that runs on Panzura controllers. Geographically dispersed controllers allow users who connect to CloudFS to experience a high‐speed file system, regardless of location. Updates to the file system are shared in the background, in near real time, with all the other controllers in the cluster. Integrated with CloudFS is a global file locking technology, called Global Read Write (GRW), which controls read and write file locking. This technology allows many users and work sharing applications to leverage the global CloudFS without suffering file locking or performance issues. When deploying Panzura controllers at multiple sites, users connect to their local controller but share the resources of the entire global file system. The practical aspect of a CloudFS deployment is best described through a real world example. Consider a company, AEC Corporation, with three sites, Paris, New York, and London, as shown in the previous figure. Paris is the primary site, and the domain name for AEC Corporation is aec‐example.com.

(26)

The three Panzura controllers are deployed as follows. 1. The controller in Paris, paris.aec‐example.com, is configured as a master controller. — A project directory is created: /cloudfs/paris/aec‐project‐01 — A SMB/CIFS share is created for the new directory: /aec‐project‐01 Users at this site connect to the SMB/CIFS share \\paris.aec.com\aec‐project‐01 2. The controller in New York office, new‐york.aec‐example.com, is configured as a subordinate to the Paris master controller. Users at this site connect to the SMB/CIFS share\\new‐york.aec.com\aec‐project‐01 3. The controller in London, london.aec‐example.com, is configured as a subordinate to the Paris master controller. Users at this site connect to the SMB/CIFS share\\london.aec.com\aec‐project‐01

Deploying Panzura CloudFS

The next sections describe the high‐level steps to deploy Panzura CloudFS. For details, see the the

Panzura Storage Controller Installation Guide and “Configuration” on page 43. Step 1: Install and Configure Panzura Controllers Step 2: Seed CloudFS with Files and Directories Step 3: Connect Users with CloudFS Step 4: Observe CloudFS Performance Step 5: Tune CloudFS Performance

Step 1: Install and Configure the Panzura Controllers

A CloudFS deployment runs on a cluster of globally distributed Panzura controllers, physical or virtual. The controllers are configured on their local networks, attached to DNS, and connected to the cloud back‐end. Before deploying controllers, decide which controller will be the designated master controller, and which controllers are to be subordinates of the master.

Install and configure Panzura controllers

1. Deploy the physical Panzura controllers. 2. Configure network access. 3. Set unique hostnames for each controller. 4. Identify the master and subordinate controllers and set the Configuration Mode. 5. Enter license information on the master and subordinate controllers. 6. Configure the cloud connector on the master and subordinate controllers. 7. Join the Active Directory Domain. 8. Reboot the subordinate controllers.

(27)

Note

When a subordinate controller reboots, the CloudFS configuration details are replicated from the master controller to the subordinate controller.

Step 2: Seed CloudFS with Files and Directories

It is important to seed CloudFS with project data before users access their local Panzura controller for the first time. In preparation for seeding data, create a directory structure that supports both current and future projects. Then upload project data and confirm that each controller in the cluster has the same view of the global file system and directory structure.

Seed CloudFS with File Data

1. Mount your local Panzura controller to your desktop via a SMB/CIFS share. 2. Create a directory structure that supports current and future project files. 3. Upload project files to the appropriate directories on the controller. 4. Wait for the data to synchronize. 5. Mount each of the remote controllers. 6. Confirm that the entire file system can be viewed from each controller. 7. Observe CloudFS performance using the ingress and egress rate counters in the Panzura Web UI.

Windows Tools for Seeding Data

Microsoft Windows offers GUI and command line tools for migrating data: The Windows Explorer GUI can be used to drag and drop files to the CloudFS SMB/CIFS share. However, this method of copying files does not preserve file or folder permissions (ACLs). Robocopy (Robust File Copy) is a command‐line utility—included with Windows Server 2012 and 2008—used to copy files and preserve file and folder permissions. Robocopy is scriptable, logs the copy process, features retry capabilities, and works around locked files.

Linux Tools for Seeding Data

Linux OS distributions include the rsync tool that can be used to copy files and directories from one server to another over an SSH connection. It is scriptable, preserves file permissions, and copies only new or changed files to the destination folder.

Understanding Snapshots

Panzura uses snapshots to capture the state of the file system at a given point in time. In PZOS there are two types of snapshots; system managed and user managed. The system managed snapshots are used to provide file system consistency between controllers. In a process called syncing, PZOS takes the changes (deltas) that occur to files and to the file system metadata, captures the delta information in a snapshot, and sends them to the cloud. The metadata portion of these changes is retrieved from the cloud by all other Panzura controllers in the cluster where they are used to update the state of the file system and maintain currency. This system updating occurs continuously across all controllers, with each controller sending and receiving extremely small metadata snapshot deltas and using them to update the file system.

(28)

User managed snapshots are controlled by the administrator to provide file system backups in the shared cloud storage back‐end. You can schedule automatic snapshot creation or take snapshots on demand. They are visible to the end user so they can be used to retrieve old versions of files without involving IT administration. Panzura guarantees that each controller will support more than 10,000 user‐managed snapshots.

How long will it take to seed CloudFS with data?

When files are uploaded to a SMB/CIFS share on the local Panzura controller, the files and metadata are immediately uploaded to the shared cloud storage back‐end. The files and metadata become available to all the other controllers in the cluster, which immediately download the metadata and synchronize their file systems. When working with normal amounts of data, the cycle of uploads and downloads is nearly instantaneous and invisible to the end user. However, when seeding large amounts of data to a controller, be aware that it takes time to upload files and file system metadata to the shared cloud storage back‐end. The time to complete this upload and download cycle is governed by the speed of the network links connecting the clients to the controllers and the controllers to the shared cloud storage back‐end. When data is uploaded to a share on the local controller, the PZOS operating system creates a system snapshot to capture the state of the file system and to identify the files that have been created or changed. Before uploading files to the cloud, the data in the files is broken down into smaller chunks of data, called drive files, which are uploaded sequentially to the shared cloud storage back‐end. The file data is uploaded first followed by the file system metadata snapshot. Remote controllers constantly poll the shared cloud storage back‐end looking for new metadata snapshots. When new metadata snapshots are found, they are downloaded one by one and applied to the local file system. After the last metadata snapshot is downloaded and applied, the global file system is fully synchronized among all the controllers. The time required move data to CloudFS and share the updated file system metadata with all the controllers in the cluster is a function of the following values: T1 = time to transfer files on a LAN from the local file server to the local Panzura controller. T2 = time to upload the files (drive files) and the metadata snapshot from the Panzura controller to the shared cloud storage back‐end. T3 = time to download the metadata snapshot from shared cloud storage to the remote Panzura controllers. The remote controllers poll the cloud every 30 seconds, so the time to find and begin to download the metadata snapshot is no more than 30 seconds. The time required to move files between a file server and the Panzura controller is governed by the speed of the local area network. Actual network speed is determined by the available bandwidth on the network. With that in mind, the following formula calculates the minimum amount of time to move data to the Panzura controller, and between controllers and the cloud. time (sec) = amount of data (GB) * 8 network speed (Gb/s) For example, with 100GB of data and a 1Gb/s LAN, the calculations are as follows: T1 = 100*8/1 = 800 seconds T2 = 100*8/1 = 800 seconds

(29)

Thus, the minimum time to upload and share 10GB of data with a 1Gb/s LAN is: T1+T2+T3 = 800+800+30 = 1630 seconds = 27.2 minutes. Note that network connections to the cloud are frequently a lot less than 1Gb/sec

Step 3: Connect Users to CloudFS

When CloudFS is running efficiently, it provides fast access to files that are distributed in the global file system. However, it takes time to distribute files to the appropriate controllers. Until files are cached in the local controllers, it appears that the system is running slowly. By design, the decision to cache data is an automated process that is triggered when a user accesses the data. Therefore, users will experience slow system performance as the system becomes balanced and data is cached at the appropriate controller. Also, when a large amount of data is uploaded to CloudFS, users who want to access that data will experience slow access times until the files are downloaded and cached to their local controller. Subsequent file access will be fast, and updates to cached files will be shared quickly among all the controllers in the cluster. Windows Explorer users will experience a delay when viewing a directory with a large number of new files for the first time. Windows Explorer must open every file in the directory before it can display the directory listing. If these files are not yet cached locally, Windows Explorer becomes unresponsive and appears to hang. Once the Panzura controller downloads all of the files from that directory to the CloudFS cache, Windows Explorer performs normally. In all cases, data locality rules can be used to prepopulate the cache and improve performance for a particular folder. Even crawling the file folders (reading the files) can be used to populate the cache with data before users connect to the controller. It is a common practice for a knowledgeable system administrator to “walk” particular project directories in specific locations ahead of time to improve the local user’s first experience by ensuring the files they’re likely to use are already cached in the local controller.

Connect Users to CloudFS

1. Users mount their local controller via SMB/CIFS. 2. Users browse file directories, then access and update files. 3. Files become cached locally and I/O performance increases. 4. Observe CloudFS performance using the ingress and egress rate counters in the Panzura Web UI.

Step 4: Observe CloudFS Performance

PZOS provides tools for monitoring CloudFS. Three counters are used to observe the flow of data through CloudFS: rate of data ingress, rate of data egress, and the synchronization of system snapshots. The ingress and egress counters are viewed from the dashboard in the Web UI. The synchronization counters are viewed from the Diagnostic Tools menu in the Web UI. The following discussion of data flow within CloudFS uses a two‐controller deployment, with controllers named LOCAL and REMOTE.

(30)

Figure 7 Network Throughput Graph

Observe uploads on the LCOAL Panzura controller

1. Open the Dashboard page in the Web interface. 2. Configure the Network Throughput graph to use a refresh rate of 30 seconds, and view data for last 5 minutes. 3. Copy files to a SMB/CIFS share on the local controller. — The ingress rate increases as data is copied to the controller. — The egress rate increases as drive files and snapshots are copied to the cloud back‐end. Only after the drive file uploads are completed will the file system metadata snapshot be uploaded. The egress rate returns to zero after all drive files and snapshots are successfully uploaded.

(31)

Observe downloads on the REMOTE Panzura controller

1. Open the Dashboard page in the Web interface. 2. Configure the Network Throughput graph to use a refresh rate of 30 seconds, and view data for last 5 minutes. — The remote controller polls the shared cloud back‐end storage every 30 seconds for the latest snapshots. — The snapshot sequence number of the latest snapshot is compared with the snapshot sequence number of the local metadata snapshot. — If the sequence numbers don’t match, the controller will proceed to download metadata snapshots, one by one, until the sequence numbers match again. — The ingress rate increases as the metadata snapshots are downloaded. — The ingress rate returns to zero after all the file system metadata snapshots are successfully downloaded. — When completed, the LOCAL and REMOTE file system are synchronized.

Confirm that the file system is synchronized across all controllers

In addition to observing the Ingress and Egress rates on the Panzura controllers, it is also useful to check whether the LOCAL and REMOTE file systems share the latest snapshot sequence number. It is only when the file systems on the LOCAL and REMOTE controllers report the same snapshot sequence number that all file systems are synchronized. Follow the steps outlined below to determine whether the file systems on the controllers are synchronized with each other. The following examples use LOCAL as the hostname of the local controller and REMOTE as the hostname of the remote controller. Substitute the actual hostnames of your controllers when executing the diagnostic commands.

Verify that the local controller has uploaded all file data and file system metadata snapshots

1. In the Maintenance tab on the LOCAL Web UI, select run‐cmd from Diagnostic Tools drop‐down menu. 2. Type zfs get –p all zroot/LOCAL and press Enter. 3. Look for snapshot number for last_gen_snap and last_cloud_uploaded_snap. 4. If they’re the same, then all of the system‐generated snapshots have been uploaded to the cloud. If last_cloud_uploaded_snap is lower than last_gen_snap, then the controller is still working to upload all of the file data and file system metadata snapshots.

Verify that the remote controller has downloaded all available file system metadata

snapshots

1. In the Maintenance tab on the REMOTE Web UI, select run‐cmd from the Diagnostic tools drop‐ down menu. 2. Type zfs get –p abs_rcvd_snap zroot/REMOTE and press Enter. 3. When REMOTE and LOCAL file systems are synchronized, the snapshot sequence number on the REMOTE controller will match the last_cloud_uploaded_snap on the LOCAL controller. If the sequence number is lower than last_cloud_uploaded_snap on the LOCAL controller, then the REMOTE system is still working to download the file system metadata snapshots.

(32)

Step 5: Tune CloudFS Performance

PZOS provides an automated, intelligent read cache (IRC) that increases file I/O performance. Over time and through general usage, the system dynamically populates the IRC with hot data from files being read by users. The caching algorithm monitors the frequency of file access, and how recently files were accessed, to determine what data to cache and what data to eject from the cache. CloudFS performance can be tuned to increase the performance of specific files and directories with the use of data locality rules. Data locality rules govern what files are cached locally on the controller. These rules are also used to pre‐populate the cache in to guarantee LAN speed access to files before users access them for the first time. The Panzura technical white paper, Panzura Global Cloud Storage System: Caching and Pinning, contains an in-depth description of caching and pinning.

(33)

This chapter describes the CloudFS Map view (CloudFS UI), which lets you see the health of the overall controller network. “Video Overview” on page 33 “Using the CloudFS UI” on page 33 “Google map” on page 34 “Information windows” on page 37 “Site to site connectivity” on page 39 “Monitoring details” on page 41

Video Overview

These videos provide an introduction to the CloudFS UI (lengths given in minutes). Part 1: CloudFS UI & Map View 2:08 Part 2: CloudFS Status Details 4:41 Part 3: New Controller Dashboard 2:24 Part 4: CloudFS File Browser 1:47 Part 5: CloudFS Monitoring Reports 3:12 Watch all videos in one session 14:00

Using the CloudFS UI

The CloudFS UI provides a graphical view of all the configured controllers. It displays the current synchronization status and allows you to step back in time to see behavior of the following statistics: Cache status – Amount of cache space used (bytes). Sync status – Synchronization delay (minutes, hours, or days). Cloud status – Number of cloud I/O operations (upload and download). SMB users – Number of clients connected to CloudFS.

(34)

CloudFS UI

The following figure shows the CloudFS UI. The UI includes a Google maps image of the geographical area containing the controllers along with custom controls.

External controls

The following controls are external to the Google maps interaction: Time control. Allows you to keep the 1 minute refresh or turn automatic refresh off (1 minute is default). Update now. Click to update the map and associate data on demand.

Google map

The Google Map image shows the controllers based on their site location entered in the System Settings (Basic) page of the Web UI. You can specify a location using any method accepted by Google maps, including address, city, or latitude/longitude. Google Maps places markers on the map at the correct location. Tip: When you have a map view that you like, save it as a bookmark in your browser. When you reopen the page using the bookmark, the zoom level, pan position, selected statistics, and site‐to‐site filter are all preserved; however, the time on the slider is reset to the current time.

(35)

When the map is loaded initially (or CloudFS tab is clicked), the zoom level is adjusted automatically and centered so to show all the controller markers on the map. You can zoom in and out, pan, change between satellite and map view, or do other functions supported in Google Maps. Each controller has an icon that shows its function. The color of the icon marker indicates the highest event severity during the selected hour. The map also displays these additional icons and properties.

Determining the location of a controller on the map

The location of each controller is derived from the information entered in Location field on the

Configuration > Basic Settings > System Settings page (“System (Basic)” on page 27).

The controller sends the specified location to Google to obtain the corresponding latitude and longitude and stores the information for use in the CloudFS UI. In some cases, however, this process doesn’t work. For example, if the controller is in a private cloud and the network doesn’t include a router to reach

Master controller

Subordinate controller

Standby controller

Blue bubble The controllers are currently synchronized, or they aren’t cur‐ rently synchronized, but an estimate of the time to synchronize is displayed. The estimate is displayed in the bubble. Red bubble The current trend shows increasing lags in synchronization, and no estimate can be provided. Cluster When controllers are too close together to be viewed individu‐ ally on the map at the current zoom level, they are shown with an icon that represents a cluster and shows the number of con‐ trollers in the cluster. Group icon When multiple controllers are at the same location, the map displays a group icon, regardless of the zoom level. Expanded group icon Click a group icon to show the individual icons that indicate the type of controller.

(36)

Google, the address is not translated into latitude and longitude and the controller is not displayed in the correct location on the map. In this situation, you can obtain the latitude and longitude independently (from a site such as http:// www.latlong.net/Show‐Latitude‐Longitude.html), and specify the values in the Location field on the System Settings page. The setting must be entered in the format given in the following example, including the square brackets. After the entry is saved, the controller is able to use the values to place the controller on the map. [latlng=37.279463,‐121.943682]

History bar

The history bar area contains controls that allow you to look at selected statistics for any selected time in the past two weeks. Examining statistics and different times allows you to establish the status and health of the subcomponents at those times and can provide insight to help analyze problems that occurred. The slider bar lets you move back and forth in time to select a time of interest. The map time corresponds to the time selected with the slider bar. The statistics selector allows you to display status for a specific statistic. Click the arrow to show or hide the options. Click the name of an option to see the list and select another. The statistic count history shows a count of the selected statistic at each point over the past two weeks, corresponding to the slider bar. For example, the history chart in this figure shows the highest number of clients connected to the system (SMB users) between 6 and 8 days ago.

(37)

Information windows

When you click a controller icon on the map, an information window opens. The information in the window varies according to the type of statistic selected in the history area. The following figure shows example information windows for the cache status, sync status, cloud status, and SMB users statistics. The data is based on the following criteria. For information on viewing additional details, see (see “Monitoring details” on page 41.

Heat maps

Heat maps give additional information about the selected statistic and associated events. The chart in the history area shows aggregate counts for a statistic, but not the distribution of the statistics across the managed controllers. That information is provided by the heat maps.

Chart type Criteria for heat map

Cache Status Cache space used, based on pinned files and user data Sync Status Deficit (number of snapshots behind the latest snapshot) Cloud Status Total cloud upload and download failures (failure count) SMB Status Total number of SMB users

(38)

In this example, the selected statistic is cache status. In this example, for the SMB users statistics, all clients are connecting to the Louisiana controller. The heat map color gradient indicates intensity, as measured by the count of the selected statistic. Higher counts are indicated by a larger portion of the heat map in darker blue and yellow (highest counts). In this example, the controller in Los Angeles has a higher statistics count than the controller in Texas. The diameter of the heat map is always the same, but the intensity varies with the statistic count.

(39)

The data for the heat maps is based on the criteria listed in the information windows (see “Information windows” on page 37.

Site to site connectivity

The Site to Site control on the CloudFS map lets you examine the state of the links between controllers. Click the icon on the left of the Site to Site Propagation Time bar to display the available settings.

(40)

Select a source hostname or IP address and a destination. Click OK to shown the links between the specified controllers. The link lines are color‐coded based on site‐to‐site latency, as follows: grey ‐ unknown or data not available green ‐ between 0 to 175ms yellow ‐ between 176 to 250ms red ‐ above 250ms Hover over a link to display details about the link, as shown here.

(41)

Monitoring details

You can view monitoring details for a selected status type (cache status, sync status, cloud status, or SMB status) by clicking the bar chart icon in an information window (see “Information windows” on page 37). The Monitoring page opens to show data on the selected status type (for example, the following figure shows data for cache status).

The following controls are available on the Monitoring page:

Metrics. The available metrics are listed in the legend below the chart. Click a particular metric to toggle its display on or off.

(42)

Time range. Click to specify a time range.

Show events. Select the check box to show events on the chart. You can show all events, or just the info, major, or critical ones. Move your cursor over an event on the chart to display its details.

(43)

This chapter describes the settings on the Configuration pages for the Panzura Storage Controller (controller). See the Panzura Storage Controller Installation Guide for instructions on installation and initial setup.

Basic Settings

“System (Basic)” on page 44 “Network” on page 46 “Time” on page 51 “Encryption and Certificates (Basic)” on page 52 “CloudFS” on page 56 “Active Directory (Basic)” on page 58 “Dynamic Sites” on page 60 “NFS” on page 61 “SMB (Basic)” on page 64 “Snapshot” on page 65 “License Manager” on page 67

Advanced Settings

“System (Advanced)” on page 77 “KMIP” on page 79 “High Availability” on page 81 “Encryption and Certificates (Advanced)” on page 83 “Active Directory (Advanced)” on page 88

“SMB (Advanced)” on page 90“Snapshot Manager” on page 92

“Cache” on page 93

“Data Locality Settings” on page 94

“Bandwidth Limit” on page 99

“SNMP” on page 101

(44)

Basic Settings

The following sections describe how to configure basic settings.

System (Basic)

X Configuration > Basic Settings > System Use this page to set the controller hostname, enter location and contact information, and assign a configuration mode. Configuration modes: Master – This controller is the master for a group of distributed controllers. The other controllers are subordinates. All licenses and configuration settings are added to the master controller and automatically propagate to the subordinate controllers. Subordinate – This controller is a subordinate that is managed by a master. By default, all controllers are set as subordinates. Each deployment must have at least one master. By default, all controllers are masters and maintain their own security and configuration. When you add a subordinate, you must specify the master that will manage it. All configuration settings on the master are automatically propagated to the subordinates. After setting up the hostname and IP address information during installation, you do not need to configure any additional settings on the subordinates. If you have two masters in the same CloudFS list, ensure that their encryption certificates are the same at all times, as all systems in the same CloudFS list must have same encryption certificates.

Caution

Every time a subordinate is rebooted or the master configuration changes, the configuration of the subordinate is updated. If you change a controller from a master to a subordinate, the current configuration is overwritten with the configuration from the master. This can cause loss of user access to SMB/CIFS shares.

(45)

Figure 8 System Settings (Basic) After setting values, click Save. Table 1 System Settings (Basic) Item Description Hostname Set the controller hostname. The name must be unique within the CloudFS. No other controllers within the same CloudFS can have the same name. This simplifies administration and access to the managed controllers. Use the local DNS server at each site to return the IP address of the local controller. Changing the Hostname It is possible to change the hostname of a controller after it has been deployed. Enter the new hostname in the field and click Save. After renaming you must rejoin the Active Directory domain. Note that the names of SMB/CIFS shares and NFS exports do not change. This is because the filesystem name does not change. Location Location of the controller. Allows you to identify the location of the con‐ troller in the support emails and for the CloudFS UI (see “Determining the location of a controller on the map” on page 35). Contact Enter the contact email for the person responsible for controller administration. Configuration Mode Select whether the controller is a master or subordinate. Default is Master. When you select Subordinate, a field appears for you to add the hostname of the associated master. This field is visible only if Subordinate mode is selected