ii
Rackspace Private Cloud Object Storage Deployment
RPC v10 (2015-11-09)Copyright © 2015 Rackspace All rights reserved.
This documentation is intended for Rackspace customers who have installed an OpenStack-powered pri-vate cloud according to the recommendations of Rackspace and want to deploy OpenStack Object Storage (swift) as well.
Table of Contents
1. Overview ... 1
1.1. System requirements for Object Storage ... 2
1.2. Installation workflow ... 3
2. Networking for Object Storage ... 4
3. Example of Object Storage installation architecture ... 6
4. Prerequisites ... 8
5. Configure and mount storage devices ... 9
6. Configure an Object Storage deployment ... 11
6.1. Configuring Object Storage ... 11
6.2. Storage Policies ... 16
7. Object Storage playbooks ... 18
7.1. Running Object Storage playbooks ... 18
8. Verify the installation ... 20
9. Integrate Object Storage with the Image Service ... 21
10. Object Storage monitoring ... 23
10.1. Service and response ... 23
10.2. Object Storage monitors ... 23
A. Object Storage configuration files ... 25
A.1. swift.yml example configuration file ... 25
A.2. user_variables.yml configuration file ... 32
11. Additional resources ... 37
iv
List of Figures
1.1. Installation workflow ... 3 3.1. Object Storage architecture ... 7
List of Tables
1
1. Overview
Object Storage (swift) is a multi-tenant object storage system. It is highly scalable, can man-age large amounts of unstructured data, and provides a RESTful HTTP API.
Note
Before installing Object Storage v 2.2.0, Rackspace Private Cloud v10 (Juno) must be installed. Object Storage integrates with some components from the v10 release of Rackspace Private Cloud (for example, keystone, keystone v3). Object Storage includes the following components:
Proxy servers ( swift-proxy-server)
Accepts OpenStack Object Storage API and raw HTTP requests to upload files, modify metadata, and create containers. It also serves file or container listings to web browsers. The proxy server takes each request and looks up locations for the account, container, or object and routes the requests correctly. The proxy server also han-dles API requests. To improve performance, the proxy server can use an optional cache that is usually deployed with memcache.
Account servers ( swift-ac-count-server)
Manages accounts defined with Object Storage.
Note
Accounts are the root storage location for data.
Container servers ( swift-con-tainer-server)
Manages the mapping of containers or folders, within Object Storage.
Note
Containers are user-defined segments of the account namespace that provides the stor-age location where objects are found. Object servers (
swift-ob-ject-server)
Manages actual objects, such as files, on the storage nodes.
Note
Objects are the actual data stored in Object Storage (swift).
Ring A set of hash tables that associate each object to a spe-cific physical device. There is one ring per type of da-ta manipulated by Object Storage (Objects, Conda-tainers and Accounts). The set of rings are shared among every
Object Storage node (storage and proxy). Each ring de-termines the physical devices (hard disks) where each object, container, and account will be stored. The num-ber of devices on which an object is stored depends on the number of replicas (copies) specified for the Object Storage cluster.
Various periodic processes Perform housekeeping tasks on the large data store. The replication services ensure consistency and availabil-ity through the cluster. Other periodic processes include auditors, updaters, and reapers.
WSGI middleware Handles authentication (usually OpenStack Identity). Object Storage integrates with the Compute layer and can be used as a back end for Im-age Service (glance), providing a horizontally scalable store for all imIm-age and snapshot da-ta. Object Storage focuses on storing non-transactional dada-ta. Object Storage data (the ac-counts, containers, and objects) are resources that are stored on physical hardware. Object Storage proxy containers are installed on specific infrastructure nodes. The oth-er Object Storage nodes are storage nodes and can have any combination (at a device or drive level) of object, container, or account servers running on them. For example, given two servers with five drives in each, each drive can be part of the ring for any combination of object, account, or container (including each having different storage policies within the object server).
Object Storage uses the following nodes:
• Three existing infrastructure nodes to run the swift-proxy-server processes. The proxy servers proxy requests to the appropriate storage nodes.
• Multiple storage nodes that run the swift-account-server, swift-contain-er-server, and swift-object-server processes which control storage of the ac-count databases, the container databases, as well as the actual stored objects.
The following Object Storage features are not supported in Rackspace Private Cloud Object Storage:
• Ring management and ring storage repositories
3
The amount of disk space depends on how much can fit into the rack efficiently. At Rackspace, storage servers are generic 4U servers with 24 2T SATA drives and 8 cores of processing power. RAID on the storage drives is not required and is not recommend-ed. Object Storage's disk usage pattern is unsuitable for RAID. Operating system Rackspace Private Cloud Object Storage runs on Ubuntu.
Software Rackspace Private Cloud Object Storage requires Rackspace Private Cloud v10 Software (Juno).
Networking 1 Gbps or 10 Gbps is suggested internally. For Rackspace Private Cloud Object Storage, an external network should connect any-thing external to the proxy servers. The storage network is intend-ed to be isolatintend-ed on a private network or multiple private net-works.
Database A SQLite database is part of the Rackspace Private Cloud Object Storage container and account management process.
Permissions Rackspace Private Cloud Object Storage can be installed with either root permissions or with as a user with sudo permissions if the su-doers file is configured to enable all the permissions.
1.2. Installation workflow
The following diagram shows the general workflow for an Object Storage installation.
2. Networking for Object Storage
This section offers recommendations for network resources and provides guidelines to help network administrators understand the recommended networks and public IP addresses. Bandwidth of at least 1 Gbps is suggested.
Object Storage uses the following networks: Public network (Publicly routable
IP range)
• Mandatory network that connects to the proxy serv-er.
• Provides public IP accessibility to the API endpoints within the cloud infrastructure.
• Minimum size: One IP address for each proxy server. Storage network (RFC1918 IP
Range, not publicly routable)
• Must be specified in the rpc_user_config.yml file.
Note
The used_ips section in
rpc_user_config.yml must include the storage IPs of the swift storage and management networks, so they are not allocated to a container.
• Add swift_proxy to the "group_binds" variable for the storage network.
• Mandatory network that is not accessible from out-side the cluster. All nodes connect to this network. • Manages all inter-server communications within the
Object Storage infrastructure.
• Minimum size: One IP address for each storage node and proxy server.
expan-5
By default, all of the Object Storage services, as well as the rsync daemon on the storage nodes, are configured to listen on their STORAGE_LOCAL_NET IP addresses.
When configuring a replication network in the ring, the account, container, and object servers listen on both the STORAGE_LOCAL_NET and STORAGE_REPLICATION_NET IP addresses. The rsync daemon only listens to the STORAGE_REPLICATION_NET IP ad-dress.
3. Example of Object Storage installation
architecture
This section provides an example Object Storage installation architecture. Object Storage uses the following constructs:
Node A host machine that runs one or more OpenStack Object Storage services.
Proxy node Runs proxy services.
Storage node Runs account, container, and Object services. Contains the SQLite databases.
Ring A set of mappings between OpenStack Object Storage data and physical devices.
Replica A copy of an object. By default, three copies are maintained in the cluster.
Zone A logically separate section of the cluster. Related to independent failure characteristics.
Region (optional) A logically separate section of the cluster representing distinct physical locations such as cities or countries. Similar to zones, but regions represent the physical locations of portions of the cluster. To increase reliability and performance, additional proxy servers can be added.
The ring guarantees that every replica is stored in a separate zone. This is applicable only if the zones are greater than or equal to the replica count. A zone is a group of nodes that are as isolated as possible from other nodes (separate servers, network, power, even geog-raphy). The following diagram shows one possible configuration for a minimal installation. In this example, each storage node is a separate zone in the ring. At a minimum, five zones are recommended.
7
4. Prerequisites
Enable the trusty-backports repository. The trusty-backports repository is required to in-stall Object Storage. Add repository details in /etc/apt/sources.list, and update the package list:
$ cd /opt/openstack-ansible/rpc_deployment # ansible hosts -m shell -a "sed -r -i 's/^# \
(deb.*trusty-backports.*)$/\1/' /etc/apt/sources.list; apt-get update"
Note
If using a version prior to 10.1.14, use the
/opt/openstack-ansi-ble/rpc_deployment directory for any new deployments. If any update di-rectories were cloned prior to the Juno release, or any local clones that have not been manually updated, use the /opt/os-ansible-deployment direc-tory.
9
5. Configure and mount storage devices
This section offers a set of prerequisite instructions for setting up Object Storage storage devices. The storage devices must be set up before installing Object Storage.
Procedure 5.1. Configuring and mounting storage devices
RPC Object Storage requires a minimum of three Object Storage devices with mounted stor-age drives. The example commands in this procedure assume the storstor-age devices for Object Storage are devices sdc through sdg.
1. Determine the storage devices on the node to use for Object Storage.
2. Format each device on the node used for storage with XFS. While formatting the de-vices, add a unique label for each device.
Note
Without labels, a failed drive can cause mount points to shift and data to become inaccessible.
For example, create the file systems on the devices using the mkfs command $ apt-get install xfsprogs
$ mkfs.xfs -f -i size=1024 -L sdc /dev/sdc $ mkfs.xfs -f -i size=1024 -L sdd /dev/sdd $ mkfs.xfs -f -i size=1024 -L sde /dev/sde $ mkfs.xfs -f -i size=1024 -L sdf /dev/sdf $ mkfs.xfs -f -i size=1024 -L sdg /dev/sdg
3. Add the mount locations to the /etc/fstab file so that the storage devices are re-mounted on boot. The following example mount options are recommended when us-ing XFS.
Finish all modifications to the /etc/fstab file before mounting the new filesystems created within the storage devices.
LABEL=sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8, nobootwait 0 0
LABEL=sdd /srv/node/sdd xfs noatime,nodiratime,nobarrier,logbufs=8, nobootwait 0 0
LABEL=sde /srv/node/sde xfs noatime,nodiratime,nobarrier,logbufs=8, nobootwait 0 0
LABEL=sdf /srv/node/sdf xfs noatime,nodiratime,nobarrier,logbufs=8, nobootwait 0 0
LABEL=sdg /srv/node/sdg xfs noatime,nodiratime,nobarrier,logbufs=8, nobootwait 0 0
4. Create the mount points for the devices using the mkdir command. $ mkdir -p /srv/node/sdc
$ mkdir -p /srv/node/sdd $ mkdir -p /srv/node/sde $ mkdir -p /srv/node/sdf
$ mkdir -p /srv/node/sdg
The mount point is referenced as the mount_point parameter in the swift.yml file (/etc/rpc_deploy/conf.d/swift.yml). $ mount /srv/node/sdc $ mount /srv/node/sdd $ mount /srv/node/sde $ mount /srv/node/sdf $ mount /srv/node/sdg
To view an annotated example of the swift.yml file, see Appendix A, Object Storage
configuration files [25].
For the following mounted devices:
Table 5.1. Mounted devices
Device Mount location
/dev/sdc /srv/node/sdc
/dev/sdd /srv/node/sdd
/dev/sde /srv/node/sde
/dev/sdf /srv/node/sdf
/dev/sdg /srv/node/sdg
The entry in the swift.yml would be:
# drives: # - name: sdc # - name: sdd # - name: sde # - name: sdf # - name: sdg # mount_point: /srv/node
11
6. Configure an Object Storage
deployment
Object Storage is configured using the /etc/rpc_deploy/conf.d/swift.yml file and the /etc/rpc_deploy/user_variables.yml file.
The group variables in the /etc/rpc_deploy/conf.d/swift.yml file are used by the Ansible playbooks when installing Object Storage. Some variables cannot be changed af-ter they are set, while some changes require re-running the playbooks. The values in the swift_hosts section supersede values in the swift section.
To view the configuration files, including information about which variables are required and which are optional, see Appendix A, Object Storage configuration files [25].
6.1. Configuring Object Storage
Procedure 6.1. Updating the Object Storage configuration
swift.yml
file
1. Update the global override values: # global_overrides: # swift: # part_power: 8 # weight: 100 # min_part_hours: 1 # repl_number: 3 # storage_network: 'br-storage' # replication_network: 'br-repl' # drives: # - name: sdc # - name: sdd # - name: sde # - name: sdf # mount_point: /mnt # account: # container: # storage_policies: # - policy: # name: gold # index: 0 # default: True # - policy: # name: silver # index: 1 # repl_number: 3 # deprecated: True
part_power Set the partition power value based on the total amount of storage the entire ring will use.
Multiply the maximum number of drives ever used with this Object Storage installation by 100 and round that val-ue up to the closest power of two valval-ue. For example, a
maximum of six drives, times 100, equals 600. The nearest power of two above 600 is two to the power of nine, so the partition power is nine. The partition power cannot be changed after the Object Storage rings are built.
weight The default weight is 100. If the drives are different sizes, set the weight value to avoid uneven distribution of da-ta. For example, a 1 TB disk would have a weight of 100, while a 2 TB drive would have a weight of 200.
min_part_hours The default value is 1. Set the minimum partition hours to the amount of time to lock a partition's replicas after a partition has been moved. Moving multiple replicas at the same time might make data inaccessible. This value can be set separately in the swift, container, account, and policy sections with the value in lower sections superseding the value in the swift section.
repl_number The default value is 3. Set the replication number to the number of replicas of each object. This value can be set separately in the swift, container, account, and policy sec-tions with the value in the more granular secsec-tions super-seding the value in the swift section.
storage_network By default, the swift services will listen on the default man-agement IP. Optionally, specify the interface of the stor-age network.
Note
If the storage_network is not set, but the storage_ips per host are set (or the storage_ip is not on the
storage_network interface) the proxy serv-er will not be able to connect to the storage services.
inter-13
drives Set the default drives per host. This is useful when all hosts have the same drives. These can be overridden on a per host basis.
mount_point Set the mount_point value to the location where the swift drives are mounted. For example, with a mount point of /mnt and a drive of sdc, a drive is mounted at / mnt/sdc on the swift_host. This can be overridden on a per-host basis.
storage_policies Storage policies determine on which hardware data is stored, how the data is stored across that hardware, and in which region the data resides. Each storage policy must have an unique name and a unique index. There must be a storage policy with an index of 0 in the swift.yml file to use any legacy containers created before storage poli-cies were instituted.
default Set the default value to yes for at least one policy. This is the default storage policy for any non-legacy containers that are created.
deprecated Set the deprecated value to yes to turn off storage poli-cies.
Note
For account and container rings,
min_part_hours and repl_number are the only values that can be set. Setting them in this section overrides the defaults for the spe-cific ring.
2. Update the Object Storage proxy hosts values: # swift-proxy_hosts: # infra-node1: # ip: 192.0.2.1 # infra-node2: # ip: 192.0.2.2 # infra-node3: # ip: 192.0.2.3
swift-proxy_hosts Set the IP address of the hosts that Ansible will connect to to deploy the swift-proxy containers. The swift-proxy_hosts value should match the infra nodes. 3. Update the Object Storage hosts values:
# swift_hosts: # swift-node1: # ip: 192.0.2.4 # container_vars: # swift_vars:
# zone: 0 # swift-node2: # ip: 192.0.2.5 # container_vars: # swift_vars: # zone: 1 # swift-node3: # ip: 192.0.2.6 # container_vars: # swift_vars: # zone: 2 # swift-node4: # ip: 192.0.2.7 # container_vars: # swift_vars: # zone: 3 # swift-node5: # ip: 192.0.2.8 # container_vars: # swift_vars: # storage_ip: 198.51.100.8 # repl_ip: 203.0.113.8 # zone: 4 # region: 3 # weight: 200 # groups: # - account # - container # - silver # drives: # - name: sdb # storage_ip: 198.51.100.9 # repl_ip: 203.0.113.9 # weight: 75 # groups: # - gold # - name: sdc # - name: sdd # - name: sde # - name: sdf
swift_hosts Specify the hosts to be used as the storage nodes. The ip is the address of the host to which Ansible connects. Set the name and IP address of each Ob-ject Storage host. The section is not
15
neither are specified, both will default to the host IP address.
Note
Overriding these values on a host or drive basis can cause prob-lems if the IP address that the service listens on is based on a specified storage_network or replication_network and the ring is set to a different IP address.
zone The default is 0. Optionally, set the Object Storage zone for the ring.
region Optionally, set the Object Storage region for the ring.
weight The default weight is 100. If the drives are different sizes, set the weight value to avoid uneven distribu-tion of data. This value can be specified on a host or drive basis (if specified at both, the drive setting takes precedence).
groups Set the groups to list the rings to which a host's drive belongs. This can be set on a per drive basis which will override the host setting.
drives Set the names of the drives on this Object Storage host. At least one name must be specified.
weight The default weight is 100. If the drives are different sizes, set the weight value to avoid uneven distribu-tion of data. This value can be specified on a host or drive basis (if specified at both, the drive setting takes precedence).
In the following example, swift-node5 shows values in the swift_hosts section that will override the global values. Groups are set, which overrides the global settings for drive sdb. The weight is overridden for the host and specifically adjusted on drive sdb. Also, the storage_ip and repl_ip are set differently for sdb.
# swift-node5: # ip: 192.0.2.8 # container_vars: # swift_vars: # storage_ip: 198.51.100.8 # repl_ip: 203.0.113.8 # zone: 4 # region: 3 # weight: 200 # groups: # - account
# - container # - silver # drives: # - name: sdb # storage_ip: 198.51.100.9 # repl_ip: 203.0.113.9 # weight: 75 # groups: # - gold # - name: sdc # - name: sdd # - name: sde # - name: sdf
4. Ensure the swift.yml is in the /etc/rpc_deploy/conf.d/ folder.
Procedure 6.2. Allowing all Identity users to use Object Storage
1. Set swift_allow_all_users in the user_variables.yml file to True. Any users with the _member_ role (all authorized Identity (keystone) users) can create con-tainers and upload objects to Object Storage.
Note
If this value is False, then by default, only users with the admin or swift-operator role are allowed to create containers or manage tenants.
When the backend type for the Image Service (glance) is set to swift, the Image Service can access the Object Storage cluster regardless of whether this value is True or False.
2. Run the Identity playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/openstack/keystone.yml
6.2. Storage Policies
Storage Policies allow segmenting the cluster for various purposes through the creation of multiple object rings. Using policies, different devices can belong to different rings with varying levels of replication. By supporting multiple object rings, Object Storage can segre-gate the objects within a single cluster.
17
• Differing storage implementations: A policy can be used to direct traffic to collected nodes that use a different disk file (for example, Kinetic, GlusterFS).
Most storage clusters do not require more than one storage policy. The following problems can occur if using multiple storage policies per cluster:
• Creating a second storage policy without any specified drives (all drives are part of only the account, container, and default storage policy groups) creates an empty ring for that storage policy.
• A non-default storage policy is used only if specified when creating a container, using the X-Storage-Policy: <policy-name> header. After the container is created, it uses the created storage policy. Other containers continue using the default or another stor-age policy specified when created.
7. Object Storage playbooks
The Ansible Object Storage playbooks prepare the target hosts for Object Storage services and depend on the values in the swift.yml file. The playbooks consist of the following operations:
•host-setup: This play sets up the rsyslog containers on the new storage hosts, the swift-proxy containers on the infrastructure hosts, and additional initial setup of the hosts.
•rsyslog-install: This play installs the rsyslog containers on the Object Storage hosts.
•rsyslog-config: This play configures the rsyslog containers on the Object Storage hosts.
•swift-all: This play runs all the other Object Storage plays. It is used for new installs or it can be run to propagate any changes that have been made to the swift.yml file. The following are the individual plays within swift-all:
•swift-common: This play runs the common Object Storage setup plays (for example, general configuration for hosts, creating required directories, and installing packages). •swift-build-rings: This play builds the Object Storage rings (based on the
rpc_inventory.json file which is created from the swift.yml). This play can be rerun to adjust the rings and redistribute the rings to the Object Storage nodes based on changes to the swift.yml file.
•swift-proxy: This play installs the init_scripts/conf files/service for swift-proxy. Because the swift-proxy service is not installed as a Object Storage service it is installed on the swift proxy hosts instead of the Object Storage hosts. •swift-storage: This play runs the swift-account, swift-container, and swift-object
plays which result in the services/init_scripts and config files for the Object Storage services being installed, configured, and started. The following list is the names of the individual plays within swift-storage:
•swift-account: This play installs the init_scripts/conf files/service for the account server.
19 2. Run the host-setup playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/setup/host-setup.yml
3. Run the rsyslog-install playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/infrastructure/rsyslog-install.yml
4. Run the rsyslog-config playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/infrastructure/rsyslog-config.yml
5. Run the swift-all playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/openstack/swift-all.yml
8. Verify the installation
These commands can be run from the proxy server or any server that has access to the Identity Service.
1. Ensure that the credentials are set correctly in the /root/openrc file and then source it:
$ source /root/openrc 2. Run the following command:
$ swift stat
Account: AUTH_11b9758b7049476d9b48f7a91ea11493 Containers: 0
Objects: 0 Bytes: 0
Content-Type: text/plain; charset=utf-8 X-Timestamp: 1381434243.83760
X-Trans-Id: txdcdd594565214fb4a2d33-0052570383 X-Put-Timestamp: 1381434243.83760
3. Run the following commands to upload files to a container. Create the test.txt and test2.txt test files locally if needed (can contain anything).
$ swift upload myfiles test.txt $ swift upload myfiles test2.txt
4. Run the following command to download all files from the myfiles container:
$ swift download myfiles
test2.txt [headers 0.267s, total 0.267s, 0.000s MB/s] test.txt [headers 0.271s, total 0.271s, 0.000s MB/s]
21
9. Integrate Object Storage with the
Image Service
The images created by the Image Service (glance) can be optionally stored using Object Storage.
Note
If there is an existing Image Service (glance) backend (for example, cloud files) but want to add Object Storage (swift) to use as the Image Service back end, re-add any images from the Image Service after moving to Object Storage. If the Image Service variables are changed (as described below) and begin using Object storage, any images in the Image Service will no longer be available.
Procedure 9.1. Integrating Object Storage with Image Service
This procedure requires the following: • RPC v10 (Juno)
• Object Storage v 2.2.0
1. Update the glance options in the /etc/rpc_deploy/user_variables.yml file:
# Glance Options glance_default_store: swift glance_swift_store_auth_address: '{{ auth_identity_uri }}' glance_swift_store_container: glance_images glance_swift_store_endpoint_type: internalURL glance_swift_store_key: '{{ glance_service_password }}' glance_swift_store_region: RegionOne glance_swift_store_user: 'service:glance'
•glance_default_store: Set the default store to swift.
•glance_swift_store_auth_address: Set to the local authentication address using the '{{ auth_identity_uri }}' variable.
•glance_swift_store_container: Set the container name.
•glance_swift_store_endpoint_type: Set the endpoint type to inter-nalURL.
•glance_swift_store_key: Set the Image Service password using the '{{ glance_service_password }}' variable.
•glance_swift_store_region: Set the region. The default value is RegionOne. •glance_swift_store_user: Set the tenant and user name to
'service:glance'.
3. Run the Image Service (glance) playbook:
$ ansible-playbook -e @/etc/rpc_deploy/user_variables.yml \ playbooks/openstack/glance-all.yml
23
10. Object Storage monitoring
Rackspace Cloud Monitoring Service allows Rackspace Private Cloud (RPC) customers to monitor system performance, and safeguard critical data.
10.1. Service and response
When a threshold is reached or functionality fails, the Rackspace Cloud Monitoring Ser-vice generates an alert, which creates a ticket in the Rackspace ticketing system. This ticket moves into the RPC support queue. Tickets flagged as monitoring alerts are given highest priority, and response is delivered according to the Service Level Agreement (SLA). Refer to the SLA for detailed information about incident severity levels and corresponding response times.
Specific monitoring alert guidelines can be set for the installation. These details should be arranged by a Rackspace account manager.
10.2. Object Storage monitors
Object Storage has its own set of monitors and alerts. For more information about in-stalling the monitoring tools, see the Rackspace Private Cloud Installation Guide. The following checks are performed on services:
• Health checks on the services on each server.
Object Storage makes a request to /healthcheck for each service to ensure it is re-sponding appropriately. If this check fails, determine why the service is failing and fix it accordingly.
• Health check against the proxy service on the virtual IP (VIP).
Object Storage checks the load balancer address for the swift-proxy-server service, and monitors the proxy servers as a whole rather than individually which is managed by the individual service health checks. If this check fails, it suggests that there is no access to the VIP or all of the services are failing.
The following checks are performed against the output of the swift-recon middleware: • md5sum checks on the ring files across all Object Storage nodes.
This check ensures that the ring files are the same on each node. If this check fails, deter-mine why the md5sum for the ring is different and deterdeter-mine which of the ring files is correct. Copy the correct ring file onto the node that is causing the md5sum to fail. • md5sum checks on the swift.conf across all swift nodes.
If this check fails, determine why the swift.conf is different and determine which of the swift.conf is correct. Copy the correct swift.conf onto the node that is causing the md5sum to fail.
• Asyncs pending
This check monitors the average number of async pending requests and the percentage that are put in async pending. This happens when a PUT or DELETE fails (due to, for ex-ample, timeouts, heavy usage, failed disk). If this check fails, determine why requests are failing and being put in async pending status and fix accordingly.
• Quarantine
This check monitors the percentage of objects that are quarantined (objects that are found to have errors and moved to quarantine). An alert is set up against account, con-tainer, and object servers. If this fails, determine the cause of the corrupted objects and fix accordingly.
• Replication
This check monitors replication success percentage. An alert is set up against account, container, and object servers. If this fails, determine why objects are not replicating and fix accordingly.
25
Appendix A. Object Storage
configuration files
Table of Contents
A.1. swift.yml example configuration file ... 25 A.2. user_variables.yml configuration file ... 32 Object Storage is configured using the /etc/rpc_deploy/conf.d/swift.yml file and the /etc/rpc_deploy/user_variables.yml file. This appendix shows both configu-ration files and indicates which variables are required and which are optional.
A.1.
swift.yml
example configuration file
---# Copyright 2015, Rackspace US, Inc. #
# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0 #
# Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
#
# Overview # ======== #
# This file contains the configuration for the OpenStack Ansible Deployment # (OSA) Object Storage (swift) service. Only enable these options for
# deployments that contain the Object Storage service. For more information on # these options, see the documentation at
#
# http://docs.openstack.org/developer/swift/index.html #
# You can customize the options in this file and copy it to # /etc/openstack_deploy/conf.d/swift.yml or create a new # file containing only necessary options for your environment # before deployment.
#
# OSA implements PyYAML to parse YAML files and therefore supports structure # and formatting options that augment traditional YAML. For example, aliases # or references. For more information on PyYAML, see the documentation at #
# http://pyyaml.org/wiki/PyYAMLDocumentation #
# Configuration reference # =======================
#
# Level: global_overrides (required)
# Contains global options that require customization for a deployment. For # example, the ring stricture. This level also provides a mechanism to # override other options defined in the playbook structure.
#
# Level: swift (required) # Contains options for swift. #
# Option: storage_network (required, string)
# Name of the storage network bridge on target hosts. Typically # 'br-storage'.
#
# Option: repl_network (optional, string)
# Name of the replication network bridge on target hosts. Typically # 'br-repl'. Defaults to the value of the 'storage_network' option. #
# Option: part_power (required, integer)
# Partition power. Applies to all rings unless overridden at the 'account' # or 'container' levels or within a policy in the 'storage_policies' level.
# Immutable without rebuilding the rings. #
# Option: repl_number (optional, integer)
# Number of replicas for each partition. Applies to all rings unless # overridden at the 'account' or 'container' levels or within a policy # in the 'storage_policies' level. Defaults to 3.
#
# Option: min_part_hours (optional, integer)
# Minimum time in hours between multiple moves of the same partition. # Applies to all rings unless overridden at the 'account' or 'container' # levels or within a policy in the 'storage_policies' level. Defaults # to 1.
#
# Option: region (optional, integer)
# Region of a disk. Applies to all disks in all storage hosts unless # overridden deeper in the structure. Defaults to 1.
#
# Option: zone (optional, integer)
# Zone of a disk. Applies to all disks in all storage hosts unless # overridden deeper in the structure. Defaults to 0.
#
# Option: weight (optional, integer)
# Weight of a disk. Applies to all disks in all storage hosts unless # overridden deeper in the structure. Defaults to 100.
27 # # swift: # storage_network: 'br-storage' # replication_network: 'br-repl' # part_power: 8 # repl_number: 3 # min_part_hours: 1 # region: 1 # zone: 0 # weight: 100 #
# Note: Most typical deployments override the 'zone' option in the # 'swift_vars' level to use a unique zone for each storage host. #
# Option: mount_point (required, string)
# Top-level directory for mount points of disks. Defaults to /mnt. # Applies to all hosts unless overridden deeper in the structure. #
# Level: drives (required)
# Contains the mount points of disks.
# Applies to all hosts unless overridden deeper in the structure. #
# Option: name (required, string)
# Mount point of a disk. Use one entry for each disk.
# Applies to all hosts unless overridden deeper in the structure. #
# Example: #
# Mount disks 'sdc', 'sdd', 'sde', and 'sdf' to the '/mnt' directory on all # storage hosts: # # mount_point: /mnt # drives: # - name: sdc # - name: sdd # - name: sde # - name: sdf #
# Level: account (optional)
# Contains 'min_part_hours' and 'repl_number' options specific to the # account ring.
#
# Level: container (optional)
# Contains 'min_part_hours' and 'repl_number' options specific to the # container ring.
#
# Level: storage_policies (required)
# Contains storage policies. Minimum one policy. One policy must include # the 'index: 0' and 'default: True' options.
#
# Level: policy (required)
# Contains a storage policy. Define for each policy. #
# Option: name (required, string) # Policy name.
#
# Option: index (required, integer)
# Policy index. One policy must include this option with a '0' # value.
#
# Option: policy_type (optional, string)
# Defines policy as replication or erasure coding. Accepts 'replication'
# 'erasure_coding' values. Defaults to 'replication' value if omitted. #
# Option: ec_type (conditionally required, string)
# Defines the erasure coding algorithm. Required for erasure coding # policies.
#
# Option: ec_num_data_fragments (conditionally required, integer) # Defines the number of object data fragments. Required for erasure # coding policies.
#
# Option: ec_num_parity_fragments (conditionally required, integer) # Defines the number of object parity fragments. Required for erasure # coding policies.
#
# Option: ec_object_segment_size (conditionally required, integer) # Defines the size of object segments in bytes. Swift sends incoming # objects to an erasure coding policy in segments of this size. # Required for erasure coding policies.
#
# Option: default (conditionally required, boolean)
# Defines the default policy. One policy must include this option # with a 'True' value.
#
# Option: deprecated (optional, boolean) # Defines a deprecated policy.
#
# Note: The following levels and options override any values higher # in the structure and generally apply to advanced deployments. #
# Option: repl_number (optional, integer)
# Number of replicas of each partition in this policy. #
# Option: min_part_hours (optional, integer)
# Minimum time in hours between multiple moves of the same partition # in this policy.
#
# Example: #
# Define three storage policies: A default 'gold' policy, a deprecated # 'silver' policy, and an erasure coding 'ec10-4' policy.
29 # ec_num_data_fragments: 10 # ec_num_parity_fragments: 4 # ec_object_segment_size: 1048576 # # ---#
# Level: swift_proxy-hosts (required)
# List of target hosts on which to deploy the swift proxy service. Recommend # three minimum target hosts for these services. Typically contains the same # target hosts as the 'shared-infra_hosts' level in complete OpenStack # deployments.
#
# Level: <value> (optional, string) # Name of a proxy host.
#
# Option: ip (required, string)
# IP address of this target host, typically the IP address assigned to # the management bridge.
#
# Level: container_vars (optional) # Contains options for this target host. #
# Level: swift_proxy_vars (optional)
# Contains swift proxy options for this target host. Typical deployments # use this level to define read/write affinity settings for proxy hosts. #
# Option: read_affinity (optional, string)
# Specify which region/zones the proxy server should prefer for reads # from the account, container and object services.
# E.g. read_affinity: "r1=100" this would prefer region 1
# read_affinity: "r1z1=100, r1=200" this would prefer region 1 zone 1 # if that is unavailable region 1, otherwise any available region/ zone.
# Lower number is higher priority. When this option is specified the # sorting_method is set to 'affinity' automatically.
#
# Option: write_affinity (optional, string)
# Specify which region to prefer when object PUT requests are made. # E.g. write_affinity: "r1" - favours region 1 for object PUTs #
# Option: write_affinity_node_count (optional, string)
# Specify how many copies to prioritise in specified region on # handoff nodes for Object PUT requests.
# Requires "write_affinity" to be set in order to be useful. # This is a short term way to ensure replication happens locally, # Swift's eventual consistency will ensure proper distribution over # time.
# e.g. write_affinity_node_count: "2 * replicas" - this would try to # store Object PUT replicas on up to 6 disks in region 1 assuming # replicas is 3, and write_affinity = r1
#
# Example: #
# Define three swift proxy hosts: # # swift_proxy-hosts: # # infra1: # ip: 172.29.236.101 # container_vars:
# swift_proxy_vars: # read_affinity: "r1=100" # write_affinity: "r1" # write_affinity_node_count: "2 * replicas" # infra2: # ip: 172.29.236.102 # container_vars: # swift_proxy_vars: # read_affinity: "r2=100" # write_affinity: "r2" # write_affinity_node_count: "2 * replicas" # infra3: # ip: 172.29.236.103 # container_vars: # swift_proxy_vars: # read_affinity: "r3=100" # write_affinity: "r3" # write_affinity_node_count: "2 * replicas" # # ---#
# Level: swift_hosts (required)
# List of target hosts on which to deploy the swift storage services. # Recommend three minimum target hosts for these services.
#
# Level: <value> (required, string) # Name of a storage host.
#
# Option: ip (required, string)
# IP address of this target host, typically the IP address assigned to # the management bridge.
#
# Note: The following levels and options override any values higher # in the structure and generally apply to advanced deployments. #
# Level: container_vars (optional) # Contains options for this target host. #
# Level: swift_vars (optional)
# Contains swift options for this target host. Typical deployments # use this level to define a unique zone for each storage host. #
# Option: storage_ip (optional, string)
# IP address to use for accessing the account, container, and object # services if different than the IP address of the storage network
31 # Weight of all disks. #
# Level: groups (optional)
# List of one of more Ansible groups that apply to this host. #
# Example: #
# Deploy the account ring, container ring, and 'silver' policy. # # groups: # - account # - container # - silver #
# Level: drives (optional)
# Contains the mount points of disks specific to this host. #
# Level or option: name (optional, string)
# Mount point of a disk specific to this host. Use one entry for # each disk. Functions as a level for disks that contain additional # options.
#
# Option: storage_ip (optional, string)
# IP address to use for accessing the account, container, and object # services of a disk if different than the IP address of the storage # network bridge on the target host. Also requires manual
# configuration of the host. #
# Option: repl_ip (optional, string)
# IP address to use for replication services of a disk if different # than the IP address of the replication network bridge on the target
# host. Also requires manual configuration of the host. #
# Option: region (optional, integer) # Region of a disk.
#
# Option: zone (optional, integer) # Zone of a disk.
#
# Option: weight (optional, integer) # Weight of a disk.
#
# Level: groups (optional)
# List of one or more Ansible groups that apply to this disk. #
# Example: #
# Define four storage hosts. The first three hosts contain typical options # and the last host contains advanced options.
# # swift_hosts: # swift-node1: # ip: 172.29.236.151 # container_vars: # swift_vars: # zone: 0 # swift-node2: # ip: 172.29.236.152 # container_vars:
# swift_vars: # zone: 1 # swift-node3: # ip: 172.29.236.153 # container_vars: # swift_vars: # zone: 2 # swift-node4: # ip: 172.29.236.154 # container_vars: # swift_vars: # storage_ip: 198.51.100.11 # repl_ip: 203.0.113.11 # region: 2 # zone: 0 # weight: 200 # groups: # - account # - container # - silver # drives: # - name: sdc # storage_ip: 198.51.100.21 # repl_ip: 203.0.113.21 # weight: 75 # groups: # - gold # - name: sdd # - name: sde # - name: sdf
A.2.
user_variables.yml
configuration file
---# Copyright 2014, Rackspace US, Inc. #
# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at
33 glance_ceilometer_enabled: False nova_ceilometer_enabled: False ## Aodh Options aodh_db_type: mongodb aodh_db_ip: localhost aodh_db_port: 27017 ## Glance Options
# Set glance_default_store to "swift" if using Cloud Files or swift backend # or "rbd" if using ceph backend; the latter will trigger ceph to get # installed on glance
glance_default_store: file glance_notification_driver: noop
# `internalURL` will cause glance to speak to swift via ServiceNet, use # `publicURL` to communicate with swift over the public network
glance_swift_store_endpoint_type: internalURL
# Ceph client user for glance to connect to the ceph cluster #glance_ceph_client: glance
# Ceph pool name for Glance to use #glance_rbd_store_pool: images #glance_rbd_store_chunk_size: 8 ## Nova
# When nova_libvirt_images_rbd_pool is defined, ceph will be installed on nova # hosts.
#nova_libvirt_images_rbd_pool: vms
# by default we assume you use rbd for both cinder and nova, and as libvirt # needs to access both volumes (cinder) and boot disks (nova) we default to # reuse the cinder_ceph_client
# only need to change this if you'd use ceph for boot disks and not for volumes
#nova_ceph_client: #nova_ceph_client_uuid:
# This defaults to KVM, if you are deploying on a host that is not KVM capable # change this to your hypervisor type: IE "qemu", "lxc".
# nova_virt_type: kvm
# nova_cpu_allocation_ratio: 2.0 # nova_ram_allocation_ratio: 1.0
# If you wish to change the dhcp_domain configured for both nova and neutron # dhcp_domain:
## Glance with Swift
### Extra options when configuring swift as a glance back-end. ### By default it will use the local swift install
### Set these when using a remote swift as a glance backend #glance_swift_store_auth_address: "https://some.auth.url.com" #glance_swift_store_user: "OPENSTACK_TENANT_ID:OPENSTACK_USER_NAME" #glance_swift_store_key: "OPENSTACK_USER_PASSWORD" #glance_swift_store_container: "NAME_OF_SWIFT_CONTAINER" #glance_swift_store_region: "NAME_OF_REGION" ## Cinder
# Ceph client user for cinder to connect to the ceph cluster #cinder_ceph_client: cinder
## Ceph
# Enable these if you use ceph rbd for at least one component (glance, cinder, nova)
#ceph_apt_repo_url_region: "www" # or "eu" for Netherlands based mirror #ceph_stable_release: hammer
# Ceph Authentication - by default cephx is true #cephx: true
# Ceph Monitors
# A list of the IP addresses for your Ceph monitors #ceph_mons:
# - 10.16.5.40 # - 10.16.5.41 # - 10.16.5.42
# Custom Ceph Configuration File (ceph.conf)
# By default, your deployment host will connect to one of the mons defined above to
# obtain a copy of your cluster's ceph.conf. If you prefer, uncomment ceph_conf_file
# and customise to avoid ceph.conf being copied from a mon. #ceph_conf_file: | # [global] # fsid = 00000000-1111-2222-3333-444444444444 # mon_initial_members = mon1.example.local,mon2.example.local,mon3.example. local # mon_host = 10.16.5.40,10.16.5.41,10.16.5.42
# # optionally, you can use this construct to avoid defining this list twice: # # mon_host = {{ ceph_mons|join(',') }}
# auth_cluster_required = cephx # auth_service_required = cephx
## SSL Settings
# Adjust these settings to change how SSL connectivity is configured for # various services. For more information, see the openstack-ansible # documentation section titled "Securing services with SSL certificates". #
## SSL: Keystone
# These do not need to be configured unless you're creating certificates for # services running behind Apache (currently, Horizon and Keystone).
ssl_protocol: "ALL -SSLv2 -SSLv3"
# Cipher suite string from https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl_cipher_suite: "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH +AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
35
# RabbitMQ over encrypted connections on port 5671. To opt-out of this default,
# set the rabbitmq_use_ssl variable to 'false'. The default setting of 'true' # is highly recommended for securing the contents of RabbitMQ messages. #rabbitmq_use_ssl: true
## Additional pinning generator that will allow for more packages to be pinned as you see fit.
## All pins allow for package and versions to be defined. Be careful using this as versions
## are always subject to change and updates regarding security will become your problem from this
## point on. Pinning can be done based on a package version, release, or origin. Use "*" in the
## package name to indicate that you want to pin all package to a particular constraint.
# apt_pinned_packages:
# - { package: "lxc", version: "1.0.7-0ubuntu0.1" }
# - { package: "libvirt-bin", version: "1.2.2-0ubuntu13.1.9" } # - { package: "rabbitmq-server", origin: "www.rabbitmq.com" } # - { package: "*", release: "MariaDB" }
## Environment variable settings
# This allows users to specify the additional environment variables to be set # which is useful in setting where you working behind a proxy. If working behind
# a proxy It's important to always specify the scheme as "http://". This is what
# the underlying python libraries will handle best. This proxy information will be
# placed both on the hosts and inside the containers. ## Example environment variable setup:
# proxy_env_url: http://username:[email protected]:9000/
# no_proxy_env: "localhost,127.0.0.1,{% for host in groups['all_containers'] %}{{ hostvars[host]['container_address'] }}{% if not loop.last %},{% endif %} {% endfor %}"
# global_environment_variables: # HTTP_PROXY: "{{ proxy_env_url }}" # HTTPS_PROXY: "{{ proxy_env_url }}" # NO_PROXY: "{{ no_proxy_env }}"
## Multiple region support in Horizon:
# For multiple regions uncomment this configuration, and # add the extra endpoints below the first list item. # horizon_available_regions:
# - { url: "{{ keystone_service_internalurl }}", name: "{{ keystone_service_region }}" }
# - { url: "http://cluster1.example.com:5000/v2.0", name: "RegionTwo" } ## SSH connection wait time
# If an increased delay for the ssh connection check is desired, # uncomment this variable and set it appropriately.
#ssh_delay: 5 ## HAProxy
# Uncomment this to disable keepalived installation (cf. documentation) #haproxy_use_keepalived: False
#
# HAProxy Keepalived configuration (cf. documentation)
haproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}/32" haproxy_keepalived_internal_vip_cidr: "{{internal_lb_vip_address}}/32" #haproxy_keepalived_external_interface:
#haproxy_keepalived_internal_interface:
# Defines the default VRRP id used for keepalived with haproxy. # Overwrite it to your value to make sure you don't overlap
# with existing VRRPs id on your network. Default is 10 for the external and 11 for the
# internal VRRPs
#haproxy_keepalived_external_virtual_router_id: #haproxy_keepalived_internal_virtual_router_id:
# Defines the VRRP master/backup priority. Defaults respectively to 100 and 20 #haproxy_keepalived_priority_master:
#haproxy_keepalived_priority_backup:
# All the previous variables are used in a var file, fed to the keepalived role.
# To use another file to feed the role, override the following var: #haproxy_keepalived_vars_file: 'vars/configs/keepalived_haproxy.yml'
37
11. Additional resources
These additional resources are designed to help you learn more about the Rackspace Pri-vate Cloud Software and OpenStack.
• If you are an advanced user and are comfortable with APIs, the OpenStack API documen-tation is available in the OpenStack API Documentation library.
•OpenStack API Quick Start
•Programming OpenStack Compute API
•OpenStack Compute Developer Guide
•Rackspace Private Cloud Knowledge Center
•OpenStack Manuals
•OpenStack - Nova Developer Documentation
•OpenStack - Glance Developer Documentation
•OpenStack - Keystone Developer Documentation
•OpenStack - Horizon Developer Documentation
•OpenStack - Cinder Developer Documentation
•OpenStack - Swift Developer Documentation
11.1. Document Change History
This version replaces and obsoletes all previous versions. The most recent versions are listed in the following table:
Revision Date Summary of Changes
September 12, 2015 • Rackspace Private Cloud v10.1.14 Software update release September 3, 2015 • Rackspace Private Cloud v10.1.13 Software update release August 15, 2015 • Rackspace Private Cloud v10.1.12 Software update release July 31, 2015 • Rackspace Private Cloud v10.1.11 Software update release July 7, 2015 • Rackspace Private Cloud v10.1.10 Software update release June 19, 2015 • Rackspace Private Cloud v10.1.9 Software update release June 5, 2015 • Rackspace Private Cloud v10.1.8 Software update release June 5, 2015 • Rackspace Private Cloud v10.1.7 Software update release May 15, 2015 • Rackspace Private Cloud v10.1.6 Software update release May 8, 2015 • Rackspace Private Cloud v10.1.5 Software update release April 22, 2015 • Rackspace Private Cloud v10.1.4 Software update release April 10, 2015 • Rackspace Private Cloud v10.1.3 Software update release
February 26, 2015 • Rackspace Private Cloud v10.1.2 Software Unlimited Availability release December 19, 2014 • Rackspace Private Cloud v10.1.1 Software Limited Availability release December 5, 2014 • Rackspace Private Cloud v10.1.0 Software Limited Availability release November 14, 2014 • Rackspace Private Cloud v10.0.0 Software Limited Availability release
