Because the Ceph documentation website can change over time, the installation flow has been sourced from Ceph
documentation used when configuring the sample reference configuration. The sourced instructions have been modified to include customizations made, and fixed on the choices of Ubuntu and Ceph distribution. For additional details, the Ceph quick start instructions are also linked in the ‘for more information’ section.
It’s recommended to perform cluster install where the cluster can get access to the internet. This allows the OS package manager to download Ceph and general OS packages from normal repositories. The ceph-deploy program depends on using the package manager to install, and it is much more straightforward than maintaining internal repositories. If site security doesn’t allow internet access, the installation instructions will have to be modified according to specific site requirements, whether the install uses a local repository or source install.
All samples are from installation on Ubuntu 12.04.3
Naming Conventions
The monitor and object gateway systems are named: hp-cephmon01 through hp-cephmon03 The OSD hosts are named hp-osdhost01 through hp-osdhost10
When an operation is generic to the type of system, it’ll be referred to as <node01> through <nodexx>
Ceph Deploy Setup
Initial cluster creation and staging is executed from the first monitor, hp-cephmon01.
Add the release key. You may need to edit wgetrc for proxy use; see Initial Configuration Modification below for syntax: wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
Add Ceph Packages to the repository:
echo deb http://ceph.com/debian-dumpling/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list Update the repository and install ceph-deploy:
sudo apt-get update && sudo apt-get install ceph-deploy
Ceph Node Setup
Create a user on each Ceph node: ssh <existing login user>@ceph-server
sudo useradd -d /home/ceph -m ceph -s /bin/bash sudo passwd ceph
Add root privileges for the user on each Ceph node:
echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph sudo chmod 0440 /etc/sudoers.d/ceph
Install ssh server (if necessary) on each Ceph Node: sudo apt-get install openssh-server
Configure the ceph-deploy admin node with password-less SSH access to each Ceph Node. When configuring SSH access, do not use sudo or the root user. Leave the passphrase empty:
ssh-keygen
Generating public/private key pair.
Enter file in which to save the key (/ceph-client/.ssh/id_rsa): Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your public key has been saved in /ceph-client/.ssh/id_rsa.pub. Copy the key to each Ceph Node
ssh-copy-id ceph@<node01> …
ssh-copy-id ceph@<nodexx>
Modify the ~/.ssh/config file of the ceph-deploy admin node so that it logs in to Ceph Nodes as the user created (e.g., ceph). Host <node01>
Hostname <node01 fully qualified domain name> User ceph
…
Host <nodexx>
Hostname <nodexx fully qualified domain name> User ceph
Ensure connectivity using ping with short hostnames (hostname –s).
Create a Cluster
Start cluster installation
Create the cluster staging directory then set up the initial config file and monitor keyring. mkdir cluster-stage; cd cluster-stage
ceph-deploy new <initial-monitor-node(s) fully qualified domain names> Initial Configuration modification
Making some configuration modifications at this step will avoid restart of affected services during this install. It’s recommended to make ceph.conf changes in this staging directory rather than /etc/ceph/ceph.conf so new configuration updates can push to all nodes with ‘ceph-deploy --overwrite-conf config push <nodes>’ or ‘ceph-deploy --overwrite-conf admin <nodes>’.
Set replica counts to 3 and min count for writes to 2 so pools are at enterprise reliability levels. Replication at this level consumes more disk and network bandwidth but allows repair without data loss risk from additional device failures. This also allows for a quorum on object coherency since odd counts > 1 can agree on a majority.
<cluster creation dir>/ceph.conf osd_pool_default_size = 3 osd_pool_default_min_size = 2
If the object gateway is installed per the Ceph default instructions, related pools will be created automatically on demand as the object gateway is utilized—which means starting with defaults. The default of 8 PGs is low, although it may be appropriate for object counts in very lightly utilized pools. Too boost defaults based on cluster size, here are the configuration parameters.
<cluster creation dir>/ceph.conf [global]
…
osd_pool_default_pg_num = <default_pool_placement_group_count> osd_pool_default_pgp_num = <default_pool_placement_group_count>
If you want to offload cluster network traffic like our sample reference configuration did, you’ll need to specify both public (data) and cluster network settings in ceph.conf using the network and netmask slash notation.
<cluster creation dir>/ceph.conf [global]
…
public_network = <public network>/<netmask> cluster_network = <cluster network>/<netmask>
Install Ceph Software
This step pulls down the Ceph distribution packages and installs onto all cluster role servers.
If using ceph-deploy to install Ceph packages and using a proxy server to get to the internet, edit wgetrc’s proxy
configuration under all Ubuntu nodes. Otherwise, ‘ceph-deploy install’ will get stuck trying to get the release key with wget. Aptitude should be configured with proper proxy settings during OS installation.
Example from /etc/wgetrc
https_proxy = <proper proxy server url> http_proxy = <proper proxy server url>
ceph-deploy install --release dumpling <node01>...<nodexx> Create Monitors
Add initial monitors and gather the keys.
ceph-deploy mon create-initial <monitor01>…<monitorxx> Add OSDs
The typical manual flow for adding an OSD to the cluster with SSD journals is below, with the SSD as /dev/sdt and the OSD on /dev/sda for the target host.
If there’s no partition table on the target journal SSD, create one. ssh hp-osdhost01 sudo “parted -s /dev/sdt mklabel gpt”
Create partition on the target journal SSD. ssh hp-osdhost01 -s mkpart cephjournal01 0G 4G
Create new partition table on the OSD. ceph-deploy has failed trying to clear a partition table when repurposing a drive; an explicit redo of the table has proved more reliable.
ssh hp-osdhost01 sudo "parted -s /dev/sda mklabel gpt"
Prepare and activate the OSD (create command does both it in one step) ceph-deploy --overwrite-conf osd create hp-osdhost01:sda:sdt1
The scripts below are simple examples for setting up all drives on a box in a batch. These are not robust (no real error handling, output help, etc.) but can be a useful starting point for command syntax/function. All these scripts assume the install instructions above such that no ssh password entry is necessary.
A more robust creation mechanism would probably leverage orchestrating software, but even with occasional hiccups the scripts below generally suffice if adding new OSDs is not all that common of a task.
Sample script for creating SSD journal partitions, 4 per ssd interleave. #!/bin/bash
tgtsys=${1}
if [ -z "${tgtsys}" ]; then echo "No target system." exit 1
fi
tgtdrv=${2}
if [ -z "${tgtdrv}" ]; then echo "No target disk." exit 1
ssh ${tgtsys} sudo parted -s ${tgtdrv} mklabel gpt p_layout=( 0G 4G 8G 12G 16G )
start_idx=0 end_idx=1
while [ ${end_idx} -lt ${#p_layout[@]} ]; do
ssh ${tgtsys} sudo parted ${tgtdrv} -s mkpart cephjournal${end_idx} ${p_layout[${start_idx}]} ${p_layout[${end_idx}]} (( start_idx=end_idx ))
(( end_idx++ )) done
Sample script for adding OSDs to the cluster. #!/bin/bash
destbox=${1}
if [ -z "${destbox}" ]; then echo "No target system." exit 1
fi
partdev=$(echo sd{a..t} )
journaldev=( $(echo sd{u..y}{1..4}) ) journal_idx=0
for devid in ${partdev}; do echo "working on ${devid}"
ssh ${destbox} sudo "parted -s /dev/${devid} mklabel gpt"
ceph-deploy --overwrite-conf osd create ${destbox}:${devid}:${journaldev[${journal_idx}]} (( journal_idx++ ))
done
Create Admin Node
The server is administered on the same box as the primary monitor/object gateway. Adding read permissions on the admin keyring and ceph configuration allows cluster administrator operations without having to be root.
ceph-deploy admin hp-cephmon01
sudo chmod +r /etc/ceph/ceph.client.admin.keyring sudo chmod +r /etc/ceph/ceph.conf
Verify Cluster Health
The cluster should be complete. Check health status of the cluster and cluster state information to make sure the cluster looks like it should.
ceph health ceph -s
An example of command output from a healthy cluster configuration: cloudplay@hp-cephmon02:~$ ceph -s
cluster 8fd2af32-987c-48a7-9a7b-e932bd88024b health HEALTH_OK
monmap e1: 3 mons at {hp-cephmon01=10.9.25.17:6789/0,hp-cephmon02=10.9.25.18:6789/0,hp-
cephmon03=10.9.25.19:6789/0}, election epoch 8, quorum 0,1,2 hp-cephmon01,hp-cephmon02,hp-cephmon03 osdmap e822: 200 osds: 200 up, 200 in
pgmap v106577: 6336 pgs: 6324 active+clean, 12 active+clean+scrubbing; 12639 GB data, 38329 GB used, 508 TB / 545 TB avail
mdsmap e1: 0/0/1 up
cloudplay@hp-cephmon02:~$ ceph health HEALTH_OK
Default Object Storage Placement Group Count
The majority of placement groups should lie in the pools with the most RADOS objects. In an object storage focused cluster, this pool will default to .rgw.buckets. Using the cluster tuning guidelines for placement groups, this step is a good place to
create the default pool here so object gateway install doesn’t create one sub-optimal default placement group settings. Remember to balance object gateway usage with amount of rbd storage required.
sudo ceph osd pool create .rgw.buckets <pg_count>
Add Object Gateways
The ceph-deploy package does not support object gateways, but changes to the configuration are driven from the staging directory created in the above cluster installation step. If testing involves redoing the cluster from scratch frequently, this is manual enough that it is worth scripting or otherwise orchestrating. For the sample reference configuration, object gateways are installed on all of the monitors and load balanced.
All of the steps below are performed on the target system directly, except for the push of the ceph.conf which occurs on the staging system (in this case hp-cephmon01). Installation must be performed and individually tailored for each system performing the object gateway role.
Apache/FastCGI W/100-Continue
The Ceph community provides a slightly optimized version of the apache2 and fastcgi packages. The material difference is that the Ceph packages are optimized for the 100-continue HTTP response, where the server determines if it will accept the request by first evaluating the request header. If there are specific apache requirements, it may be possible to run with the stock server.
Add ceph-apache.list file to APT sources.
echo deb http://gitbuilder.ceph.com/apache2-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-apache.list
Add ceph-fastcgi.list file to APT sources.
echo deb http://gitbuilder.ceph.com/libapache-mod-fastcgi-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph-fastcgi.list
Update repository and install Apache and FastCGI.
sudo apt-get update && sudo apt-get install apache2 libapache2-mod-fastcgi Configure Apache/FastCGI
Open the apache2.conf file:
sudo vim /etc/apache2/apache2.conf
Add a line for the server name in the Apache configuration file. Provide the fully qualified domain name of the server machine.
Edit /etc/apache2/apache2.conf:
ServerName <fully qualified domain name>
Enable the URL rewrite modules for Apache and FastCGI sudo a2enmod rewrite
sudo a2enmod fastcgi
Restart Apache so that the foregoing changes take effect. sudo service apache2 restart
Enable SSL
Because this sample configuration is targeted at enterprise customers, SSL is configured. Ensure dependencies are installed.
sudo apt-get install openssl ssl-cert Enable the SSL module.
sudo a2enmod ssl Generate a Certificate sudo mkdir /etc/apache2/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key -out /etc/apache2/ssl/apache.crt
Restart Apache
sudo service apache2 restart Install Ceph Object Gateway
The Ceph packages don’t pull down object gateway software by default, so add that now. sudo apt-get install radosgw
Add gateway configuration to Ceph
HP recommends this step and the configuration step be executed from the deployment directory used for cluster creation. For each object gateway host, there’s a separate section for their definition. When running the scripts to start the service the host-name field matches the proper configuration to start, and the gateway name is the piece of the <cluster>-<id> combo that identifies the cephx user authenticating for that gateway instance. For example, with hp-cephmon01 the matching gateway name is gateway01, so the below would be for [client.radosgw.gateway01].
[client.radosgw.<gateway name>] host = <object gateway host-name>
keyring = /etc/ceph/keyring.radosgw.gateway rgw socket path = /tmp/radosgw.sock log file = /var/log/radosgw/radosgw.log Redeploy Ceph Configuration
Strictly speaking only the new object gateway needs the update, but it’s a best practice to keep the configuration files in sync. If not manually editing /etc/ceph/ceph.conf on the gateway machine, run this command to deploy the config file changes to the cluster.
ceph-deploy --overwrite-conf config push <nodes> Create Data Directory
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.<gateway name> Create Gateway Configuration
Create an rgw.conf file under the /etc/apache2/sites-available directory on the host where the Ceph Object Gateway was installed. This configuration accomplishes a few things:
• Configure FastCGI as an external server to Apache.
• Sets a rewrite rule for Amazon S3 compatible interface (not use under this test). • Configure the mod_fastcgi module.
• Allow encoded slashes, provide log file paths, and turn off server signatures. • Enable standard HTTP and SSL config.
The below is a literal config for hp-cephmon01, replace ServerName and ServerAdmin with the appropriate name of the host where the object gateway is being installed.
FastCgiExternalServer /var/www/s3gw.fcgi -socket /tmp/radosgw.sock <VirtualHost *:80> ServerName <hp-cephmon01> ServerAlias *.ldev.net ServerAdmin [email protected] DocumentRoot /var/www RewriteEngine On
RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] <IfModule mod_fastcgi.c> <Directory /var/www> Options +ExecCGI AllowOverride All SetHandler fastcgi-script Order allow,deny
Allow from all AuthBasicAuthoritative Off </Directory> </IfModule> AllowEncodedSlashes On ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined ServerSignature Off </VirtualHost> <VirtualHost *:443> ServerName <hp-cephmon01> ServerAlias *.ldev.net ServerAdmin [email protected] DocumentRoot /var/www RewriteEngine On
RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] <IfModule mod_fastcgi.c> <Directory /var/www> Options +ExecCGI AllowOverride All SetHandler fastcgi-script Order allow,deny Allow from all
AuthBasicAuthoritative Off </Directory>
</IfModule>
AllowEncodedSlashes On
ErrorLog /var/log/apache2/error.log
CustomLog /var/log/apache2/access.log combined ServerSignature Off SSLEngine on SSLCertificateFile /etc/apache2/ssl/apache.crt SSLCertificateKeyFile /etc/apache2/ssl/apache.key SetEnv SERVER_PORT_SECURE 443 </VirtualHost>
Enable the Configuration
Enable the site for rgw.conf, disable the default site. sudo a2ensite rgw.conf
sudo a2dissite default
Add Ceph Object Gateway Script
Create the object gateway script in /var/www/s3gw.fcgi #!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.<gateway name> Make sure the script is executable.
Generate Keyring and Key for the Gateway
Here a keyring is created on the object gateway install system. These steps also set up read access for administrative ease of use, and attach the gateway user to the cluster and keyring file. For simplicity, this config doesn’t bother merging gateway keyring files across object gateways.
• sudo ceph-authtool --create-keyring /etc/ceph/keyring.radosgw.gateway • sudo chmod +r /etc/ceph/keyring.radosgw.gateway
• sudo ceph-authtool /etc/ceph/keyring.radosgw.gateway -n client.radosgw.<gateway name> --gen-key • sudo ceph-authtool -n client.radosgw.<gateway name> --cap osd 'allow rwx' --cap mon 'allow rw'
/etc/ceph/keyring.radosgw.gateway
• sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.<gateway name> -i /etc/ceph/keyring.radosgw.gateway
Restart Services and Start the Gateway sudo service ceph restart
sudo service apache2 restart sudo /etc/init.d/radosgw start Create a Gateway User
To use the Swift and S3 APIs through the object gateway, a user account is required. This was done extensively for the seeding part of the test with an automatic script. Since tests used the Swift API and SW_AUTH through the object gateway, each account involves setting up a user, a swift subuser and a key for the subuser to authenticate to.
sudo radosgw-admin user create --uid=testusr --display-name="Test User"
sudo radosgw-admin subuser create --uid=testusr --subuser=testusr:swift --access=full sudo radosgw-admin key create --subuser=testusr:swift --key-type=swift --gen-secret
You may want to modify read permissions for /etc/ceph/ceph.client.admin.keyring to allow radosgw-admin usage without sudo.
To validate the object gateway is working, you can utilize swift client to do a ‘list’ on a user account created. Even without any objects written, the command should return without error if the object gateway is working. When using the subuser secret key, watch out for keys with escapes of ‘/’ (\/ represents just /). You may need to delete the escape character depending on how you’re using the key.